Why doesn't AlphaGo expect its opponent to play the best possible moves? In the game won by Lee Sedol, AlphaGo was apparently surprised by a brilliant and unexpected move from Lee Sedol.
After analysing the logs, the Deep Mind CEO said that AlphaGo had evaluated a 1/10000 probability for that specific move to be played by Lee Sedol.

What I don't understand here is : whatever the probability for a good move to be played, why taking the risk ? Why not instead expecting the opponent to always plays the best moves ?

Of course it's always possible that you miss the best move your opponent could play when playing Monte Carlo to evaluate his possibilities, but here it seems that the move was founded. If AlphaGo knew that its strategy could be counter by such a move, why not choosing another strategy, where the worst case scenario would be less "worse".

It appears that AlphaGo did not rate the move as a best possible move for Lee Sedol, just as one that was within its search space. To put into context the board is 19x19, so a 1 in 10000 chance of a move is much lower than chance of the square being picked at random. That likely makes the move that it "found" not worth exploring much deeper.

It is important to note too that the probabilities assigned to moves are equivalent to AlphaGo's rating for quality of that move - i.e. AlphaGo predicted that this was a bad choice for its opponent. Another way of saying this, is "there is a probability p that this move is the best possible one, and therefore worth investigating further". There is no separate quality rating - AlphaGo does not model "opponent's chance of making a move" separately from "opponent's chance of gaining the highest score from this position if he/she makes that move". There is just one probability covering both those meanings 1

As I understand it, AlphaGo rates the probabilities of all possible moves at each game board state that it considers (starting with the current board), and employs the most search effort for deeper searches on the highest rated ones. I don't know the ratios or how many nodes are visited in a typical search, but expect that a 1 in 10000 rating would not have been explored in much detail if at all.

It is not surprising to see the probability calculation in the system logs, as the logs likely contain the ratings for all legal next moves, as well as ratings for things that didn't actually happen in the game but AlphaGo considered in its deeper searches.

It is also not surprising that AlphaGo failed to rate the move correctly. The neural network is not expected to be a perfect oracle that rates all moves perfectly (if it was, then there would be no need to search). In fact, the opposite could be said to be the case - it is surprising (and of course an amazing feat of engineering) just how good the predictions are, good enough to beat a world-class champion. This is not the same as solving the game though. Go remains "unsolved", even if machines can beat humans, there is an unknown amount of additional room for better and better players - and in the immediate future that could be human or machine.

There are in fact two networks evaluating two different things - the "policy network" evaluates potential moves, and the output of that affects the Monte Carlo search. There is also a "value network" which assesses board states to score the end point of the search. It is the policy network that predicted the low probability of the move, which meant that the search had little or no chance of exploring game states past Lee Sedol's move (if it had, maybe the value network would of detected a poor end result from playing that through). In reinforcement learning, a policy is set of rules, based on known state, that decide between actions that an agent can take.


How to tune XGBClassifier's each and every hyper-parameter "efficiently"? It's a generic question on tuning hyper-parameters for XGBClassifier()
I have used gridsearch but as my training set is around 2,00,000 it's taking huge time and heats up my laptop.

I need codes for efficiently tuning my classifier's parameters for best performance.

Its kind of difficult to avoid a large processing time when you have many parameters and a large sample.

A better method than gridsearch is Bayesian Optimization, this library even has done examples for xgboost ( https://github.com/fmfn/BayesianOptimization )
A good trick as well is to take a reduced sample of your data set, if you have over 2 million samples, you might acquire very similar results with smaller subsets (given that they have a similar distribution).


Distributed DL model with Tensorflow Suppose I want to develop and train a big end-to-end deep learning model using Tensorflow (1.15, for legacy reasons). The objects are complex, with many types of features that can be extracted: vector of numeric features of fixed length, sequences, unordered sets, etc. Thus, the model will include many submodules to deal with various types of features.

I have access to a server with several GPUs, so I want to distribute the model across them. What is the best way to do so? So far I'm thinking about placing subsystems on separate GPUs, but this presents some questions:

How costly would be the transfer of computation results between GPUs? Tensorflow does it automatically, right?
How costly would gradient computation and descent be, considering variables are placed on different GPUs? Would gradients also be computed on the same GPUs as their corresponding variables?

I invite you to look at the Horovod project on github.
It's the most efficient way to currently execute distributed training with tensorflow. They have tutorials and benchmarks resource available


Can I do incremental learning with the sklearn implementation of Linear Discriminant Analysis I have a large number of pictures that I would like to use LDA on. However, it requires too much memory, so I was wondering if it would be possible to make the learning incremental, using a sklearn implementation, or maybe another library?

Yes, you can. Instead of using lda.fit(...)or lda.fit_transfrom(...) you basically just need to call lda.partial_fit(...). Here you take a mini-batch to update your model.
See this link for more information.


I want to create an additional feature(column) based on some manipulation of values from existing features Consider my data-frame to be like this ('x','y','z' are features):

I want to create a python function which will take an expression as a string (something like this: 'x+y-2z') and create a new feature by evaluating the expression. Output should be like:

I want to generalize this function so that it will work for different data-frames with different column(feature) names in the expression.

Edit- I have a prototype of the desired function(named 'parser'):

def parser(exp):

df['new_col'] = df.apply(lambda row: row.x+row.y-2*row.z, axis=1)

However, I want to generalize this part - row.x+row.y-2*row.z so that it will adjust itself according to the string(i.e. expression) provided as its argument.

Welcome to the community!

The code below is a starter. You can go on by naming the column and adding it to the original DataFrame:

import pandas as pd

data = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30],'z':[100,200,300]})
print(data)
def my_fun(data,expression,variables):
    for v in variables:
        expression = expression.replace(v,'data.'+str(v))
    return eval(expression)

my_fun(data,'2*x+y',['x','y'])

output

   x   y    z
0  1  10  100
1  2  20  200
2  3  30  300

0    12
1    24
2    36
dtype: int64

There are two ways in general:

Use eval function to evaluate/execute expression in string form as I did.
Use symbolic libraries, most commonly used is SymPy, to use symbolics directly as variables.

Hope it helps. Good Luck!


sklearn : missing pruning for decision trees Why pruning is not currently supported in scikit-learn? How can we tune the decision trees to make a workaround?

This is something which is planned to be done. Setting the minimum number of samples required at a leaf node or a split as well as setting the maximum depth of the tree are how you want to work around this.


On the offline evalution of recommender system There are mainly three ways to evaluate a recommender system: offline, online and user study.
For most academic papers, offline evaluation is used to show the improvements:

They split the offline dataset into training and testing sets
They train models on the training dataset
They evaluate them on the testing dataset.

However, it seems that for most nonsequential recommendation works, the dataset splitting is random, without considering the temporal sequential order of the records. I think this may cause a problem of overestimating.

For example, consider evaluating itemCF on MovieLens: if the dataset is random split, the order of the samples will be shuffled. It means that we may use future data to train a model and make a prediction about the past:

  The training dataset contains that Bob bought a Harry Porter book in April 2019 (which is the second purchase of Harry Porter), and the
  testing dataset contains that Bob bought a Harry Porter book in March
  2019 (which is the first purchase of Harry Porter). An item-based CF
  method may recommend Harry Porter for the first purchase due to the
  second purchase, which is against the law of causation with a result
  of overestimating the performance of item-based CF.

It is kind of cheating. And I fail to see this problem being discussed somewhere.
So, I doubt the real performance of the offline evaluation for recommender system and would like to hear more voices about this.

Sequential data (i.e. data that can be understood in sequences, such as time series) require apposite train-test splitting. When dealing with sequences, the split must be temporal as well.

It's the same principle that that must be followed when training language models, to make the most common example. This is well known in the literature.

I suggest you to find a timestep threshold in your data, and use that to split between train and test set.


How should one deal with implicit data in recommendation A recommendation system keeps a log of what recommendations have been made to a particular user and whether that user accepts the recommendation. It's like

user_id item_id result
1       4       1
1       7       -1
5       19      1
5       80      1

where 1 means the user accepted the recommendation while -1 means the user did not respond to the recommendation.

Question: If I am going to make recommendations to a bunch of users based on the kind of log described above, and I want to maximize MAP@3 scores, how should I deal with the implicit data (1 or -1)?

My idea is to treat 1 and -1 as ratings, and predict the rating using factorization machines-type algorithms. But this does not seem right, given the asymmetry of the implicit data (-1 does not mean the user does not like the recommendation).

Edit 1
Let us think about it in the context of a matrix factorization approach. If we treat -1 and 1 as ratings, there will be some problem. For example, user 1 likes movie A which scores high in one factor (e.g. having glorious background music) in the latent factor space. The system recommends movie B which also scores high in "glorious background music", but for some reason user 1 is too busy to look into the recommendation, and we have a -1 rating movie B. If we just treat 1 or -1 equally, then the system might be discouraged to recommend movie with glorious BGM to user 1 while user 1 still loves movie with glorious BGM. I think this situation is to be avoided.

Your system isn't just trained on items that are recommended right? if so you have a big feedback loop here. You want to learn from all clicks/views, I hope.

You suggest that not-looking at an item is a negative signal. I strongly suggest you do not treat it that way. Not interacting with something is almost always best treated as no information. If you have an explicit signal that indicates a dislike, like a down vote (or, maybe watched 10 seconds of a video and stopped), maybe that's valid.

I would not construe this input as rating-like data. (Although in your case, you may get away with it.) Instead think of them as weights, which is exactly the treatment in the Hu Koren Volinsky paper on ALS that @Trey mentions in a comment. This lets you record relative strength of positive/negative interactions.

Finally I would note that this paper, while is very likely to be what you're looking for, does not provide for negative weights. It is simple to extend in this way. If you get that far I can point you to the easy extension, which exists already in two implementations that I know of, in Spark and Oryx.


Forecasting an individual based on a representative group I’m trying to create a model to forecast demand/expenditure for an individual based on historical data of many individuals over time, but I’m having trouble finding examples of this.

To give a simple example of what I’m trying to do:

Let’s say there’s an ice-cream store that children in the surrounding area visit every week. We’ve got the history of all purchases at the store and details of who made the purchase, so we can track the purchase history of each individual. We also have details about each individual, e.g. age, gender, school, no. of siblings etc.

We can assume that a child will buy from the store once a week until they ‘leave’ for good and stop purchasing.

What I want to do is create purchase forecasts for each child over a few weeks. The bit I’m struggling with is where a new child, say Timmy, has only been purchasing for a few weeks and so doesn't have enough of a history to forecast based solely on his own data (but we do have his details).

My assumption is that demand can be better predicted by taking a child's details into account, so I'd like to find a group in the historical data that best represents Timmy, and leverage their data in some way when creating Timmy's forecast. Are there any standard methods/practices for this type of forecasting?

Any advice or guidance would be much appreciated.

I think you are doing a supervised classification problem.  You have labelled data — previous customers, which you could label as different groups (e.g. high spend / low spend) along with predictors about the customer — their details (e.g. age).

You could think of each of these customer details representing dimensions on a plot, wth your task being to find out where different groups of customers are clustered along these dimensions.  Once you know that you can predict what group a new customer will likely belong to, based on their details.

There are many methods for solving this sort of problem, including but not limited to support vector machines, gradient descent and nearest neighbour.  Which one you choose will depend on the amount and sort of data you have.  A really good guide if you are starting out is this flowchart in the SciKit learn documentation, which isn't exhaustive, but gives a useful overview.


What is the meaning of a quadratic relation when r = 0? A website (on page 4) says:

  The correlation coefficient is a measure of linear relationship and thus a value
  of  r = 0 does not imply there is no relationship between the variables. For
  example in the following scatterplot which implies no (linear)
  correlation however there is a perfect quadratic relationship.

What is the meaning of a perfect quadratic relationship?

Below I have graphed a linear and quadratic function, $y = 50x + 3$ and $y=5x^2 - 1000$, respectively.

Before calculating anything, we can observe that $x$ and $y$ are related to one another in some form. If we describe in words, we can say for the left plot that as $x$ increases so does $y$.

Similarly, for the right plot, we can say that as $x$ moves towards 0, from the left, $y$ decreases towards 0, and as $x$ moves away from 0 towards the right, $y$ increases.

In fact, $x$ and $y$ have a perfect linear relationship in the left plot, while $x$ and $y$ has a perfect quadratic relationship in the right plot. We can say this because the points, in both plots, lie on the red line

If we take it a step further and calculate the correlation between x and y.

We observe for the linear relationship that $x$ and $y$ has a correlation of 1. In contrast, for the quadratic relation, $x$ and $y$ has a correlation of 0.

Below you will find R code for the plot above, in addition, code to calculate the correlation coefficients

set.seed(1)
library(ggplot2)
library(dplyr)
library(gridExtra)

# Linear Function
linear = function(x){(50)*x + 3 }

# Quadratic Function
quadratic = function(x){(5)*x^2 - 1000}

# Create Data Frame
df = data.frame(x = c(-20:20),
                y = c(sapply(-20:20, quadratic),sapply(-20:20, linear)),
                type = c(rep('quadratic',41),rep('linear',41)) )

# Plot Functions
lp = ggplot(subset(df,type == 'linear'), aes(x,y)) + geom_point() +
  stat_function(fun=linear, colour="red") + ggtitle('Linear Relationship')

qp = ggplot(subset(df,type == 'quadratic'), aes(x,y)) + geom_point() +
  stat_function(fun=quadratic, colour="red") + ggtitle('Quadratic Relationship')

grid.arrange(lp, qp, ncol=2)

# Calculate correlation coefficients
df %>% filter(type =='linear') %>% select(x,y) %>% cor %>% (function(x){x[1,2]})
df %>% filter(type =='quadratic') %>% select(x,y) %>% cor %>% (function(x){x[1,2]})


Why do we need 2 matrices for word2vec or GloVe Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix factorization of the ppmi of the co-occurrences of the words in the corpus.

Still, I can't understand why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V ? Is it a problem with the gradient descent or is there another reason ?

Someone told me it might be because the embeddings u and v of one word should be far enough to represent the fact that a word rarely appears in its own context. But it is not clear to me.

Might not be the answer you are seeking, but I'll still have a go:

First, quick review of word2Vec, assume we are using skip gram.

A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)

Input:  10 000
Hidden:  300
Output: 10 000

There is a matrix E between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons -  a response to a corresponding incoming 1-hot vector.

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.

However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning.  And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output

We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.

This second matrix will make an entirely different one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. In other words, this output won't be your original one-hot.

That's why a second matrix is needed.
At the output, we perform a softmax, like in a classification problem.

This allows us to express a relation "word"-->embedding-->"context-neighbor-word"

Now, backpropagation can be done, to correct the Input-Hidden weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.

In that case you wouldn't use this:

  You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task

but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.

Why a second matrix is needed during training:

Once again, recall, there is no activation at the hidden layer.

We can dictate the network what "one-hot" must have been created in response to your original "input one-hot", and can punish network if it fails to generate a correct answer.

We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word

During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.


Why is my CNN activation maximization split vertically? I am using this code in Python:
import matplotlib.pyplot as plt
import tensorflow as tf
from tf_keras_vis.activation_maximization import ActivationMaximization

def get_conv(in_model):
    target_layer = in_model.get_layer(name="block5_conv3")
    new_model = tf.keras.Model(inputs=in_model.inputs, outputs=target_layer.output)
    new_model.layers[-1].activation = tf.keras.activations.linear
    return new_model

def get_score(output):
    outputs = [o[0] for o in output]
    return outputs

model = tf.keras.applications.vgg16.VGG16()
vis = ActivationMaximization(model, get_conv)
seed = tf.random.uniform((1, *model.input.shape[1:]), 0, 255)
maps = vis(get_score, seed)
plt.imshow(maps[0])
plt.show()

And this is what I get:

I was wondering why there is a huge flat image region in the bottom half of the image.

I'm answering this myself as it may be helpful to others.
The error is in this line:
outputs = [o[0] for o in output]

I had copied and pasted this from dense-layer visualization code, where it is correct, selecting class 0 for visualization. Here, it is not wrong, but what it does is select the top row of that conv layer across all filters (and not the first filter of that layer). Hence the many small-scale features near the top row, more large-scale features near the middle, and nothing more near the bottom. I believe this may nicely visualize the perceptive field of the network.
For the record,
outputs = [o[:, :, 0] for o in output]

yields expected output.


What type of NN should be used for a turn-based game (with direct access to data)? I am very intrigued and I decided to do a little project in my free time to get a hold of it.The goal of my project is to make a NN that can play a turn-based game.

I have searched for a bit about this subject and all I could find was NNs that use images of the game to make decisions, but what if I already have the game's data directly? What type of NN would work best in this situation?

I am hoping you can point me a bit to the right direction if possible.

Thank you.

Your question is quite broad so it is not easy to give a direct answer.  However, a simple approach and a good introduction to machine learning would be to use a Multilayer Perceptron (MLP).  This is one of the most basic and fundamental neural network architectures.

You will also want to read about Backpropagation, which is the standard method of training such algorithms.

Here are some code examples/libraries (in no particular order) that can get you started:

Keras - (Python)
Theano - (Python)
DeepLearning4J - (Java)
Caffe  - (C++/Python/Matlab)
TensorFlow - (C++/Python)
Weka - (Java)


Interpreting machine learning coefficients My dog show predictive tool is having some trouble with its neural net. Broadly, I start with a couple of factors--age, weight, height, breed (which is a set of dummy variables), a subjective cuteness score--and predict whether the animal will win best in a show (a binary variable). I've got a few hundred shows, so tens of thousands of rows. I'm stuck using the nnet package instead of neuralnet, but I can pretty easily create a decent neural net using
model <- nnet(Win ~ ., x=X_training, y=Y_training, size=30, maxit=500)

which works fine. I've tested it against the holdout test data and the AUC and confusion matrix looks good, and I can use it for predicting future winners. The problem is, how do I determine which variable is most important in predicting the outcome for each new dog show? When I run a prediction for future winners, I use the following code
winpred = predict(model, x, type='raw')

which provides a score between 0 and 1 of the likelihood of winning. What I'd also like is some way of identifying for each row which variable--age, weight, height, breed dummies, and cuteness--contributed the most and least to the score in my predictions. If I were doing this for a single dog, I would use LIME or something else based on this article, but unfortunately, I want to run this on a few hundred odd dogs at a time.
Basically, I want to be able to say to dog owners that their pup is doing well because of x, but what's most holding it back is y (not terribly helpful if it's breed, of course). It seems like there isn't a way to do this with neural networks.  Are there other machine learning techniques now that would allow this?  If not, any suggestions on other models that would work?
I was thinking of using OLS as a linear probability model or logit/probit, perhaps with LASSO or Ridge.  I could use my machine learning model to score dogs, but in the background pull from the regression to identify which variables contribute most strongly, positively, and negatively, to the score.  Logit/probit are better from a statistical perspective, but is there any easy way to interpret in R/RStudio, for a few hundred dogs at a time, which attribute contributed most strongly to their score?

Neural Networks are notoriously good at performance and bad at interpretability, i.e. it's very difficult (almost impossible) to explain why a particular prediction was made. It's even more difficult to link the prediction with the features, since the NN does a lot of intermediate calculations where all the features play a role.
The notoriously good models for interpretability are the simple ones: a decision tree is very intuitive to interpret, assuming the tree is not too large. A human can actually follow the tree based on the conditions on the features.


Forecasting: How Decision Tree work? For example I have the following data structure:

user: Chris
age: 32
income: 60.000
basket value: 45

I want predict the basket value, and my features are the age and income.

With a linear regression I get a regression function as the result of the fitting for example: $$y = 0.5x + 0.785$$

Now I can use the function for prediction.

What is the form of the result of the fitting by regression decision tree? Is it also a function?

Yes. It is also a function, but not an affine transformation of the input but a relatively complex sum of products of indicator functions of the input. Usually, this function is represented by the fitted tree and not as a formula.

So e.g. if you learn a tree of depth one and the split is at age 40 with mean response of 80 if age < 40 and mean response of 100 if age $\ge$ 40, then the function could look like
$$
  \hat f(\text{age}, \text{income}) = 80 \cdot {\mathbf 1}\{\text{age} < 40\} + 100 \cdot {\mathbf 1}\{\text{age} \ge 40\}
$$
You can maybe imagine how long the formula is if the depth is 7...


Creating an Artifical Neural Network that produces a set of possible outputs I'm new to machine learning, and was working on creating an ANN which would classify each observation to a certain value. I have worked primarily with the sigmoid function up to this point to get the probability of an observation (true/false or binary output).

In this instance, I want each observation to be classified to one of 5 values: 0, 1, 2, 3, 4. I am using the sklearn library's StandardScalar function to scale the input data. What would be recommended in terms of:

Best way to scale the output data, and
Appropriate activation function to use for the output layer.

Thanks!

You don't need to scale the output data. For classification with a ANN the best activation is the softmax function:

f(x) = e^x_i / sum e^x_j

Which normalizes an input vector by applying the exponential function element wise, then dividing by the sum. This produces a discrete probability distribution.

Then your output layer should have 5 output neurons, apply softmax, and you will get a discrete probability distribution over the set [0, 1, 2, 3, 4]. Then you don't need to normalize the data, just do a one-hot encoding. Note that this might differ depending on the framework used.


Multivariate Time Series Binary Classification I have continuous (time series) data. This data is multivariate. Each feature can be represented as time series (they are all calculated on a daily basis). Here is an example.

Days    F1  F2  F3  F4  F5  Target
Day 1   10  1   0.1 100 -10 1
Day 2   20  2   0.2 200 -20 1
Day 3   30  3   0.3 300 -30 0
Day 4   40  4   0.4 400 -40 1
Day 5   50  5   0.5 500 -50 1
Day 6   60  6   0.6 600 -60 1
Day 7   70  7   0.7 700 -70 0
Day 8   80  8   0.8 800 -80 0

F1, F2, .. F5 are my features and Target is my binary classes. If I use a window size of 3, I can convert my features into time-series data. Then, I will have [10,20,30] for feat_1, [1,2,3] for feat_2 and so on. With the window size of 3, I have 5 feats * 3 window_size, a total of 15 features if written in the same vector.

The problem with this method is putting them into the same vector might cause some problems since the feature values are different

Example of multivariate time series (15 features in 1 network):

[10, 20, 30, 1, 2, 3, 0.1, 0.2, 0.3, 100, 200, 300, -10, -20, -30]
[20, 30, 40, 2, 3, 4, 0.2, 0.3, 0.4, 200, 300, 400, -20, -30, -40]
....
[60, 70, 80, 6, 7, 8, 0.6, 0.7, 0.8, 600, 700, 800, -60, -70, -80]

The other option is to create separate time series network (RNNs mostly, LSTM or CNN or their combination) for each of the features with the same target and then combine their results. In this scenario, I have 5 different networks and all of them are univariate time series binary prediction.

Example of different networks with univariate time series data (3 features in 5 networks):

[10, 20, 30]
...                            This is for network 1
[60, 70, 80]

[1, 2, 3]
...                            This is for network 2
[6, 7, 8]

...

[-10, -20, -30]
...                            This is for network 5
[-60, -70, -80]

The problem with this one is, I might lose information of the feature correlation even though I'm putting their results into another network.

My question is, which is the best way to use when dealing with multivariate time series problems? I want to use the first method but value differences worry me. Second method is easier but I worry that I might be losing some essential information.

You can add all features as input to RNN/LSTM (Day #, F1, F2, ... F5) and binary class as output.
This article has an example of such network.


Loss function returns x whereas tensorflow shows validation loss as (x+0.0567) I have a custom loss function. In order to experiement how the loss is calculated during valiation, I update the loss function as follows:

def custom_loss(y_true, y_pred):
    return 0 * tf.reduce_mean(input_tensor=-tf.reduce_sum(input_tensor=y_true_1 * tf.math.log(y_pred_1), axis=-1)) + 1

Multiply by 0, to make the data dependent loss value as always 0.
Add a constant value like 1 to it, so that the loss function always returns a value of 1.

Now when I run model.evaluate on my data, tensorflow shows me the loss value as 1.0567.

To further experiment, I updated the constant value being returned by loss function from 1 to other constant values. Below you can see the table:

constant 0 => loss 0.0567
constant 1 => loss 1.0567
constant 2 => loss 2.0567

I was expecting the final calculated loss to be same as the constant value I passed. But that's not the case. What is the reason for a constant difference of 0.0567?

I have verified it across three versions of tensorflow: 1.14.0, 1.15.0, and 2.1.0.

On further investigation was able to narrow it down.

The loss value is not only calculated by the response from loss function, but also the various regularizers added in the intermediate layers.

And one of the FC layers in my model was indeed having a L2 regularization term as below:

model.add(layers.Dense(1024, activation='relu', kernel_regularizer=regularizers.l2(0.1))

If I remove the l2 regularization, the loss goes indeed goes to zero if the loss function returns a zero.


Is linear regression fit for this data I am predicting number of vehicles in 4 traffic junctions.

So, I have following columns in my dataset :

DateTime
Junction_ID
Number_of_vehicles

At the first glance , this problem may look like Time series regression. But, the data given seems like Linear Regression problem.

So, I have applied linear regression in the following manner :

Used get_dummies extensively for all the columns. I used dummy variables for 31 days,24 hours ,7 days of weeks and 4 Junction Ids.
Then applied Linear Regression model

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(train_data,train_vehicles)

clf.fit(x_train,y_train)

import math

pred=clf.predict(x_test)

pred.shape #got result as (12030,)

result = []
for x in pred:
result.append(math.ceil(x))

from sklearn.metrics import mean_squared_error

score=mean_squared_error(y_test, result)
rmse=math.sqrt(score)
print('RMSE is :', rmse)

I am getting RMSE value as 10.636853077462394

My questions are :

Since RMSE value is on lower side , can I say this model is decent ?
Is there any other approach which I can use on this dataset ?
Do I need to check for colinearity ?
How can I check if multiple variables are interrelated ?
Should I go for non-linear regression on this dataset ?

For the first question, it is important to recall that RMSE has the same unit as the dependent variable. It means that there is no absolute good or bad threshold, however you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not small.

I would do some feature engineering (Create more variables: Time of Day, Day of Week, Month etc...) and run it through a Neural Network and then check for accuracy.  You may want to check if there is any correlation between the 4 junctions as well but if you run a NN, you don't have to.


What is the difference between LeakyReLU and PReLU? I thought both, PReLU and Leaky ReLU are
$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$
Keras, however, has both functions in the docs.
Leaky ReLU
Source of LeakyReLU:
return K.relu(inputs, alpha=self.alpha)

Hence (see relu code)
$$f_1(x) = \max(0, x) - \alpha \max(0, -x)$$
PReLU
Source of PReLU:
def call(self, inputs, mask=None):
    pos = K.relu(inputs)
    if K.backend() == 'theano':
        neg = (K.pattern_broadcast(self.alpha, self.param_broadcast) *
               (inputs - K.abs(inputs)) * 0.5)
    else:
        neg = -self.alpha * K.relu(-inputs)
    return pos + neg

Hence
$$f_2(x) = \max(0, x) - \alpha \max(0, -x)$$
Question
Did I get something wrong? Aren't $f_1$ and $f_2$ equivalent to $f$ (assuming $\alpha \in (0, 1)$?)

Straight from wikipedia:

Leaky ReLUs allow a small, non-zero gradient when the unit is not active.
Parametric ReLUs take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters.


Generate 3D "matrix" with Pandas, based on comparing two dataframes [Python] Good morning everyone. I am working with Python and Pandas.
I have two DataFrames, of the following type:
df_C = pd.DataFrame(data=[[-3,-1,-1], [5,3,3], [3,3,1], [-1,-1,-3], [-3,-1,-1], [2,3,1], [1,1,1]], columns=['C1','C2','C3'])

   C1  C2  C3
0  -3  -1  -1
1   5   3   3
2   3   3   1
3  -1  -1  -3
4  -3  -1  -1
5   2   3   1
6   1   1   1

df_F = pd.DataFrame(data=[[-1,1,-1,-1,-1],[1,1,1,1,1],[1,1,1,-1,1],[1,-1,-1,-1,1],[-1,0,0,-1,-1],[1,1,1,-1,0],[1,1,-1,1,-1]], columns=['F1','F2','F3','F4','F5'])

   F1  F2  F3  F4  F5
0  -1   1  -1  -1  -1
1   1   1   1   1   1
2   1   1   1  -1   1
3   1  -1  -1  -1   1
4  -1   0   0  -1  -1
5   1   1   1  -1   0
6   1   1  -1   1  -1

I would like to be able to "cross" these two DataFrames, to generate or one in 3D, as follows:

The new data that is generated must compare the values of the df_F with the values of the df_C, taking into account the following:

If both values are positive, generate 1
If both values are negative, generate 1
If one value is positive and the other negative, it generates 0
If any of the values is zero, it generates None (NaN)

True table
Comparison of the data df_C vs df_F
df_C vs df_F = 3D
  +       +     1
  +       -     0
  +       0     None
  -       +     0
  -       -     1
  -       0     None
  0       +     None
  0       -     None
  0       0     None

You, who are experts in programming, could you please guide me, as I generate this matrix, I compare the values. I wish to do it with Pandas. I have done it with loops (for) and conditions (if), but it is visually unpleasant and I think that with Pandas it is more efficient and elegant.
Thank you.

Pandas does not provide a 3D data structure per se (Panel has been deprecated and removed) - however, it is possible to express this kind of data using the long format (also known as EAV) with three key columns (or index levels), instead of the wide format you are suggesting.
Starting from your definitions, I believe the following achieves your goal:
(
    df_C
    # Transform to long format (two columns: former column names under `variable`
    # and corresponding values under `value`) plus the original index.
    .melt(ignore_index=False)
    # Join with the other dataframe, similarly transformed. join() implicitly joins
    # on indexes, so this will generate all combinations of the `variable` column values.
    .join(df_F.melt(ignore_index=False), lsuffix='_C', rsuffix='_F')
    # Make the index a regular column.
    .rename_axis('index')
    .reset_index()
    # Your rules can be expressed by multiplying the two value columns and examining the sign.
    .assign(combined=lambda df: df.value_C * df.value_F)
    .assign(output=lambda df:
        # Uses the Pandas nullable boolean type (three values: True, False, NA).
        pd.Series(pd.NA, index=df.index, dtype='boolean')
        # If combined is positive, both values were non-zero with the same sign.
        .mask(df.combined > 0, True)
        # If combined is negative, both values were non-zero with opposite signs.
        .mask(df.combined < 0, False)
        # If combined is zero, either of the values was zero, and the NA is retained.
    )
    # Remove intermediary values. The first three columns can also be transformed
    # to a MultiIndex.
    [['index', 'variable_C', 'variable_F', 'output']]
)


Is it feasible to train a single model on 150+ classes? I have classes for different regions. (Let say 80 classes for 3 regions each).
Will it be ok if I train my CNN model with 240 classes or should I create 3 models for each region? The classes for each region are similar to the classes of other region with same name but there is some difference in the elements of each class from each region and hence given some other class name.

I currently have one trained model which gives good accuracy for one region with 80 categories. Now that I have to move with remaining classes, it will be time-saving if I know beforehand to train one single model with 240 categories or three different models with 80 categories each.

I have 300 images for each class on average.

A Classifier's quality depends from the number of available observations for each class. If you have 300 images for each class, as you said above, it might be enough (it depends from the quality of your data).
In any case, it would require a massive amount of data augmentation.

Training a Network on 240 classes could be very challenging, if you don't have a proper infrastructure. That all depends on training times, but I'd suggest you to break down the problem in smaller subproblems. That would also make debugging and improvements easier to implement.


Understanding how distributed PCA works As part of big data analysis project, I'm working on,

I need to perform PCA on some data, using cloud computing system.

In my case, I'm using Amazon EMR for the job and Spark in particular.

Leaving the "How-to-perform-PCA-in-spark" question aside, I want to get an understanding of how things work behind the scenes when it comes to calculating PCs on cloud-based architecture.

For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.

When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.

How then is the covariance matrix calculated in such case when each node have only partial data?

This is just an example. I'm trying to find some paper or documentation explaining all this behind-the-scenes voodoo, and couldn't find anything good enough for my needs (probably my poor google skills).

So I can basically summarize my question(s) \ needs to be the following:

1. How distributed PCA on cloud architecture works

Preferably some academic paper or other sorts of explanation which also contains some visuals

2. Spark implementation of D-PCA

How does Spark do it? Do they have any 'twist' in their architecture to do it more efficiently, or how does the RDD objects usage contribute to improving the efficiency? etc.

A presentation of even an online lesson regarding it would be great.

Thanks in advance to anyone who can provide some reading material.

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is

For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features.
When using HDFS based architecture for example, the original data is distributed across multiple nodes, I'm guessing each node receives X records.
How then is the covariance matrix calculated in such case when each node have only partial data?

I shall address that, which hopefully will clear the matter to a degree.
Let us look at a common form of covariance calculation, $\frac{1}{n}\sum(x-\bar{x})(y-\bar{y})$
This requires you to calculate the following:

$\bar{x}$
$\bar{y}$
$x-\bar{x}$ and $y-\bar{y}$
Multiply the $(x-\bar{x})$ and $(y-\bar{y})$

in a distributed manner. The rest is simple, let us say I have 100 datat points (x,y), which is distributed to 10 Apache Spark workers, each getting 10 data points.
Calculating the $\bar{x}$ and $\bar{y}$: Each worker will add $x/y$ values of 10 data points and divide this by 10 to arrive at partial mean of $x/y$ (this is the map function). Then the Spark master will run the aggregation step (in Spark DAG of the job) where the partial means from all 10 workers are taken and again added, then divided by 10 to arrive at the final $\bar{x}$ or $\bar{y}$ (the aggregate/reduce operation)
Calculating the $(x-\bar{x}) \cdot (y-\bar{y})$: Same way, distribute the data points, broadcast the $\bar{x}$ and $\bar{y}$ values to all the workers and the calculate the partial $(x-\bar{x}) \cdot (y-\bar{y})$, again run aggregation to get $\sum (x-\bar{x})(y-\bar{y})$
The above method is used for distributed calculation, you shall get the covariance, for multi-dimensional data, you can get the covariance matrix.
The point is to distribute the calculation for stages that can be distributed and then centralize the calculation stages that cannot be distributed. That is in effect one of the important aspect of Spark architecture.
Hope this helps.


Why is oversampling outperforming class weight? I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for class 0 and 1 for class 1 (in keras), I am getting extremely bad results. However, if I oversample the dataset, I am getting really good results. What is the reason behind this?

It could be your sampling strategy

If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint will be seen over and over.

You could try another oversampling strategy, for example, SMOTE or ADASYN. These techniques create data points that are closets to decision boundaries so you are less inclined to overfit on "easy" data points.

Another things you can try is oversampling the minority class and undersampling the majority class that the same time. When choosing a method to do this, pick one that can oversample near decision boundaries and undersample away from decision boundaries. For example, here is SMOTETomek. Notice how classes purple and green get mainly oversampled and class yellow mainly undersampled.

These images come from imbalanced-learn which is a Python package you can use for all these sampling strategies.

It could be your pipeline

If you use your oversampled data for testing your model performance, you could be (unwillingly) manipulating your results. You need to ensure that you use your augmented data only for training, and not for validation and testing.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+


How to create a model to recognize matching label and ROI with OCR I am trying to make a model in python using Tensorflow to process the Tesseract OCR to detecting and extracting particular ROI from Image. I want to recognize the particular fields and values from Invoice with our model. For example i want to extract the Totoal amount and itemized amount and prices which is in tabular format. I want to grab only those details with our model. I am able to grab the 4 ROI cordinates of label with its's value manual in OCR. I am able to achieve these so far:

Four coordinates with OCR output for specific ROI in image
My sample dataset is like in this format [231, 404, 352, 616, 'Total Amount'] where first four values are coordinates and the last one is a label.
I am doing manually with mouse selection with my custom made python script where i select the label and all its value in particular area, row/columns with values.

This is a sample of image:

Now i want to train and prepare a model so that when i test with other invoice image having different value my model can detect and extract only that particular area on image with label and all those values either in row or column.

Now my question is How can i achieve this, How to train a model to detect only those area having label with Total Amount and all its's value present in that area on invoice image. Thanks.

There are multiple ways to do it. Here is one technique:
Run OCR on your image and search for your desired text . in this case it is Total.
Get the pixels of that word - run image_to_data method of ocr to get pixels.
Now you can extract the ROI using those pixels via OpenCV.


Is it possible to make use of the CPU RAM, if i'm running of of VRAM, in tensorflow? I want to train a model running on tensorflow. I have a GPU but it only has 6gb of VRAM. So i was wondering if it is possible to use some of the CPU's RAM to offload the GPU? I know it will be much slower, and i can reduce the batch size and number of layers etc. Can it be done?

If I understand correctly you are essentially looking for a way to use the CPU's RAM as a swap for the GPU's RAM. Unfortunately this isn't as easy to accomplish and might require some low level work. So if you're looking for a simple argument to add to your keras model, as far as I know there is none. Some options:

Some older tensorflow APIs supported this functionality (e.g. dynamic_rnn - see swap_memory parameter). However these are now deprecated in favor of the equivalent keras layer, that to my knowledge does not support this functionality. Theoretically you could use compatibility mode to utilize this.
You can specify what device each tensor will be loaded on beforehand. This isn't equivalent to swapping, but you could offload some computation to be regularly performed on the CPU. You can do this with tf.device.
The same idea as above is often applied to loading and preprocessing the data, which is done on the CPU, while everything concerning the training ops is handled by the GPU. This is convenient because you can have a batch being loaded by the CPU at the same time that the model is being trained by the GPU. I know this is far off from what you're asking, because it requires your model to fit in the GPU entirely, however it fits in the general theme of offloading work to the CPU. This is fully supported by tf.data.Dataset.
Use some trickery like gradient checkpoints to reduce the memory footprint of your models (with the expense of computational speed). This does not have anything to do with offloading work to the CPU, but it could help you fit your model into memory.


NLP varying amount of features and BoW as feature concatenating to feedforward NN I was looking at Google's Smart linkify machine learning models, as it closely relates to a personal project. And couldn't quite understand how the features are fed to the neural network.

It's about the following:

  Given a candidate entity span, we extract: Left context: five words
  before the entity, Entity start: first three words of the entity,
  Entity end: last three words of the entity (they can be duplicated
  with the previous feature if they overlap, or padded if there are not
  that many), Right context: five words after the entity, Entity
  content: bag of words inside the entity and Entity length: size of the
  entity in number of words. They are then concatenated together and fed
  as an input to the neural network.

I have 4 primary questions which I cant find a clear answer to:

The article specifies the features are concatenated. How does
a concatenation layer work internally? Does it concatenate all the
values in a single variable, in a very literal sense? how does that
work computationally?
How can a Bag of Words be a feature, when its a key-value pair? Or is it
also just all concatenated into one variable. Which again, how can
that work computationally?
The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?
Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Perhaps a simple keras model with some hardcoded input variables would help shape the answer.

Is there more material online to understand and recreate the entity recognition model?

EDIT:

The article also mentions the lack of available context or entities. E.g. the diagram shows 4 features for entity (2 for entity start, 2 for entity end). The article mentions duplication, but duplicating 3 times doesnt sound like a great idea. Would a convolution layer with a filter (1x3) work better?

And how would a Keras model look like then? Would it have two separate input layers? One input layer with 10 input features for the context + 1 feature for BoW + 1 feature for entity length. And another input layer with 4 input features followed by a convolution layer. And then both layers lead to a concatenate layer?

Let me take a crack at your questions:

The article specifies the features are concatenated. How does a concatenation layer work internally? Does it concatenate all the values in a single variable, in a very literal sense? how does that work computationally?

The concantenation of information in this context is, concatenation of vector represententations of the text. You could concatenate using a concatenation layer as described here. This is a very common approach followed where you want to feed your network information by taking various contexts depending on the problem you need to solve.

How can a Bag of Words be a feature, when its a key-value pair? Or is it also just all concatenated into one variable. Which again, how can that work computationally?

Bag of words, is typically a vector representation of the context. The above answer should help you. You could take a look at how to combine embeddings here as well

The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?

Yes, you are spot on. You combine the embedding vectors that you generate using skipgram or cbow, or you could even use one-hot encoded vectors.

Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Its always good to pad and use well defined dimensional vectors. Helps you structure your architecture better. You could use masking layer to ensure that the network ignores the paddings.

I hope this helps.


imblearn error installing smote I wanna install smote from imblearn package and I got the Following error:

ImportError                               Traceback (most recent call last)
<ipython-input-10-77606507c62c> in <module>()
     66 len(data[data["num"]==0])
     67 #balancing dataset
---> 68 from imblearn.over_sampling import SMOTE
     69 import matplotlib.pyplot as plt
     70 sm = SMOTE(random_state=42)

~\Anaconda3\lib\site-packages\imblearn\__init__.py in <module>()
     33 """
     34
---> 35 from .base import FunctionSampler
     36 from ._version import __version__
     37

~\Anaconda3\lib\site-packages\imblearn\base.py in <module>()
     17 from sklearn.utils import check_X_y
     18
---> 19 from .utils import check_sampling_strategy, check_target_type
     20 from .utils.deprecation import deprecate_parameter
     21

~\Anaconda3\lib\site-packages\imblearn\utils\__init__.py in <module>()
      5 from ._docstring import Substitution
      6
----> 7 from ._validation import check_neighbors_object
      8 from ._validation import check_target_type
      9 from ._validation import check_ratio

~\Anaconda3\lib\site-packages\imblearn\utils\_validation.py in <module>()
     12
     13 from sklearn.base import clone
---> 14 from sklearn.neighbors.base import KNeighborsMixin
     15 from sklearn.neighbors import NearestNeighbors
     16 from sklearn.externals import six

~\Anaconda3\lib\site-packages\sklearn\neighbors\__init__.py in <module>()
      7 from .kd_tree import KDTree
      8 from .dist_metrics import DistanceMetric
----> 9 from .graph import kneighbors_graph, radius_neighbors_graph
     10 from .unsupervised import NearestNeighbors
     11 from .classification import KNeighborsClassifier, RadiusNeighborsClassifier

~\Anaconda3\lib\site-packages\sklearn\neighbors\graph.py in <module>()
      5 # License: BSD 3 clause (C) INRIA, University of Amsterdam
      6
----> 7 from .base import KNeighborsMixin, RadiusNeighborsMixin
      8 from .unsupervised import NearestNeighbors
      9

~\Anaconda3\lib\site-packages\sklearn\neighbors\base.py in <module>()
     20 from .kd_tree import KDTree
     21 from ..base import BaseEstimator
---> 22 from ..metrics import pairwise_distances_chunked
     23 from ..metrics.pairwise import PAIRWISE_DISTANCE_FUNCTIONS
     24 from ..utils import check_X_y, check_array, gen_even_slices

ImportError: cannot import name 'pairwise_distances_chunked' from 'sklearn.metrics' (C:\Users\ASUS\Anaconda3\lib\site-packages\sklearn\metrics\__init__.py)

Try closing and restarting ipython. imblearn requires scikit-learn >= 0.20 and sometimes the runtime loads in an older version of scikit-learn.

If the issue still persists, then reinstall all packages together to make sure they are compatible.


Significant difference in testing vs cross validation accuracy What could be the possible reasons for a significant difference in cross validation and testing f1_scores? I am performing 3 fold Stratified cross validation and the testing f1_score is almost 0.15 less than cross validation score. How can I come up with a more effective cross validation strategy so that the two scores are closer?

Significant differences between the calculated classification performance in cross-validation and in the final test set appear obviously, when the model is overfitted.

A good indicator for bad (i.e., overfitted) models is a high variance in the F1-results of single iterations in the cross-validation.

Possible strategies to get a better estimation of the model performance would be:

useing more folds (e.g., 10-fold cross-validation, or leave-one-out cross-validation)
considering simpler models (e.g., less variables, more general parameters)
considering other machine learning algorithms


Finding Patterns in Error Reports - How to Approach? I am new to data science and have been doing research to familiarize myself and try to find a solution to my problem but I have not come across anything that seems to fit. I am looking to learn and use Python for this process.

My situation is as follows:
I have multiple error logs for the cars I am testing on. Every time there is an error (there are multiple types) I have a log that has many different characteristics of the car at that time of the error. These include things like time of day, speed, location, temperature, and many more.
EDIT: If needed I could also have data for normal operation as well.

My goal is to find patterns in the characteristics that have a correlation with and could be contributing to the error. I am not looking to develop a model that will predict future errors. I am only focused on seeing the patterns themselves and understanding them, and having it become more accurate the more data I give it when we encounter the error again.

For example, I want to feed it all my data for one type of error, and have it tell me something along the lines of "When speed is above x you will likely get this error", "When car is in reverse and running at night you will likely get this error", "There is no correlation between outside temperature and this error", etc. And then proceed to do this for every type of error.

Some of the variables could be dependant on each other. Some of the variables might not be relevant/meaningful.

How can I do this in Python? Any guidance is appreciated. I have tried to be as specific as I can and will update my post if further details are needed.

You need data for when the error occurred and for normal operation as well, so that you can gain insights from comparing these two groups of log data in analyses and visualizations, e.g. box plots.

You also should consider to regard a time span of log data from before the error happened as possibly related to the occurance of the error. So, you should group your log data into the following two groups by adding a boolean feature to each log entry or data set row: (1) a specific error happened within the next x seconds / minutes / hours / days and (0) a specific error did not happen within the next x seconds / minutes / hours / days.

It is always a good idea to visualize your data, because it helps you to understand your data, e.g. to see if your data is "dirty" or if your assumptions about a specific feature behavior are correct or not as well as related behavior of two or three features. In chapter 4 of the "Python Data Science Handbook" by Jake VanderPlas, the author shows how plot data using python and matplotlib.

In a next step, you can do preprocessing tasks like e.g. to clean your data so that you can later apply statistical methods or machine learning to it to find feature behavior related to error occurrances.

And here is a link to the free online course at edX "Analyzing Data with Python" which you could take.


Metrics for same signal Is there a metric in tensorflow.keras.metrics that counts how many time the predicted output and the real output have the same signal?

For example, if the y_pred = [-0.1, 0.3, -0.22, -1] and y_val = [-1.3, 1.2, 0.5, -2], then the output should be 0.75.

I am working on a project that takes as input, along with other features, a column of real values around 0, and should predicted if the next elements of this column should be negative or positive (regardless of their absolute value). As it's only logical, I am using tanh as the activation function, but if I use as metrics accuracy or binary_accuracy, the output is always zero.

It is better to formulate this problem in terms of classification, not regression. For example, you can apply sign transformation on your labels (e.g. $y = numpy.sign(y)$) before training the model, then fit the model with classification loss (like binary cross-entropy), and then you can use accuracy for measuring the performance. Using accuracy with regression setup outputs zero because comparing real numbers usually outputs False (like 1.2 != 1.19).


Using Machine Learning to Predict Temperature I am a beginner in ML and I want to create a smart thermostat, that after collecting enough data from the interaction with the user, it will start to set the home temperature by itself.

What I got so far is the hardware prototype that lets the user set the temperature, and in the same time it posts the Environment and the UserSetTemperature to ThingSpeak (to easily store the data for later access)

The other part is a python algorithm that gets the data from ThingSpeak and it converts it into a Pandas DataFrame.

The data frame looks like bellow:

timeStamp                      environment_temp    user_set_temp

2018-05-27T00:12:43Z               20            21
2018-05-27T00:17:27Z               20            22
2018-05-27T00:17:59Z               20            24
2018-05-27T00:20:01Z               20            21
2018-05-27T00:23:14Z               20            24
2018-05-28T09:39:07Z               20            22
2018-05-28T10:40:17Z               20            23
2018-05-28T20:12:47Z               20            25
2018-05-28T20:14:16Z               23            25
2018-05-30T20:29:30Z               18            24

And here is where I got stuck. I don't know how to use this data with the ML libraries in order to make predictions on how the temperature should be set when the environment temperature is x.

I tried to use the sklearn train_test_split() and LinearRegression(), but with no significant result. I really don't know how to use this data

Every suggestion will be highly appreciated!!

I would not recommend going ahead with the data that you know might be wrong.

Looking at your current data, the reason you got a bad result with linear regression is because the relation between them is not linear for the current data. For eg. There is high variation in response (i.e. user_set_temp)  for same value of your predictor (i.e. environment_temp).

First, get hold of correct temperature for your recorded timestamps from local weather data to replace environment temp with this data, till you get the issue rectified with your original recorded environment_temp. When you rectify the issue, then I would recommend you to use both weather and environment temp to predict, as a person might set the temperature depending upon a combination of both.

After you get a good representative data, this should be a reasonable procedure to help you predict the temperature:

Exploratory data analysis for uni-variate timeseries:
This will help you observe patterns in data, which will help you decide on which new features to engineer/ create from timestamps (step 2) you have of recordings. Also, look at the acf/ pacf plots to help figure out the lags in data that might help you predict better. Fit a basic uni-variate model like STL or ARIMA model to get a good base model for prediction (this model you will compare new models with to see how they perform)
Feature Engineering: Time of day, day of week, week of month etc identified in step 1. Also, look for any other features that might help your algorithm find patterns.
Regression Models: Test with advanced algorithms like RandomForest, Gradient Boosting and RNN-LSTM to check how they perform against each other and the base model.


Different number of features in train vs test when using Label Encoding This is not a duplicate of Different number of features in train vs test
There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding over OneHotEncoding. However, some categories in the validation set do not appear in the training set, and they cause my model to perform very poorly. I was scraping the internet but couldn't find a way.
Is there any way I can use to solve this issue?
Thank you in advance.

To solve the high cardinality problem, follow the recommendation of Erwan. To solve the second issue you are mentioning, fit the OneHotEncoder on the train set and use the transformer on the test set, like this:
from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown = 'ignore')

enc.fit(train)

X_train = enc.transform(train)

X_test = enc.transform(test)


How to specify different columns stacked vertically within CSV using pandas? I have a CSV file that looks like below with columns labeled below data, and I'm have trouble trying to retrieve said data with pandas. I'm only able to address the columns at the top with the specific rows within that parameter with using df["crit_rating"][0:3], however i'm unable to use df["agility] to address as a seperate column. Is there a way to do this without having the make another CSV file or reformatting the file?
crit_rating, DPS, DPS-Error
-1250, 4439.51, 11.2203
-1225, 4438.28, 11.3488
-1200, 4439.3, 11.2504
-1175, 4442.01, 13.0366

agility, cDPS, cDPS-Error
-1250, 1400.14, 3.62795
-1225, 1473.09, 4.28675
-1200, 1547.47, 4.47641
-1175, 1620.25, 4.63525

Where I'd like it to be:
agility, DPS, DPS-Error, crit_rating, cDPS, cDPS-Error
-1250, 1400.14, 3.62795, -1250, 4439.51, 11.2203
-1225, 1473.09, 4.28675, -1225, 4438.28, 11.3488
-1200, 1547.47, 4.47641, -1200, 4439.3, 11.2504
-1175, 1620.25, 4.63525, -1175, 4442.01, 13.0366

You can use pd.read_csv funcation with perameters of nrows and skiprows to read two different data frame and then concat them horizontally
import pandas as pd
df = pd.read_csv('your csv file path',nrows=4)
df1 = pd.read_csv('your csv file path',skiprows=5)
df = pd.concat([df, df1], axis = 1, sort = False)
print(df)

__output__

   crit_rating      DPS   DPS-Error  agility     cDPS   cDPS-Error
0        -1250  4439.51     11.2203    -1250  1400.14      3.62795
1        -1225  4438.28     11.3488    -1225  1473.09      4.28675
2        -1200  4439.30     11.2504    -1200  1547.47      4.47641
3        -1175  4442.01     13.0366    -1175  1620.25      4.63525

same you can implement with a single line of code i.e.
df = pd.concat([pd.read_csv('your csv file path',nrows=4), pd.read_csv('your csv file path',skiprows=5)], axis = 1, sort = False)


First Differencing to remove seasonality and trends I am trying to remove seasonality and trends from my time series data. I found this post that said to use df_diff = df.diff().diff(12).dropna() (https://www.tobiolabode.com/blog/2020/12/30/how-to-convert-non-stationary-data-into-stationary-for-arima-model-with-python). I don't understand why we need to use diff(12) from pandas. Could someone explain this to me? Thanks!

Since the data is recorded every month (i.e. a data point for each month in the year) and we see a yearly seasonality trend we compare the data for the same month against previous year. This is done by taking the difference against the data from last year (.diff()), which is equal to going back 12 observations since each observation is data for a specific month.


Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At the moment, the code works as follows:

1) Process the text of each job listing to extract skills that are mentioned in the listing

2) For each career (e.g. "Data Analyst"), combine the processed text of the job listings for that career into one document

3) Calculate the TF-IDF of each skill within the career documents

After this, I'm not sure which method I should use to rank careers based on a list of a user's skills. The most popular method that I've seen would be to treat the user's skills as a document as well, then to calculate the TF-IDF for the skill document, and use something like cosine similarity to calculate the similarity between the skill document and each career document.

This doesn't seem like the ideal solution to me, since cosine similarity is best used when comparing two documents of the same format. For that matter, TF-IDF doesn't seem like the appropriate metric to apply to the user's skill list at all. For instance, if a user adds additional skills to their list, the TF for each skill will drop. In reality, I don't care what the frequency of the skills are in the user's skills list -- I just care that they have those skills (and maybe how well they know those skills).

It seems like a better metric would be to do the following:

1) For each skill that the user has, calculate the TF-IDF of that skill in the career documents

2) For each career, sum the TF-IDF results for all of the user's skill

3) Rank career based on the above sum

Am I thinking along the right lines here? If so, are there any algorithms that work along these lines, but are more sophisticated than a simple sum? Thanks for the help!

Perhaps you could use word embeddings to better represent the distance between certain skills. For instance, "Python" and "R" should be closer together than "Python" and "Time management" since they are both programming languages.

The whole idea is that words that appear in the same context should be closer.

Once you have these embeddings, you would have a set of skills for the candidate, and sets of skills of various size for the jobs. You could then use Earth Mover's Distance to calculate the distance between the sets. This distance measure is rather slow (quadratic time) so it might not scale well if you have many jobs to go through.

To deal with the scalability issue, you could perhaps rank the jobs based on how many skills the candidate has in common in the first place, and favor these jobs.


How do I get the values of y_true and y_pred in the confusion matrix? I am using the weka program and i want to know the values of y_true and y_pred through confusion matrix that appeared in the classification results , because I want to calculate the balanced multiclass accuracy value .

https://www.cs.waikato.ac.nz/~ml/weka/
The screencap in the Evaluate predictive accuracy of Machine Learning without Programming shows the Correctly Classified Instances and the corresponding percentage.


Can I tune a model after training it? (Convolutional Neural Network & Classification) I am relatively new to Data Science and I've recently embarked on a project. Long story short, I've trained a CNN model to distinguish between Male and Female genders. However, I wish to tune my model. I've saved the weights and bias of the existing model.

Can I use the current model with weights and bias to retrain on augmented data (from the same initial dataset)? I am guessing there are 2 possible outcomes, it tunes the model or it "resets" the weights and bias.

Can I perform hyperparameter tuning with RandomSearchCV or GridSearchCV using on my existing model? I believe this will tune the model.

My aim is to reduce overfitting.

Before answering your questions, let's understand a few points -
1. Model training is nothing but finding the right weights going through the guiding path based on the loss function o/p using y_true and y_pred.
Training starts with random initialization of weights(definitely following some rules)
2. Neural Network training is an incremental training approach i.e. it doesn't see all the data in one go like the DecisionTree

Can I use the current model with weights and bias to retrain on augmented data (from the same initial dataset)? I am guessing there are 2 possible outcomes, it tunes the model or it "resets" the weights and bias

Yes, learning will continue without any big disruption as augmented images will produce a similar gradient like the real images.
It will not reset, as I said the training is an incremental process.
Any new image which has completely different features can definitely cause a large gradient to flow and might disrupt the weights but with augmented images, I don't think any such thing should happen.

Can I perform hyperparameter tuning with RandomSearchCV or GridSearchCV using on my existing model? I believe this will tune the model.

You can do this but you don't have too many hyperparameters left as Model is fixed. You may tune Learning rate.


Importing .ipnyb file from Kaggle into local Jupyter Total beginner question here, please let me know if it would be more appropriate somewhere else.

I just created my first iPython notebook in Kaggle and I downloaded the ipnyb file. Now I have installed Jupyter locally and want to try working on the same notebook that way.

It seems like I got Jupyter working because I am able to create a new local notebook and it looks like what I would expect:

But when I open the ipnyb file that I downloaded from Kaggle, I just see what looks like raw JSON instead of a live notebook:

I also noticed that the icons of these two notebook files look different:

Any suggestions about what I might be doing wrong and how I might get my properly import my Kaggle notebook into Jupyter locally?

Thats because "ipnyb" is not proper format. Try downloading/changing it properly to .ipynb extension.


How to get sensitivity and specificity for multi-class classification for each fold of cross validation? I am working on a multi-class classification consisting of 4 classes.
I am applying 5-fold cross-validation on it and would like to get the sensitivity (recall) and specificity score for each of those folds.

I found out that using cross_validate function, I can provide it with a list of scoring parameters for each fold.

    scoring = {'accuracy' : make_scorer(accuracy),
               'precision' : make_scorer(precision_score),
               'recall' : make_scorer(recall_score),
               'f1_score' : make_scorer(f1_score)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

However, this generates an error because these functions (except accuracy) are only for binary classification and not for multi-class.

So, I decided to make my own functions for sensitivity score and specificity score that returns the mean of the 4 individual values (1 for each class). I return the mean of them, and not the individual 4 values because scorer functions that return multiple values are not permitted. That is fine with me though as I want their mean only.

Here's what I tried:

    def sensitivity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Sensitivity = TP/(TP+FN)
        return np.mean(Sensitivity)

    def specificity(y_true,y_pred):
        cm=confusion_matrix(y_true, y_pred)
        FP = cm.sum(axis=0) - np.diag(cm)
        FN = cm.sum(axis=1) - np.diag(cm)
        TP = np.diag(cm)
        TN = cm.sum() - (FP + FN + TP)
        Specificity = TN/(TN+FP)
        return np.mean(Specificity)

    scoring = {'sensitivity' : make_scorer(sensitivity),
               'specificity' : make_scorer(specificity)}

    cross_validate(neural_network, data, y, cv=5,scoring=scoring)

But it still is throwing the same error:

ValueError: Classification metrics can't handle a mix of multilabel-indicator and multiclass targets

I don't know what's not working here. I just want the mean of sensitivity for each class and mean of specificity for each class, for each of the 5 folds.

What is wrong with my approach and also is there a simpler way to do this ?

I think this error comes from confusion_matrix(), here we have three "types_of_target": multiclass, multilabel-indicator, continuous-multioutput.

For example, np.array([1, 0, 2]) is multiclass, it's one-hot-encoding np.array([[0,1,0],[1,0,0],[0,0,1]]) is multilabel-indicator, what we predict np.array([0.3,0.4,0.3],[0.7,0.2,0.1],[0.1,0.1,0.8]) is continuous-multioutput.

The input of confusion_matrix must be of type "multiclass".

I think you can try

confusion_matrix(y_true.argmax(axis=1),np.rint(y_pred).argmax(axis=1))

by converting y_true from multilabel-indicator to multiclass, and y_pred from probs(continuous-multioutput) to one-hot(multilabel-indicator) then multiclass.


Future of deep learning (compared to traditional machine learning) What do you think will be the future of deep learning? A lot of people talk about deep learning and I can see that it provides various possibilities. For instance, this question gives a nice explanation.

What's the future of deep learning? Will it become the main branch of machine learning or will the other branches of machine learning remain relevant as well? And what about different fields of research? Will it be more suitable for certain applications (like natural language processing) and less suitable for some other types of applications?

I'd love to hear your views on this, both from a practical as from a scientific point of view. Any practical experience or research that can help to resolve this question? More specifically: should we introduce deep learning in our company?

That's very interesting topic. So thumbs up from my side. Now coming to the point. Deep learning may be hot now but some variants of it or something new all together may emerge later. Let me point out the reasons I feel deep learning maybe getting old or will get.

Slow Learner

Slow learner as in, it converges to an optimal solution slowly but with GPU acceleration the training speed can be improved dramatically. The slowness is affected by the learning rate.
Adjusting the learning rate affects the reliability of the resulting deep neural net.

Huge Training Data Requirement

Deep learning requires very huge training data to achieve good performance. The presence of a huge number of parameters to adjust requires some huge example sets. The fact that deep learning requires such a large number of training examples makes it dull in some way, despite such huge training set requirement deep neural nets do have error rates of about 10%.

Overfitting

Deep learning tends to overfit easily but with the new dropout algorithm, this problem can be avoided but with some consequences such as an increase in error rates.

Poor Initial State

Deep neural nets have parameters that need to be initialized. The most used method is random initialization, this results in neural nets with very poor initial states. Compare this to a mammalian brain, the brain is born with some rigid instincts such as basic survival behavioral patterns.

Sensory Data Transformations

For example in visual object recognition tasks, images undergo various geometric and photometric transformations that need to be modeled by a recognition system to rectify new image observations. Deep learning as used today does not take such transformations into account, this is one of the reasons why deep neural nets still suffer from relatively high error rates (relative to a human).
Some other reason can be the slow transformation, algorithms mayb emerge together in future, strcture and the approaches. These are some pin points from my side. You can also go through these posts, they're also on it and to the point-

KDnuggets
Huffington Post

Cheers! :)


Help with replacing values in a data frame I'm a data science student at Mills and working on CoCalc. I am working with the 'wine quality' set that can be found on the UCI website for my final project. I'm just working with the white wine portion. The target is the quality of the wine, rated from 1-10. I would like to change the values in the column 'quality' such that if the value is from 1-4, it is replaced with 'low', if the value is from 5-6, the value is replaced with 'medium', and if the value is 8-10, it is replaced with 'high'. My code is as such:
#Read in data
white = pd.read_csv('Wine_data.csv')

#Rename columns to exlude the u
columns = ['fixed acidity', 'volatile acidity', 'citric acid','residual sugar', 'chlorides', 'free sulfur dioxide','total sulfur dioxide', 'density', 'pH', 'sulphates', 'alcohol', 'quality']
white.columns = columns

quality = np.array(white['quality'])

for i in range(len(quality)):
    if quality[i] in [1,2,3,4]:
        quality[i] = 'low'
    elif quality[i] in [5,6,7]:
        quality[i] = 'med'
    else:
        quality[i] = 'high'

However, I am getting this error:
Error in lines 13-19
Traceback (most recent call last):
  File "/cocalc/lib/python2.7/site-packages/smc_sagews/sage_server.py", line 1013, in execute
    exec compile(block+'\n', '', 'single') in namespace, locals
  File "", line 5, in <module>
ValueError: invalid literal for long() with base 10: 'med'

Here is a link to my code.
Help this nOOb please.

Assuming that the data is small enough that efficiency is not an issue, I would use the Pandas data frame and do something like:

white['quality_code'] = 'high'
white.loc[quality < 5, 'quality_code'] = 'low'
white.loc[(quality > 4) & (quality < 8) , 'quality_code'] = 'med'


When does it make sense to add numbers with different units? Given two vectors containing numbers that have different natures / units, (example length in Meters and weight in Kilograms), does it make sense to calculate euclidean distance between these two vectors or cosine similarity?
The equations imply that you have to add $meters^2$ and $kilometers^2$ which supposedly does not make sense. Yet, I see this done many times indirectly, for instance, when calculating cosine similarity of documents based on tf-idf (vector that contains objects with dissimilar natures).

Speaking as an ex-physicist, I would say it never makes sense to add quantities with different units. When problems like this do arise it makes sense to honestly define some scale constants, normalize your quantities with respect to those constants and then add them.
This 'define and normalize' will likely not change much of your procedure, but being explicit about your constants can help to avoid problems later on.
If you want to consider meters and kilogarms to be part of the same metric space, it means that somewhere in your problem there must be a constant with units kilogram/meter, i.e. linear density. Such constants frequently arise in natural sciences, and can be quite interesting in their own right. Charge of electron, speed of light, are just few. I would suggest trying to understand how this linear density constant arises in your problem, might be an interesting insight.

It will become even more interesting if you have two constants with the same units in your problem. Then their relative values can indicate qualitative switch in behaviours. For example, https://en.wikipedia.org/wiki/Reynolds_number are unit-less numbers that indicate whether or not you can have turbulence.

I understand this is not a physics question :-)


Neural Network data type conversion - float from int? In case of image classification with 0-255 pixel values of integer type, is it necessary/recommended to convert the values into float with Neural Networks? This conversion seems me unnecessary (moreover, float slows down matrix operations) but several implementations follow this practice. Why?

Technically with most languages you could pass in integer features for the input layer, since the weights will be floats, and multiplying a float by an integer will give you a float. Also, you don't usually care about partial derivatives of the input data, so it doesn't matter that the values are discrete.

However:

For all weights and neuron activations, if you are using a method based on backpropagation for training updates, then you need a data type that approximates real numbers, so that you can apply fractional updates based on differentiation. Best weight values are often going to be fractional, non-whole numbers. Non-linearities such as sigmoid are also going to output floats. So after the input layer you have matrices of float values anyway. There is not much speed advantage multiplying integer matrix with float one (possibly even slightly slower, depending on type casting mechanism). So the input may as well be float.
In addition, for efficient training, the neural network inputs should be normalised to a specific roughly unit range (-1.0 to 1.0) or to mean 0, standard deviation 1.0. Both of these require float representation. If you have input data in 0-255 range - float or not - you will usually find the network will learn less effectively.

There are exceptions to these rules in some architectures and learning algorithms, where perhaps an integer-based input would work, but for the most common NN types, including MLP and "deep" feed-forward networks, it is simpler and easier to use float data type.


What model should I use for bounding box detection? I am working on building a cow detector for a local farm. I have a dataset of images with bounding boxes (not segmentation polygons) for every cow appearing in the images (different number of cows in every image). What is the best and easiest model to use to accomplish this task? Many cows stand behind others so there will be a lot of cases of occlusion. Is there a go-to model that I can get running quickly in Keras to perform well on this dataset? Any tips would be appreciated.

There are 2 object detectors that are quite popular:

Haar Cascade Classifier, introduced by Viola and Jones: available on OpenCV (Python, Java and C++) and on Matlab computer vision toolbox (and probably on many other languages) is a great model for when deep learning is not a option.
Yolo, You Only Look Once: It is a real-time object detector and classifier, it uses a DL CNN and if I am not mistaken has cows on it's training dataset (so you can use transfer learning). It is part of the Darknet library and you should totally check the site. Also, check Tiny Yolo is you need some more speed.

You can  should also take a look at:

MMOD (max-margin object-detection): has a great implementation using dlib.
Fast R-CNN and R-CNN: Deep learning classifier, check the paper on this site.

My advice would be to go straight to YOLO and try it out without training for the COW class, then do some fine tunning by retraining few layers with your data. If you need more accuracy than that you can go for R-CNN (probably has some models already trained with cows).

if you lack on computational power you can try MMOD and HCC. But if you got the time you should check BDNN: Binary convolution neural networks for fast object detection, don't know if it is implemented in any libraries already since this paper is relatively fresh (march 2019).


Does reinforcement learning require the help of other learning algorithms? Can't reinforcement learning be used without the help of other learning algorithms like SVM and MLP back propagation? I consulted two papers:

Paper 1
Paper 2

both have used other machine learning methods in the inner loop.

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well as a variety of similar techniques, such as Temporal Difference, SARSA. All these can be used without neural networks, provided your problem is not too big (typically under a few million state/action pairs).

The simplest form of Q-learning just stores and updates a table of <state, action> => <estimated reward> pairs. There is no deeper statistical model inside that. Q-learning relies on estimates of reward from this table in order to take an action and then updates it with a more refined estimate after each action.

Q-learning and related techniques such as Temporal Difference are sometimes called model free. However, this does not refer to the absence of a statistical model such as a neural net. Instead, it means that you do not need to have a model of the system you are learning to optimise available, such as knowing all the probabilities of results and consequences of actions in a game. In model free RL, all learning can be done simply by experiencing the system as an agent (if you do have a model then it may still be used for simulation or planning). When considering whether or not you need a neural network, then the term tabular is used for systems that work with explicit value estimates for every possible state or state/action pair. And the term function approximation is used to describe how a neural network is used in the context of RL.

For large, complex problems, which may even have infinite possible states, it is not feasible to use tabular methods, and you need good generalised value estimates based on some function of the state. In those cases, you can use a neural network to create a function approximator, that can estimate the rewards from similar states to those already seen. The neural network replaces the function of the simple table in tabular Q-Learning. However, the neural network (or other supervised ML algorithm) does not perform the learning process by itself, you still need an "outer" RL method that explores states and actions in order to provide data for the NN to learn.


Automatically Remove highly correlated features using Variance Inflation Factor? I want to be able to automatically remove highly correlated features. I am performing a classification problem using a set of 20-30 features and some may be correlated.

Multiple features can be correlated at once too and I fear it may pose a problem in my Logit model significances & coefficients of the features. After removing these features I plan to then also use this reduced feature set into an Xgboost model as well.

  Multicolinearity on the other hand is more troublesome to detect because it emerges when three or more variables, which are highly correlated, are included within a model. (Reference here)

Would it be correct to remove correlated variables this way?:

X=data[feature_set]
vif = pd.DataFrame()
vif['vif_factor'] = [variance_inflation_factor(X.values,i) for i in range(X.shape[1])]
vif['features'] = X.columns

vif.sort_values('vif_factor',axis=0,inplace=True, ascending=False)

features_to_remove = vif.loc[vif['vif_factor'] > 10,'features'].values
features_to_remove = list(features_to_remove)
print(features_to_remove)

Output:

vif_factor |  feature

21         |   age
9.7        |   income
7          |   gender ....and so on

So in this case age would be removed as a feature from the model.

I might oversimplify this, but Pandas allows to drop correlated features based on a threshold. E.g. correlation >0.95

https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/


Can parallel computing be utilized for boosting? Since boosting is sequential, does that mean we cannot use multi-processing or multi-threading to speed it up? If my computer has multiple CPU cores, is there anyway to utilized these extra resources in boosting?

You can estimate in parallel each of the weak learners. For example, searching for optimal splits in 'weak' decision trees can be streamlined by utilizing large number of cores.


Random forest mode scoring We are using random forest algorithm but having some trouble understanding the scoring method it uses.
take for example the following CM of the test set:
Threshold 45 cm is:
[[67969 48031]
 [ 3321 11120]] and the prescion is: 0.18799344051632602
Threshold 50 cm is:
[[77642 38358]
 [ 4785  9656]] and the prescion is: 0.2011080101632834
Threshold 55 cm is:
[[88825 27175]
 [ 6796  7645]] and the prescion is: 0.2195577254445159
Threshold 60 cm is:
[[100411  15589]
 [  9629   4812]] and the prescion is: 0.2358707906463611
Threshold 65 cm is:
[[112421   3579]
 [ 13098   1343]] and the prescion is: 0.2728565623674755
Threshold 70 cm is:
[[115895    105]
 [ 14371     70]] and the prescion is: 0.3999999997714286
Threshold 75 cm is:
[[115998      2]
 [ 14440      1]] and the prescion is: 0.3333333222222226
Threshold 80 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 85 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0
Threshold 90 cm is:
[[116000      0]
 [ 14441      0]] and the prescion is: 0.0

This is how we used the RF and printed it's score:
grid_clf = RandomizedSearchCV(clf, param_grid, cv=tscv, verbose=10,n_iter=20,n_jobs=-1,scoring='roc_auc')
grid_clf.fit(X_train, y_train)
print(grid_clf.score(X_test,y_test))

The score we got for this model is 0.7350173458471928
As far as I understand the scoring when using roc_auc is between 0.5 and 1.
How can such a bad model received such a good score?
How is this scoring calculated?
Provided we predicted enough True Positives, we don't mind missing "1"'s and predicting False Positives.
we of course do mind predicting True Negatives
Can I change the scoring to fit what I believe are better results?
Thanks

The evaluation metric used for classification .score is accuracy.
Your model has higher accuracy and lower precision. Looking at your confusion matrix, you have class imbalances which can lead to that result.
You should not change your evaluation metric to get better results. It is better to pick the evaluation that makes the most sense for the problem. Then improve the model through collecting more data, engineering better features, changing algorithms, or tuning hyperparameters.


How do MS-TDNNs work? Multi-State Time Delay Neural Networks (MS-TDNNs) were introduced in

  Haffner, Patrick and Waibel, Alex: Multi-state time delay networks for continuous speech recognition. In Advances in neural information processing systems, 1992.

They are an extension of TDNNs. TDNNs are convolutional neural networks for automatic speech recognition (ASR), where the convolution happens over the time.

The aim of MS-TDNNs seems to be to get rid of the hybrid approach in ASR, where you need dynamic programming / HMMs to chunk the audio stream and then neural networks to recognize the phonemes. Somehow MS-TDNNs seem to do the segmentation as well.

I don't understand how. Could somebody please explain it to me?

(Related side questions: Are MS-TDNNs recurrent networks? Where exactly does the name "multi-state" come from?)

So I think you understood that a MS-TDNN has two parts: a conventional TDNN that calculates state probabilities for each frame and kind of a perceptron on top of it linking the states of several frames to a word. The later part is doing the segmentation. Its connections aren't trained, but just used for running BP and train the TDNN.

The segmentation is obtained by some other algorithm (I don't remember exactly, but something like dynamic time warping). In normal DNN-HMM hybrid systems the DNNs are trained separately to predict the correct state for each frame. The error function is calculated on frame level!

This is not optimal as we do not care so much about each state being correct, but the final text. WER is calculated on world level. MS-TDNNs try to solve this by having this extra layer that allows passing a word level based error to BP. This is very similar to todays methods called sequence training.

Neither TDNN nor MS-TDNN are recurrent networks.

If you are interested in RNNs and ASR systems that do not require HMMs I would recommend you to look at CTC objective that does indeed learn the alignment automatically.


In SVM Algorithm, why vector w is orthogonal to the separating hyperplane? I am a beginner on Machine Learning.
In SVM, the separating hyperplane is defined as $y = w^T x + b$.
Why we say vector $w$ orthogonal to the separating hyperplane?

Geometrically, the vector w is directed orthogonal to the line defined by $w^{T} x = b$. This can be understood as follows:

First take $b = 0$. Now it is clear that all vectors, $x$, with vanishing inner product with $w$ satisfy this equation, i.e. all vectors orthogonal to w satisfy this equation.

Now translate the hyperplane away from the origin over a vector a. The equation for the plane now becomes: $(x − a)^{T} w = 0$, i.e. we find that for the offset $b = a^{T} w$, which is the projection of the vector $a$ onto the vector $w$.

Without loss of generality we may thus choose a perpendicular to the plane, in which case the length $\vert\vert a \vert\vert = \vert b \vert /\vert\vert w\vert\vert$ which represents the shortest, orthogonal distance between the origin and the hyperplane.

Hence the vector $w$ is said to be orthogonal to the separating hyperplane.