How many ways are there to check model overfitting? I am running xgboost on a regression classification problem where the model is predicting a score of how likely a gene is to cause a disease from 0-1.
I try to avoid overfitting in all the ways I can think of and the mean output of nested cross-validation is r2 0.88, I'm not sure if I can trust this or if there are other ways I can see if this is overfitting. The output r2 on just training and testing non-nested is: Train r2: 0.971 Test r2: 0.868.
So far I:

Remove features with a correlation >0.9 and remove any features with >50% missing data (this is hard to strengthen, a lot of genetic features simply have missing data for a lot of under studied genes in biology)
Have no imputation to avoid imputation bias, and since xgboost accepts missing data.
Scale features with MinMaxScaler() in scikit-learn - recommended as a good starting point and most features don't have a normal distribution
Compare 2 feature selection methods (one using features xgboost deems important from SHAP values and one using Boruta, both give 0.87-0.88 r2 on average of the  10 nested CV k-folds and only remove 3-4 out of 57 features)
Use nested kfold cross validation with 10 kfolds

The only other area I'm aware of that I haven't really explored is projection techniques. I am not sure which method would be best for this (my features are all numeric but mixed continuous or discrete data types) such as between UMAP, PCA or partial least squares.
Are there any other ways I can investigate overfitting? I have a biology background so any resources on this would be useful and any help appreciated.
I have also more manually removed some minority example genes before training (e.g. removed training genes with a 0.9 score which make up only about 1/8 of the training dataset) to give the trained model to predict and view how the model generalises to this 'new' hard to predict genes - gives them a 0.6-0.7 score when they are actually 0.9:
y_pred =[0.69412696, 0.709764, 0.6366122]

y_true = [0.9, 0.9, 0.9]

r2_score(y_true, y_pred) #outputs 0.0

10-fold nested cv r2 results per fold:
 'test_r2': array([0.8484691 , 0.86808136, 0.91821645, 0.93616375, 0.94435934,
       0.82065733, 0.84856025, 0.8267642 , 0.84561417, 0.89567455]

Edit:
A few other things I've tried:

I think I've misused classification here (and removed tag accordingly), I use regression models and I don't have labels and only continuous scores so I don't get true positives, false positives etc. to be able to do ROC. I'm not sure what other metrics are good or better than R2 for regression that I can use.

I have tried applying imputation to compare other models (random forest, SVM, and logistic rgeression with elasticnet or lasso), all models perform notably lower than gradient boosting (0.59 average nested r2 is the highest with random forest) - but I was originally concerned with biased data from imputation, is imputation worth doing to counteract overfitting?

I use GridSearch in scikit-learn for all my models with nested cross-validation, I should have included this information originally as I have been trying to always do this.

I have a biology background, so not sure about best practices for machine learning, but from this I'm suspecting random forest is better and I should be trying to do a better parameter tuning than I currently do for it, and trusting that model's result on nested CV. Is this the best approach?
Also not sure if how I tune my random forest is reasonable, currently I use:
rfr = RandomForestRegressor(random_state=seed)
rfr_params={'n_estimators':[100, 500, 1000],
             'min_samples_split': [50, 100],
             'min_samples_leaf': [50, 100],}

The direct way to check your model for overfitting is to compare its performance on a training set with its performance on a testing set; overfitting is when your train score is significantly above your cv score. According to your comments, your r2 score is 0.97 on the training set, and 0.86 on your testing set (or similarly, 0.88 cv score, mean across 10 folds). That's somewhat overfitting, but not extremely so; think if 0.88 is "good enough" for your requirements

The r2 score is 1 - MSE of errors / variance of true values. In the example you showed, all of the three true values were the same; i.e. their variance is zero. The r2 score should've been a negative infinite, but apparently sklearn corrects this to 0; you can verify that changing y_true to [0.9, 0.9, 0.90001] changes your r2 score to a very large negative number (around -2*10**9). This is why checking r2 against a small sample is not a good idea; the mean of the small sample contains too much important information.

You added that you want to know which parameters to tune in order to prevent over-fitting. In your edit to your question, you said you're using grid-search over n_estimators (3 options), min_samples_split (2 options) and min_sample_leaf (2 options).
There are other parameters you can try, and in my experience max_depth is important to tune. This question on Stack Overflow and this question on Cross Validated deal with overfitting, and there are good options there.  I'd add that if you're trying many options, then maybe you'd be better off doing using Bayesian Optimization (there's a package that functions well with SKLearn: https://scikit-optimize.github.io/stable/auto_examples/sklearn-gridsearchcv-replacement.html).


Convolutional Neural Networks layer sizes I am trying to understand an article Backpropagation In Convolutional Neural Networks

But I can not wrap my head around that diagram:

The first layer has 3 feature maps with dimensions 32x32. The second layer has 32 feature maps with dimensions 18x18. How is that even possible ? If a convolution with a kernel 5x5 applied for 32x32 input, the dimension of the output should be $(32-5+1)$ by $(32-5+1)$ = $28$ by $28$.

Also, if the first layer has only 3 feature maps, the second layer should have multiple of 3 feature maps, but 32 is not multiple of 3.

Also, why is the size of the third layer is 10x10 ? Should it be 9x9 instead ? The dimension of the previouse layer is 18x18, so 2x2 max pooling should reduce it to 9x9, not 10x10.

Actually I guess you are making mistake about the second part. The point is that in CNNs, convolution operation is done over volume. Suppose the input image is in three channels and the next layer has 5 kernels, consequently the next layer will have five feature maps but the convolution operation consists of convolution over volume which has this property: each kernel will have its width and height, moreover, a depth. its depth is equal to the number of feature maps, here channels of the image, of the previous layer. Take a look at here.


ROC-AUC curve as metric for binary classifier without machine learning algorithm Maybe it's a trivial question, but I'm a bit confused right now... I'll explain: I've some elements in my data, each with a value between 0 and 1 and an associated label (1, 0).
I need to test some thresholds and then compute the ROC-AUC curve.
For example with a threshold = 0.4, all the values greater than 0.4 will be predicted as true (1) and all the values under 0.4 will be predicted as false (0), then I'll compare the result with the actual labels to calculate the True Positive Rate and the False Positive Rate and finally build the ROC-AUC curve.

So, my question is, considering that, based on the threshold chosen, I already know under which prediction an element will fall, I don't need a machine learning classifier to do the default training and testing stages, right?

Your intuition is correct: you don't necessarily need a machine learning algorithm to calculate a ROC curve and, from what you say, it seems you already have the necessary ingredients in place.

In fact, the ROC curve was historically developed under circumstances that had nothing to do with machine learning; from Wikipedia:

  The ROC curve was first developed by electrical engineers and radar engineers during World War II for detecting enemy objects in battlefields and was soon introduced to psychology to account for perceptual detection of stimuli. ROC analysis since then has been used in medicine, radiology, biometrics, forecasting of natural hazards, meteorology, model performance assessment, and other areas for many decades and is increasingly used in machine learning and data mining research.


MOOCs for Python in Data Science Thanks also to SE, I've recently changed job and now I'm working in Data Science, mainly on Analytics for the IoT (Internet of Things). Analytics are applications on cloud platforms which collect real-time, streaming sensor data from industrial machines and allow to estimate their actual performance, predict the probability of a failure and the time before it happens, detect anomalies, etc..

Until now, I've been using R to build Statistical Learning models on datasets which would fit in the memory of my workstation, so I'm not a novice for what it concerns Statistical modeling and Data Science. However, I'm a novice with Python, and I need to learn it, expecially the part of the Python ecosystem related to Data Science, because that's what my team uses.

I don't need to develop the cloud platform: I just need to develop the "core" Analytics. I got the book by Jake Van der Plas:

https://www.amazon.com/Python-Data-Science-Handbook-Essential/dp/1491912057

but I would like to also follow a MOOC on using Python for Data Science. Can you suggest me one?

DISCLAIMER I already asked this question on CV and it wasn't considered very appropriate there. Since here my other question on MOOCs

MOOC or book on Deep Learning in Python for someone with a basic knowledge of neural networks

was well-received on this site, I thought of asking again here, after deleting the one on CV (no cross-posting). Hope this is fine.

Check out Applied Data Science with Python Specialization from Coursera. It is a series of 5 courses from the University of Michigan.


Trying to use different image size to test my trained model I have built my model using images which are all 512x384, I then exported the model through .pkl and am hosting it on Render, the UI is built on React App where the user will input their chosen image
I am aware that I could resize the images, either on the client side when the user inputs the image, on the server side using python, or I could change my model to accept different image sizes, could anyone advise what the simplest, most elegant solution would be for this?

Generally image classification models require input images to be reshaped, because they use fully connected layers to classify features learned by the previous convolutional layers, and shapes must match to make classification. On the other hand, fully connected CNNs only use convolutional layers, so they can work images with any shape.
If your model has fully connected layers, then you should reshape your input images.  If you send your images via network to get predictions, then reshaping images before sending can decrease network load, and response time for big images. If it is not the case I think you simply can reshape them right before you input them to model.


Reverse engineering a distance metric from the output of a k-NN Suppose that someone has trained a nearest-neighbor algorithm based on some unknown metric. I have a large dataset of $N$ observations and $P$ features. For each observation, I am given $K$ indices corresponding to the $K$ "nearest" points in the (same) dataset.

What is the general type of strategy to learn this metric? Does this type of problem have a standard name? Are there any Kaggle-type competitions with such a task, or is this unusual?

Interesting question. If you supposed that the distance metric was some common one, of course you could just try all the common ones and see what is consistent with the nearest neighbors it returns.

I can't think of a way to learn this reliably, because you don't have any info about the actual distances. Maybe treat all nearest-neighbor pairs as having 'small' distance and the rest 'large' and training a deep model that tries to learn to predict it across pairs of points. Whatever it comes up with as a metric would approximate it.


Tensorflow - logistic regression -oneHot Encoder - Transformed array of different size for both train and test x_train = tr1.loc[:, ['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
#x_train.shape - (120 x 4)

y_train = tr1.loc[:, ['Species']]
#shape - 120 x 3

x_test = test1.loc[:, ['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width']]
#shape 30 x 4
y_test = test1.loc[:, ['Species']]
# shape 30 x 3

oneHot = OneHotEncoder()
oneHot.fit(x_train)
# transform
x_train = oneHot.transform(x_train).toarray()
 # fit our y to oneHot encoder
 oneHot.fit(y_train)
 # transform
 y_train = oneHot.transform(y_train).toarray()

 oneHot.fit(x_test)
 # transform
 x_test = oneHot.transform(x_test).toarray()
 # fit our y to oneHot encoder
  oneHot.fit(y_test)
   # transform
  y_test = oneHot.transform(y_test).toarray()

   print("Our features X_test1 in one-hot format")
    print(x_test)

Shape of x_train:  (120, 15)
Shape of y_train:  (120, 3)
Shape of x_test:  (30, 14)
Shape of y_test:  (30, 3)

a) After conversion why is the size x_test = 30 x 14  I assume it has to be  30 x 15 ?

# hyperparameters
learning_rate = 0.0001
num_epochs = 100
display_step = 1

# for visualize purpose in tensorboard we use tf.name_scope
with tf.name_scope("Declaring_placeholder"):
# X is placeholdre for iris features. We will feed data later on
x = tf.placeholder(tf.float32, shape=[None, 15])
# y is placeholder for iris labels. We will feed data later on
y = tf.placeholder(tf.float32, shape=[None, 3])

with tf.name_scope("Declaring_variables"):
    # W is our weights. This will update during training time
    W = tf.Variable(tf.zeros([15, 3]))
    # b is our bias. This will also update during training time
    b = tf.Variable(tf.zeros([3]))

with tf.name_scope("Declaring_functions"):
# our prediction function
y_ = tf.nn.softmax(tf.add(tf.matmul(x, W), b))

b) did I define x, y, W, b correctly because when I run the accuracy I get this error "ValueError: Cannot feed value of shape (30, 14) for Tensor 'Declaring_placeholder_10/Placeholder:0', which has shape '(?, 15)'
"

Your shape is (30, 14) and not (30, 15) because there are only 14 unique values in your test (one is missing). In any case you shouldn't fit the encoder on the test set, just on the training set. Then just transform on the test set and you'll get the correct dimensions.

Also as far as I can see W and b are declared correctly. I'd ask however you take a minute to format your question a bit better next time.


Is it good practice to remove the numeric values from the text data during preprocessing? Im doing preprocessing on a text dataset. I have certain numerics in it like:

date(1st July)
year(2019)
tentative values (3-5 years/ 10+ advantages).
unique values (room no 31/ user rank 45)
percentage(100%)

Is it recommended to discard this numerics before creating a vectorizer(bow/tf-idf) for any model(classification/regression) development?
Any quick help on this is much appreciated. Thank you

Is it recommended to discard this numerics before creating a
vectorizer(bow/tf-idf) for any model(classification/regression)
development?

It depends on the problem statement for example year could be significant if you want to find the trend and year has many unique value but if it's constant then you can remove it.
To add to that if you are doing sentiment analysis then numeric variables don't make much sense.


Is there any APIs for crawling abstract of paper? If I have a very long list of paper names, how could I get abstract of these papers from internet or any database?

The paper names are like "Assessment of Utility in Web Mining for the Domain of Public Health".

Does any one know any API that can give me a solution? I tried to crawl google scholar, however, google blocked my crawler.

Look it up on:

Google Scholar link
Citeseer link

If you get a single exact title match then you have probably found the right article, and can fill in the rest of the info from there.  Both give you download links and bibtex-style output.  What you would likely want to do though to get perfect metadata is download and parse the pdf (if any) and look for DOI-style identifier.

Please be nice and rate-limit your requests if you do this.


Can preprocessing the whole population cause data leakage? Introduction

I understand the problem of data leakage that could be caused by the preprocessing step when our training and test sets are just samples of an unknown population. The preprocessing parameters should be calculated from the training set only, then we just apply the same procedure to validation/test set, since this would be the way to proceed with any other sample from the unknown population (in production stage, for example).

Question

What about the situation where we have the whole population at hand? Could we calculate the preprocessing parameters (scaling factors, encoding, etc.) from the entire population?

Extra Context

We have the whole population and the modeling process would depend of user input. The training set is defined by the user input and the trained model is used to classify the population.

If you have the entire population, there is no need for inference. Thus data leakage is not an issue. You can fit any transformation on the data without a concern for its effect on prediction because there is no prediction step.


Why is $Y=\beta_0 x^{\beta_1} e$ a linear model? Why is $Y=\beta_0 x^{\beta_1} e$ a linear model? When we apply the transform, it becomes $lnY = ln\beta_0+\beta_1 lnx +lne$, and why is it still linear when the $\beta_0$ part is under ln?

The term "linearity" is context-dependent, so a linear regression model is not necessarily the same as a linear function.
A linear function is classified as such via the superposition principle, requiring both additivity and homogenity. We can generalize linear maps with to multivariate functions simply:

Additivity: $f(\vec{x}_1+\vec{x}_2)=f(\vec{x}_1)+f(\vec{x}_2)\enspace \forall\ \vec{x}_1,\vec{x}_2\in \Bbb{R}^n$
Homogeneity: $f(\alpha \vec{x})=\alpha f(\vec{x})\enspace \forall\ \vec{x}\in \Bbb{R}^n$ and $\alpha\in \Bbb{R}$

So the equation $f(\beta_0,\beta_1)=\beta_0 x^{\beta_1}e$ is a nonlinear function because it is not additive nor homogeneous.
However, w.r.t. linear regression, linear models are of the form $Y=\beta_0+\beta_1f_1(x_1)+\beta_2f_2(x_2)+\cdots+\beta_nf_n(x_n)+c$, regardless of whether any $f_i$ is a nonlinear map.
So, after a nonlinear transformation such as natural logarithm, you can consider the resulting regression model as the form $Y'=\beta_0'+\beta_1ln(x)+1$ where the primes denote the natural log transformed variables. This demonstrates linearity between $Y'$ and the parameters $\beta_0'$ and $\beta_1$.


label encoding or one-hot encoding or none when using decision tree? I've been learning about decision tree from multiple resources but still not fully understanding data preprocessing step.
from https://www.youtube.com/watch?v=PHxYNGo8NcI&t=535s&ab_channel=codebasics it uses decision tree with label encoder and in another resource it says we don't need to convert categories to strings, I'm confused.
Given I have data that looks like
   gender       level        score
    male          1           34
    female        2           77
    female        1           44

If we are using label encoder we would only need to convert gender however if that maps male = 0, female = 1 wouldn't the machine treat female > male? and if it ignores ordinality it will ignore level1 < level2 and treat as if level 1 and level 2 are same level which is not true.
What is the right preprocessing step and why?

If we are using label encoder we would only need to convert gender however if that maps male = 0, female = 1 wouldn't the machine treat female > male?

You are correct, using label encoder to encode categorical features is wrong in general, for the reason you mention. Note that scikit documentation advises against using it with features, it's supposed to be used only with a response variable.
In the particular case of a binary variable like "gender" to be used in decision trees, it actually does not matter to use label encoder because the only thing the decision tree algorithm can do is to split the variable into two values: whether the condition is gender > 0.5 or gender == female would give the exact same results.
Also note that whether the variable is interpreted as ordinal or not is a matter of implementation. For example in Weka it's possible to specify that a feature is categorical ("nominal").

and if it ignores ordinality it will ignore level1 < level2.

Not necessarily, because in theory it's possible to have features with different types (e.g. some categorical and some numerical). However this may depend on the implementation as well.


What are the differences between Convolutional1D, Convolutional2D, and Convolutional3D? I've been learning about Convolutional Neural Networks. When looking at Keras examples, I came across three different convolution methods. Namely, 1D, 2D & 3D. What are the differences between these three layers? What are their use cases? Are there some links or references to show their use cases?

Conv1D is used for input signals which are similar to the voice. By employing them you can find patterns across the signal. For instance, you have a voice signal and you have a convolutional layer. Each convolution traverses the voice to find meaningful patterns by employing a cost function.

Conv2D is used for images. This use case is very popular. The convolution method used for this layer is so called convolution over volume. This means you have a two-dimensional image which contains multiple channels, RGB as an example. In this case, each convolutional filter should be a three-dimensional filter to be convolved, cross-correlated actually, with the image to find appropriate patterns across the image.

Conv3D is usually used for videos where you have a frame for each time span. These layers usually have more parameters to be learnt than the previous layers. The reason we call them $3D$ is that other than images for each frame, there is another axis called time containing discrete values, and each of them corresponds to a particular frame.


Face recognition - How to make an image classifier with large number of classes? I am planning to make an image classifier that identifies the face of every player in the English Premier League. I have a couple of questions (since until now I have only worked with small or academic datasets).
My questions:

How do I download this many different images? Since it's pretty hard to manually download the pictures individually, is there a way to automate it?
I'm following this platform and am required to make a different class for each player. How should I go about making different classes for so many players?

PS: I am a newbie, so I apologize if my questions are vague. Please comment if any additional clarification is needed.

How do I download these many different images? Since it's pretty hard to manually download the pictures of all, is there a way to automate it?

First of all, you need to get an idea of how your classifier will be, will it classify variable size images or fixed size images? Will it receive only one face as an input or more than one? Will it classify black and white images or with more than one channel? Will it be a multioutput classifier or a single output classifier?

Before starting you have to answer this and other questions. Based on the later questions you have to get your data from a different source or in a different way. Instead of thinking these questions you can search for a dataset that already has the images collected and organized and adapt to the dataset, Kaggle has lots of public datasets. Collecting the training data by your own can be tedious work, and more if you need thousands of samples, I recommend you to make a data mining application the crawls through different pages and automatically downloads the images, although you have to be careful not to violate any privacy laws.

  I am using Fast.ai and we are required to make a different class for each player. How should I go about making different classes for so many players?

Now that you have all the data clean and tidy you have to classify the images. There are lots of ways to classify images using deep neural networks, you could use simple feedforward neural networks or recurrent neural networks, but the most commonly used and the one I recommend you are convolutional neural networks. If you do not have a lot of experience classifying images I recommend you to start with an easier project such as the MNIST dataset, which consists of black an white images of handwritten digits, this way you can practice with the different hyperparameters and experiment better, rather than starting with such an ambitious classification task. Either way, to classify the images into multiple outputs you will have to add a final layer to your neural network with a number of hidden neurons equal to the number of classes to distinguish, then you should add a softmax activation function to that layer. Finally, for the model to learn you should use a categorial cross entropy loss/cost function.

I don't know if I answered your question or not at all so please don't hesitate into telling me or asking more questions.

Either way, glad to help


Backpropagation and gradient descent I just want to clear one doubt - we use gradient descent to optimize the weights and biases of the neural network, and we use backpropagation for the step that requires calculating partial derivatives of the loss function, or am I misinterpreting something?

Yes you are correct. Gradient descent (or various flavors of it) is the mechanism by which you find a local minima of your loss space using some learning rate. Backpropagation calculates the gradient of the error function with respect to the NN's weights and biases.


Proceeding with various methods for news recommendation I am beginner in ML (i have done only Andrew Ng's ML course) and i have to work on news recommendation.

I went through this paper which mentions different methods used for news recommendation (at 7th page) and most of them are using some sort of probabilistic methods (bayesian networks, latent dirichlet allocation, naive bayes model, probabilistic matrix factorization models). Also, some news recommenders  are based on multi-armed-bandit problem (e.g. yahoo's front page). I have zero knowledge about these methods.

I am very confused regarding what should be my next step. Right now, i am planning to go ahead with a very basic hybrid approach (with collaborative filtering and content based filtering). But it seems like i will have to explore these fields very soon, so what will be a good strategy (and resource) to explore these fields?

Recommender Systems are a huge topic of its own right and goes without saying, with a lot of research going on.

This book does a deep-dive into recommender systems and may not be something you want, but it's helpful as a reference. It seems like you were unsure what those terms mean. The Berekely AI Course covers most of these topics and their lectures are available for free.

You might be able to make better choices after going through some of the above materials.


XGBClassifier and RandomForestClassifier variable importance plots are very different I have one dataset, and decided to use XGBClassifier to get a variable importance plot from it.

I was a little surprised that some features that I assumed were insightful had no appreciable value.  With this in mind, I used an sklearn RandomForestClassifier to create another variable importance plot.

To my surprise the two were very different! Many variables that appear on one don't appear on the other.

I used as similar parameters as I could, between the two models.

Here's what they look like to give a general idea:

Has anyone had this similar experience, and if so, what kinds of reasoning for this might there be? It's a little frightening to think that many Kagglers and others really depend on these plots and to see that they can vary so much...makes me wonder.

Thanks.

There are multiple reasons why this could happen. The first, and probably the most important one is that while both of the algorithms are tree based ensemble methods, they are very different from each other. Second of all, these feature importances are just some heuristic that do some calculations on where the features are used. I assume these heuristics are very different for both the algorithms

In this case however the differences might just be too big to only point to these two reasons. What could have happened is that your features are highly correlated. If two columns are highly correlated then the algorithms become kind of indifferent to which one to use, which means the two algorithms might both pick a different one each time they come across this choice. You could take a look if column 0 and column 7 are very correlated.

Another option is that some of your columns are categorical and that the implementations deal with this in a different way. Python doesn't work cleanly with categorical values in Scikit-learn. One-hot encoding is not a nice representation for undeep trees but if you believe this may be causing it you could preprocess them like that to see if the feature importances become more similar.


How to impute using simple imputer (custom function) I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to

remove nans and replace with mode
replace +infs with max and -infs with min
use standard scaler.

then for using xgboost i would like to:

simply replace -infs/+infs with very large or -ve large numbers.

i have been playing with sklearn pipeline and i would like to know how i can pass the custom imputers through the pipeline? e.g:

logistic_pipeline = Pipeline( steps = [('imputer', SimpleImputer(strategy = 'most frequent') ),
                                  ( 'std_scaler', StandardScaler() ),
                        ( 'model', LinearRegression() )] )

but how do i incorprate the following function into it where i am replacing infs from the training datase (df) with the max of that column . then using this max to populate it into the test.. how can i do this using pipeline?

def replace_pos_inf(df, dftest, numeric_features):

    for col in df[numeric_features].columns:
        m = df.loc[df[col] != np.inf, col].max()
        df[col].replace(np.inf,m,inplace=True)
        dftest[col].replace(np.inf,m,inplace=True)

    for col in df[numeric_features].columns:
        mini = df.loc[df[col] != -np.inf, col].min()
        df[col].replace(-np.inf,mini,inplace=True)
        dftest[col].replace(-np.inf,mini,inplace=True)

    return df,dftest

Since you want to save the training min/max and use those to replace inf's in the test set, you need a custom transformer.  To build a robust transformer, you should use some of sklearn's validation functions.  And it's best to work in numpy, since as you point out an earlier transformer in a pipeline will have already converted an input dataframe to an nparray.  (You could stick with dataframes, and either convert in your transformer (losing some efficiency), or make sure your transformer always comes first in a pipeline (alright if you're the only one using your code).)

Then, here's a simple version.  I'm opting not to have the numeric_features as input; you're probably better off using a ColumnTransformer to do that kind of selection for you.

from sklearn.utils import check_array
from sklearn.utils.validation import check_is_fitted
from sklearn.base import TransformerMixin, BaseEstimator

class ReplaceInf(TransformerMixin, BaseEstimator):
    '''Replace +-np.inf with the max/min finite values in each column.
    Attributes
    ----------
    mins_ : np.ndarray
        Per-column minimum finite values.
    maxs_ : np.ndarray
        Per-column maximum finite values.
    '''

    def fit(self, X, y=None):
        # validate and convert if possible:
        X = check_array(X, force_all_finite=False)
        # using `where=np.isfinite(X)`, nan's won't affect the min/max calculation,
        #  and using `clip` to transform will preserve nan's as well.
        self.mins_ = np.amin(X, axis=0, where=np.isfinite(X), initial=np.inf)
        self.maxs_ = np.amax(X, axis=0, where=np.isfinite(X), initial=-np.inf)
        return self

    def transform(self, X):
        X = check_array(X, force_all_finite=False)
        return np.clip(X, self.mins_, self.maxs_)


Binary classfication vs One-class classification Why do we need samples of both classes for the training of binary classification algorithms, if one-class algorithms can do the job with only samples from one class?

I know that one-class algorithms (like one-class svm) were proposed with the absence of negative data in mind and that they seek to find decision boundaries that separate positive samples (A) from negative ones (Not A).

Hence the traditional binary classification problem (between (A) and (B) for example) can be formulated as a classification of (A) and (not A = B).
Is it about better classification results or am I missing something?
Thank you in advance

Binary classification is needed when requirement is to capture data into two classes. if you can't capture data in two classes where you need only one you go for One-class.
you can check this link for better explanation
http://rvlasveld.github.io/blog/2013/07/12/introduction-to-one-class-support-vector-machines/

If you take binary classification, svm tries to find best possible space between A and B. If there is only one class A model tries to create a boundary around it and classify. Take for example patient disease classification: For +ve some symptoms t1, t2, t3, t4, t5 for -ve he has t1, t2, t7. in the above case it is difficult to classify using one class because model classifies patient having t1, t2 as +ve because of proximity to +ve class. The second label gives you more info for better classification.


How to apply MultiOutputClassifier to a dataset for Naive-Bayes algorithm I have a dataset which is as follows, (it's taken from an article online and I have been trying to Naive Bayesian algorithm on it)
Original Dataset
y attribute
After having done some manipulations (following the article), these are my new datasets for training and testing,
X Train
y Train
Now, it contains a multilabel and I have been asked to look at Multioutput classification for the problem. I have been trying to understand this classification and tried to implement it myself too, but I couldn't get it to done. First of all, I tried following this sample code given on the website,
from sklearn.datasets import make_classification
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.utils import shuffle
import numpy as np
X, y1 = make_classification(n_samples=10, n_features=100, n_informative=30, n_classes=3, random_state=1)
y2 = shuffle(y1, random_state=1)
y3 = shuffle(y1, random_state=2)
Y = np.vstack((y1, y2, y3)).T
n_samples, n_features = X.shape # 10,100
n_outputs = Y.shape[1] # 3
n_classes = 3
forest = RandomForestClassifier(n_estimators=100, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
multi_target_forest.fit(X, Y).predict(X)

But, since, I am new to all this, I didn't understand anything at all.. I didn't understand why he did the make_classification call, and then shuffled the data and etc.
I tried to implement it on my y_train variable and then placed it in my model.fit for Naive-Baysen algorithm,
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
Yt = np.vstack(y_train).T
n_samples, n_features = X_train.shape # 10,100
n_outputs = Yt.shape[1] # 3
n_classes = 3
forest = RandomForestClassifier(n_estimators=100, random_state=1)
multi_target_forest = MultiOutputClassifier(forest, n_jobs=-1)
model.fit(X_train, multi_target_forest)

But it gave the same error which I was receiving previously, which meant that I didn't do the multioutputclassification properly,
ValueError: y should be a 1d array, got an array of shape () instead.

Can anyone help me in telling how to actually implement this classification, so that the Y variable can be used for the Naive Baysen?

The examples's make_classification, shuffles, and stack are just to generate a multilabel dataset. You've already processed to y_train and don't need any of that.
Then, you want a base estimator to be Naive Bayes, not a random forest as in the example.  And all sklearn models should be fit as model_instance.fit(X, y, sometimes_optional_parameters); the error being raised is because you passed a model object instead of y (and also are using the wrong model object).  So the following should do what you want:
nb = GaussianNB()
multi_target_nb = MultiOutputClassifier(nb, n_jobs=-1)
multi_target_nb.fit(X_train, y_train)


how to see decision tree when running in anaconda? In Jupyter displaying the DT is done as follows:

# Display in jupyter notebook
from IPython.display import Image
Image(filename = 'tree.png')

How to see DT in anaconda?

there is a simpler way but sometimes it doesn't work
 first, you should call your method like this

tree_clf=tree.DecisionTree()

then you can choose a name and path for it, let others remain the same as I did
so overall, it would look like:

from sklearn.tree import export_graphviz
export_graphviz(
tree_clf,
out_file=image_path("iris_tree.dot"),
feature_names=iris.feature_names[2:],
class_names=iris.target_names,
rounded=True,
filled=True
)

hope it works for you


Steps on how to use autoencoders to reduce dimensions I have a dataset that contains text columns. I have used tf-idf to convert those text columns to numerical columns. I want to reduce the dimension of the dataset since tf-idf creates a multitude of new features/columns.

I am thinking of using autoencoder to reduce the dimension of the data by using the output of the encoded layer, concatenate those output to my dataframe and use it as a training set for the random forest.

My questions are: Do the above steps sound reasonable? In particular, should I train the autoencoder on the whole dataset and then use the output of the encoded layer (after training) as the new columns to feed random forest? [See code below]

I am using Keras so after defining the autoencoder model, I fit the whole dataset and then also predict on the whole dataset using only the encoded layer. And then using those predictions as my updated data.

Here is the code I am using:

def autoencoder(df):
    #return the lower dimensional data to feed directly into train_test_split
    df_copy = df.copy()
    input_dim = Input(shape = (df_copy.shape[1], ))

    # DEFINE THE ENCODER LAYER
    encoded = Dense(int(df_copy.shape[1]/2), activation = 'relu')(input_dim)

    # DEFINE THE DECODER LAYER
    decoded = Dense(df_copy.shape[1], activation = 'sigmoid')(encoded)

    # COMBINE ENCODER AND DECODER INTO AN AUTOENCODER MODEL
    autoencoder = Model(input = input_dim, output = decoded)

    # CONFIGURE AND TRAIN THE AUTOENCODER
    autoencoder.compile(optimizer = 'adadelta', loss = 'mean_squared_error')
    autoencoder.fit(df_copy.values, df_copy.values, nb_epoch = 10, batch_size = int(df_copy.shape[1]/3))

    # THE ENCODER TO EXTRACT THE REDUCED DIMENSION FROM THE ABOVE AUTOENCODER
    encoder = Model(input = input_dim, output = encoded)
    encoded_input = Input(shape = (int(df_copy.shape[1]/2), ))
    encoded_out = encoder.predict(df_copy.values) #Note how I am training on the same data I am predicting on

    return pd.DataFrame(encoded_out)

I am new to autoencoders so any suggestions and help on how to use autoencoders in this context will be appreciated.

I will repost my answer on Quora here, about the "should I train the autoencoder on the whole dataset" part of the questions.

TL;DR: NO, you always use ONLY the training set for training, even for unsupervised learning with autoencoders!

What is the goal of using a test set ? It is meant to represent data that you model (including both the autoencoder and the random forest) have NEVER seen before.

Autoencoders are unsupervised learning models, but it does not mean that they cannot overfit. They can learn latent codes that do not generalize. Imagine an AE that encodes every data point x1, x2, … as a real number (one-dimensional code):

x1 -> 0.0001, x2 -> 0.0002, etc.

Do you think this autoencoder will generalize well ? No, it just remembered the entire training set and did not learn useful representations. That is why we use regularization in autoencoders, in very different forms : undercompleteness, sparsity, denoising, variational autoencoders (VAE)…

Conclusion, if you include the test set to train your autoencoder, it may in certain cases overfit, your performance estimates will be biased and you lose all the purpose of a test set.

(don’t hesitate to ask if you want me to detail some aspects of my answer or have more questions on this matter)

Concerning the architecture of you AE:

using a sigmoid output activation only makes sense if you data takes values between 0 and 1, as a sigmoid function squashes the output into the [0, 1] range. So either you normalize your data to the [0, 1] range, or you use a linear output activation.
With a sigmoid output activation, you should use a binary cross-entropy loss, which is designed for binary variables following Bernoulli distributions. With a linear output, use the mean squared error loss, which is designed for real-valued output (following Gaussian distributions).
Maybe Adam optimizer will converge faster that Adadelta. Try for yourself.


Predict quantitative response variable from quantitative and qualitative variables I have to predict a quantitative response variable Y from p quantitative and q qualitative variables. There's around 100,000 rows in the data frame. What predictive methods can I use for that purpose?

You should use one of the regression methods for predicting a quantitative variable. For the qualitative features, you can binarize them to pass to the learning algorithm. If you use Python, scikit-learn provide classes for doing that in the preprocessing module.


Which time series analysis is appropriate for pooled time series data analysis? I have two data sets one is cross sectional census data with 10 years interval and another one is time series data  (monthly) for several years. Now I want to perform statistical time series analysis on this both dataset in a single model. Is it possible?

It really depends on the outcome you want from your analysis.

The most straightforward approaches might be resampling:

Interpolate the 10-year census data to month granularity. The interpolation might be linear, polynomial, etc. This effectively imposes a strong assumption in the dynamics of the data. This may make sense for some variables but not for others. And you are probably introducing a lot of variance in the obtained conclusions.
Aggregate the monthly time series into 10-year granularity. This may not be very useful if you want to predict something with an horizon less than 10 years...


Average Precision if Target Class is Not in Evaluation Suppose I have 5 classes, denoted by 1, 2, 3, 4, and 5, and this is used in object detection.
When evaluating an object detection performance, suppose I have classes 1, 2, and 3 present, but classes 4 and 5 are not present in the targeted values.
Will each of classes 4 and 5 have average precision of 0 (due to its precision being zero as no true positives can be identified)? Or perhaps there are other considerations to take when this case occurs?

I'm not sure if there is a standard for this kind of case. I think it depends whether the system predicts any of these classes, i.e. whether there are any false positive cases:

if yes, then the precision must be zero indeed: out of all the instances predicted positive none was a true positive.
if no, I think it makes more sense to consider the precision as undefined (NaN), since no instance is predicted positive at all (that is, there is a division by zero in the calculation of precision).

This is a very borderline case, since it would be very questionable to use a test set which doesn't contain all the classes contained in the training set: such a test set cannot is not fit for the purpose of evaluation.


What is "Policy Collapse" and what are the causes? I saw the term "policy collapse" on the comments of a tutorial for  reinforcement learning.

I'm guessing that it's referred to as a policy collapse when the policy worsens over training due to a bad hyper-parameter, be it the learning rate, batch size, etc., but I couldn't find anything explaining it in clearly and in detail.

A web search for "policy collapse" "reinforcement learning" finds this question, a related one in stats.stackexchange.com and the comments section where you found the phrase. There are two other results on unrelated subjects where the words happen to appear next to each other. Then that's it - 5 results total from Google.

A google books ngrams search for policy collapse finds no references at all.

It is hard to prove a negative, but I think this is not a widely used term.

However, the comment does appear to be referring to a real phenomenon. That is where a reinforcement agent, instead of converging on the value functions for an optimal policy as it gains experience, actually diverges (and the parameters of the approximator will diverge too).

This can happen when using non-linear function approximators to estimate action-values. More generally, it tends to happen when you have the following traits in your problem:

Using a function approximator, especially a non-linear one (although even linear function approximators can diverge)
Using a bootstrap method, e.g. Temporal Difference (TD) Learning (including SARSA and Q-learning), where values are updated from the same value estimator applied to successive steps.
Off-policy training. Attempting to learn the optimal policy whilst not behaving optimally (as in Q-Learning).

In Sutton and Barto's book this is called the "deadly triad". If you do a web search for "deadly triad" "reinforcement learning" you will find many more results. It is an ongoing area of research how best to combat the effect. In the paper that introduced the DQN model learning to play Atari games, the researchers applied two things that help stabilise against the effect:

Experience replay, where transitions are not learned from immediately, but put into a pool from which mini-batches are sampled to train the approximator.
Bootstrap estimates are made from a "frozen" copy of the learning network, updated every N training steps - i.e. when calculating the TD target $R + \gamma \hat{q}(S', A', \theta)$, use this old copy of the network.

From the comment section you linked, it appears even applying these things is not a guaranteed fix and takes some judgement. In that case it was increasing the mini-batch size for experience replay that helped to stabilise an agent playing a variant of the video game Pong.


Could Deep Learning be used to crack encryption? Say you have a dataset with millions of rows and the attributes Plain Text, Key, and Output Ciphertext. Could Deep Learning, theoretically, be used to find patterns in the outputs that help decipher the ciphertext? Are there any other potential approaches?

Probably not. Modern encryption systems are designed around cryptographic random number generators, their output is designed to be statistically indistinguishable from true randomness. Machine learning is generally based on discovering statistical patterns in the data, and with truly random data there is none. Even for flawed crypto where there is some small pattern to be found, the large amount of randomness in the input will overwhelm any direct attempt to decrypt the ciphertext.

In addition, there is no heuristic you can use to tell if you are getting close to a correct decryption - a single bit out in a guess at a key for example will completely scramble the output (blame Hollywood for when it shows decryption on screen like some crossword puzzle where the correct letters drop into place). That all-or-nothingness rules out discovering algorithms via a machine learning process, even when you have the encryption key. The best you can do is brute-force all known algorithms. If you don't have the key, then you have to brute-force all possible keys too.

You could explore how difficult the problem is by attempting to guess the seed value used for a random number generator. Using the Mersenne Twister RNG (the standard one used in e.g. Python), then the input could be the bit pattern for 624 32-bit unsigned integers, and the output could be the 32 bits of the seed used to generate that series. The reason I suggest those specific numbers is because it is in fact possible to crack Mersenne Twister with that much data. However, I still think that ML approaches would be entirely the wrong tool to do so.

Another simple variant would be to see if you can teach a network to either produce or reverse a cryptographic hash. You could start with a known broken one such as MD5. Input and output could be 80 bits, which simplifies the architecture and pipeline enough that you could put together this test in a few hours. Even though MD5 is known to be compromised, I think there is zero chance you could teach a neural network to find any pattern.

One important detail: If you want to "crack" an encryption, you will not be able to use the key as a known value. Chances are though, that even if you provide the key to an ML process, it will be unable to learn how to decrypt.


Deep Reinforcement Learning - mean Q as an evaluation metric I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between the player and 6 closest enemy lasers normalized by the window height (if the player's position is $(x_p,y_p)$ and an enemy's position is $(x_e,y_e)$, the relative euclidian distance is
$\frac{\sqrt{(x_p-x_e)^2+(y_p-y_e)^2}}{HEIGHT}$ and HEIGHT is the window height). Hence the observation space dimension is (10+6), which results in an input of my deep neural network of 16 units.

My agent doesn't seem to learn (reward function doesn't increase) and I thought I'd check the mean Q values, which are the output of my main deep neural network, and, instead of increasing, I've remarked that the mean Q values stabilizes (as in figure below) instead of increasing. I've modified many tuning parameters (batch size, neural net architecture and parameters...) but I still have same problem. Any idea why the mean Q values wouldn't increase ?

Here are some results about the learner:

I think your main problem is use of relative distance as the core feature. It has two major weaknesses:

The distance to an object does not give the direction to the object. The best action choices are all critically dependent on direction. For example an enemy laser bolt 0.1 units directly above the player is an immediate danger requiring evasive action, whilst one 0.1 units to the left or right is not a danger and about to leave the game window. Your feature of relative distance does not distinguish between those scenarios, but it is a critical difference.

Slightly less important, but the raw distance does not capture any sense of movement. If enemies move consistently turn by turn, but not always in the exact same direction or same speed, then their velocities should also be part of the state.

One way you could improve the features is to add a velocity component for each item, showing how quickly it is approaching or receding from the player. This might help a little, but my feeling is that you need more data than distance and speed.
I think you should use normalised $x, y$ position as features for each item being tracked, plus normalised velocity $dx, dy$ for any object type that can change direction (if enemy lasers are always falling straight down you might not need anything for those).
In addition:

If the window edges are important, you should include at least the relative $x$ of one of them, so the agent knows its absolute position on screen and how much space it has to maneuver. This is true whether the player is blocked from moving further left or right, or whether the player "wraps around" to the other side of the screen. Both types of effect will significantly affect how the game plays near the screen edge.

In order to track predicted value, you need to track location of player missiles. It is not enough to just let the agent predict when it is best to fire - in order to accurately track a value function it needs to "see" whether the missile it fired some time steps ago is likely to hit or miss a target.

For both enemy lasers and player missiles, it is OK to filter and sort the data by some criteria (such as distance to player). As long as this is consistent it may even help a lot to have such pre-processing.


How to correct mislabeled data in dataset? I have a dataset of about 300k records. Classes are highly imbalanced (which means that one may have 30k records, and the other may have only 100). Unfortunately, about 5% of records is incorrectly labeled.

Is there any way of finding out which elements are wrong, so I would be able to discard them?

Yes! This could be an excellent test case for your classification algorithm. With only 5% mislabeling a good algorithm will be easily able to identify "outliers" by having much worse predictions for these mislabeled records.

If you were able to at least identify "correct" records to generate a training set that would be even better but with 5% mislabeling even if not it won't be a problem. This also leads to the second part of the answer, that while it might be better to remove or correct the mislabeled records it might also not matter.

This is obviously based on the assumption that the 5% errors are somewhat randomly distributed over all classes.

Finally, you did not mention any hints/data/info that could identify mislabels. Obviously if you have information about those errors doing some pre-processing to identify and remove them based on analysis / rules generation would be best.


Update a pandas data frame column using Apply,Lambda and Group by Functions I have a data frame in the format mentioned in the screenshot below. Column 'Candidate Won' has only 'loss' as the column value for all the rows. I want to update the Column 'Candidate Won' to a value 'won' if the corresponding row's '% of Votes' is maximum when grouped by 'Constituency' Column otherwise the value should be 'loss'. I want to achieve the result by using a combination of apply, lambda, and group by, instead of using loops/iterations.

DataFrame : (df_andhrapradesh)

Code below works for a specific constituency in the data frame :

df_amalapuram=df_andhrapradesh[df_andhrapradesh['Constituency']=='Amalapuram']
df_amalapuram['Candidate Won']=df_amalapuram['% of Votes'].apply(lambda x:"Won" if x==df_amalapuram['% of Votes'].max() else "Loss")

Tried something like below to make it work for the entire data frame which has different constituencies but it failed:

df_andhrapradesh['Candidate Won']=df_andhrapradesh['% of Votes'].apply(lambda x:"Won" if x==df_andhrapradesh.groupby('Constituency')['% of Votes'].max() else "Loss")

I used 'Apply' function to every row in the pandas data frame and created a custom function to return the value for the 'Candidate Won' Column using data frame,row-level 'Constituency','% of Votes'

Custom Function Code:

def update_candidateresult(df,a,b):
max_voteshare=df.groupby(df['Constituency']==a)['% of Votes'].max()[True]
if b==max_voteshare:
    return "won"
else:
    return "loss"

Final Code :

 df_andhrapradesh['Candidate Won']=df_andhrapradesh.apply(lambda row:update_candidateresult(df_andhrapradesh,row['Constituency'],row['% of Votes']),axis=1)


a simple way to test wether a tree-based classifier would transfer well to a target population? I trained a tree-ensemble classifier (XGBOOST) on population A, validated it and I'm satisfied with its accuracy (AUC 0.78). Now I'm trying to transfer it to a slightly different population B, and there the accuracy of the model deteriorates badly (AUC 0.68)

I tried isolating which of the features did not transfer well, both by simple univariate analysis (comparing distributions), and by comparing each feature correlation with the label, and couldn't find anything obvious.

Is there a way to debug and understand which of the model assumptions which held at A do not hold at B? I thought about comparing the label distributions at every node in every tree for the validation populations in A and B, thereby testing all conditional probabilities the model assumes are holding actually hold at B.

Would that help me understand what broke? or would I just get tons of tiny differences? Is there some other simple way I'm missing?

(related to this survey Machine learning learn to work well on future data distribution?)

Not sure if you've tried this already, but you might dig into XGBOOST feature importance to establish how your model is making predictions, then do a more in-depth comparison between the two populations at those splits.

This isn't very different from what you've proposed, but it does make for a more focused analysis; you might have time to look at higher order interactions between the more important features. (I'm assuming it's not possible to cheat by training a classifier on your population B and comparing   how each model makes predictions!)

ELI5 and LIME also come to mind for debugging feature importance and explaining model prediction.


Correcting Datasets with artificially low starting values I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/normalization are not that import to my question).

In most cases measurement A (the starting measurement) is a count with values greater than 1000.  We expect an increase for measurement B for positive effects and a decrease in measurement B for negative effects.

Here is the issue, some of our starting counts are nearly zero which we believe is an artifact of experimental preparation.  This of course leads to some really high ratios/scaling issues for these data points.

What is the best way to adjust these values in order to better understand the real role in our experiment?

One suggestion we received was to add 1000 to all counts (from measurement A and B) to scale the values and remove the bias of such a low starting count,  is this a viable option?  Thank you in advance for your assistance, let me know if I am not being clear enough.

Yes, the general idea is to add a baseline small count to every category. The technical term for this is Laplace smoothing. Really it's not so much of a hack, as encoding the idea that you think there is some (uniform?) prior distribution of the events occurring.


NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need) I am new to NLP and I just finished reading the paper "Attention is all you need".
I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these visualizations were produced:

I understand that the output of the self-attention sub-layer (for a single head) is a vector of size d_v that is a weighted sum of all the value vectors. Than how do they use this vector to calculate the strengths of the relations between the positions?
Any help and insight would be appreciated, thanks a lot!

So the question is concerned about understanding the self-attention mechanism in greater detail, in particular how this idea of multi-head self-attention is used to compute strength of relations between tokens.
I think it's best you look through this great tutorial on self-attention and see if this helps in your understanding of multi-head self-attention: http://www.peterbloem.nl/blog/transformers


Comparing data sets with different measurements I'm currently writing a thesis based on Cyber Crime, however I'm unsure of the proper to compare/analyse my data sets to talk about them in my thesis.

One piece of data (https://www.pandasecurity.com/mediacenter/src/uploads/2014/07/Pandalabs-2015-anual-EN.pdf on page 9) it states that the 'infection rates' of Sweden is 20.88% (bottom 3 ranking), USA at 29.48% (middle ranking) and China (first rank) having 57.24%.

Another (http://www.virusradar.com/en/home/world) , uses a different measurement to define the 'threat rates', which is different to the one above, which has Sweden at 2.13%, US at 2.87%, and China at 15.17%

Another piece of data (https://www.symantec.com/content/dam/symantec/docs/reports/istr-22-2017-en.pdf
 on page 50) states 'identity theft' in Sweden is 6 million(low ranking), US 791 million (high ranking), and China 11 million (middle ranking)

I'm unsure how to compare these, because by just looking with our eyes we can see Sweden is the lowest in all these statistics which I can use to further discuss my argument. However, these are clearly different measurements (one is overall infections, one is just identity theft, etc), and I have much more similar data which provides numbers/percentages of certain types of cyber crime by country.

My goal is to analyse the results so I can compare them to further the goals of the thesis about where cyber crime is most/least prominent and to create visualisations of this.

So would just simply ranking them be fine? (Sweden 3 because it was lowest all times?, USA 2 because it was second twice?, China 1 because it was first twice?) but that sounds very incorrect. Should I convert all my numeric data sets (like the identity theft one) to percentages then compare them all by percentage to rank/discuss accordingly? And compare them then even though they're different measurements of the same thing(country)?

Thanks for any advice.

You're right, you cannot directly compare the scores, since they are extracted from different samples, using different metrics and parameters, etc. Is exactly as comparing apples with airplanes.

But is not so wrong to play around with the placement in the rank: you are comparing placements with placements, so it makes perfectly sense. If Sweden is always at the last place, this definitely means something.

Maybe you can also think of weight those placements by some parameter/set of parameters which takes in account the subject of the ranks.
As example, seriousness of the threats (ex. infection by AdWare can be less important than identity theft, etc), coverage of the population by the authority which published the results, reputation of the authority, etc.

You can also extract uncertainties on those ranks, maybe based on the statistical samples taken in account by the authorities, and so on.

Cheers


Encode multi-class response variable In a classification problem when the response variable has multi-class, e.g., "sunny","rainy","cloudy", how should we encode it? I know that for predictors like this, usually we do One Hot Encoding, and if a predictors have too many classes, then we might just use the LabelEncode.

When this multi-class issue occurs in the response variable, I guess we can just use LabelEncode() rather use using One Hot encoding right? Because if we use One Hot encoding, then we will have 2 variables as the response variable, and the machine learning algorithm in sklearn usually expects the response variable not to be a vector right? (I mean it expects a long 1D vector with length equal to the number of observations but not a 2D matrix). But on the other hand, if we just map "sunny", "rainy","cloudy" to {1,2,3} or {0,1,2}, or whatever 3 numbers, that will create a less than or greater than relationship among "sunny", "rainy","cloudy", which is not inherited in the original problem.

It depends on the meaning of the classes, and whether they have any meaningful order.
If they are ordinal or a scale, then there is a meaningful ordering, and it can potentially be reasonable to order them and then assign labels $1, 2, \dots, n$ in order to the classes.  This is what you call "LabelEncode".  In some cases, some form of regression might also be appropriate and worth trying.
If they are categorical or nominal, with no meaningful ordering, then LabelEncode makes no sense and will be no better than a one-hot encoding -- and potentially worse.  So, the default is one-hot encoding.
If they are categorical but have some hierarchy, there are smarter methods.  You can build a tree that represents the hierarchy, then have a multi-hot encoding.  If there are $n$ classes, you will have a tree with $n$ leaves and (say) $m$ nodes in total  (where $m > n$).  Then you can encode the class as a $m$-vector, whose $i$th entry is 1 if the class is a descendant of node $i$ in the tree, or 0 otherwise.
If the classes are categorical but have various attributes, you can try an encoding that has one element for each possible attribute.  For instance, if there are $k$ binary attributes, you can have a $k$-vector with a 0 or 1 in the $i$th entry according to whether the corresponding class has the $i$th attribute or not.  You can generalize from there for more general situations.
See this and this for more about different types of measurements.


predict_proba to print specific class probablity I am having 16 labels and predict_proba is giving me probablities of all 16 categories in an array. Is there any way if I pass specific label to predict_proba it can only print me probablity of that category?

preds = model.predict(dataframe)
# getting predicted class , am interested in knowing probablity of this class.
print(preds)
# it is printing array for all 16 labels, I am keen to pass above predicted class and retrieve probability for it.
print(model.predict_proba(dataframe))

Output
[[0.07387347 0.007413   0.00354506 0.02321654 0.09627853 0.00958647
  0.00599333 0.02232621 0.12513558 0.00494633 0.07230524 0.00384056
  0.00378245 0.44431455 0.04089799 0.0625447 ]]

now model had predicted class "XYZ" how to get its index in predict_proba function, if I have that I can simply use (i being index)
print(model.predict_proba(dataframe)[0][i])

Once you fit your sklearn classifier, it will generally have a classes_ attribute. This attribute contains your class labels (as strings). So you could do something as follows:

probas = model.predict_proba(dataframe)
classes = model.classes_

for class_name, proba in zip(classes, probas):
    print(f"{class_name}: {proba}")

And to find a specific index, you can use numpy's where function:

import numpy as np

class_label = "XYZ"
class_index = np.where(model.classes_ == class_label)

proba = model.predict_proba(dataframe)[class_index]


XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used with scikit's RF regressor and xgboost's Regressor. Specifically the part about tuning for the number of trees/iterations/boosting rounds. From my understanding each of those terms are the same. They determine how many times a decision tree should be calculated based on the algorithm. However, should I be referring to them as ntrees or n_estimators? Or should I simply use early stopping rounds for my xgboost and tune the number of trees only for my rf?

My Random Forest:

rf = RandomForestRegressor(random_state = 13)
param_grid = dict(model__n_estimators = [250,500,750,1000,1500,2000],
                  model__max_depth = [5,7,10,12,15,20,25],
                  model__min_samples_split= [2,5,10],
                  model__min_samples_leaf= [1,3,5]
                  )
gs = GridSearchCV(rf
                  ,param_grid = param_grid
                  ,scoring = 'neg_mean_squared_error'
                  ,n_jobs = -1
                  ,cv = 5
                  ,refit = 'neg_mean_squared_error'
                  )

My xgboost

model = XGBRegressor(random_state = 13)
param_grid = dict(model__ntrees = [500,750,1000,1500,2000],
                  model__max_depth = [1,3,5,7,10],
                  model__learning_rate= [0.01,0.025,0.05,0.1,0.15,0.2],
                  model__min_child_weight= [1,3,5,7,10],
                  model__colsample_bytree=[0.80,1]
                  )
gs = GridSearchCV(model
                  ,param_grid = param_grid
                  ,scoring = 'neg_mean_squared_error'
                  ,n_jobs = -1
                  ,cv = 5
                  ,refit = 'neg_mean_squared_error'
                  )

As I understand it, iterations is equivalent to boosting rounds.

However, number of trees is not necessarily equivalent to the above, as xgboost has a parameter called num_parallel_tree which allows the user to create multiple trees per iteration (i.e. think of it as boosted random forest).

As an example, if the user set num_parallel_tree = 3 for 500 iterations, then number of trees = 1500 (=3*500) rather than 500.


CV(Curriculum vitae) Recommendation System guidance I am building a recommender system which matches people's CV with a vacancy.

So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV.

I want to know whether there are any other approaches to create such a recommendation engine?

Try word embeddings and unsupervised learning, for example:

  medium

main idea is that similiar users will be in similiar clusters


Generic strategy for object detection I have a huge collection of objects from which only a tiny fraction are in a class of interest. The collection is initially unlabelled, but labels can be added using an expensive operation (for example, by human).

Currently I use the simple generic machine learning strategy:

Use hand-crafted rules to select a smaller subset of objects (thus leaving out a fraction of interesting ones).
Label part of the smaller subset, and use these for training and choosing a classification algorithm and its parameters.
Classify the remaining objects in the smaller set (and also perhaps in the big set).

This has two drawbacks:

The labeller still needs to see a huge number of uninteresting objects, and therefore is able to label only a very small fraction of interesting ones.
The objects not in the smaller set are completely ignored in the learning phase, resulting in a loss of some information (the classification algorithm might not work well on this complement).

It seems that it would be better to use online learning: i.e., select the objects to show to the labeller based on the previous labels. But then it becomes no longer obvious that the result of classification algorithm retains the nice theoretical properties (i.e., statistical consistency).

Is there a general framework for active object detection which works either theoretically or practically (or both)? I could not get the complete picture from the Wikipedia article active learning.

The framework you cope with is semi supervised. You have mostly unlabelled data and you can have some labeled data by manual labelling.

Active learning is one method to cope with the situation, by focusing your labelling efforts in the most beneficial areas. You can read a survey on these techniques at Settles, Burr (2010), "Active Learning Literature Survey" (PDF), Computer Sciences Technical Report 1648. University of Wisconsin–Madison, retrieved 2014-11-18

Please note that even if you are focusing your labelling efforts using active learning, it is still a significant constraint.

There are other methods of copying with semi-supervised framework, like co training.The classical reference on co-training is "Blum, A., Mitchell, T. Combining labeled and unlabeled data with co-training. COLT: Proceedings of the Workshop on Computational Learning Theory, Morgan Kaufmann, 1998, p. 92-100"


Matching content item to a persons profile I'm working on an experiment which is essentially a content recommendation service.

I have a set of content items in the form of articles, tweets, blog posts etc which have had a set of tags associated with them based on their contents.

I also have a set of user profiles with information on the users personality, interests, dislikes, activities etc.

Each user profile then has a column for liked and disliked tags to show which content items they would enjoy reading.

I would like to be able build a service that when passed a new user profile, returns a set of tags (generated from similar existing user profiles) that can be used to find content items that the new user would enjoy reading.

This is my first experiment with machine learning so I was wondering if someone could give me some advice on how to achieve this, or an approach to get started.

Many thanks in advanced!
Harry

Look into collaborative filtering and content based filtering.
Essentially, build a user-space - using their personality, etc features. build an item space in a similar manner- using their tags, what not.
Decide on a metric for each space, so you can compare how much each item is similar to each other(same for users).
Then look at each user preferences and say, this user is similar to that user who liked this item who is close-like to that item.
If user A liked item 1 and user B is similar to user A then its probable he will like item 1 as well.

This is a simple enough example


Text Classifier with multiple bag-of-words I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself.  I've pre-processed the content column in such a way that the subject and associated metadata have been completely removed.  My thinking, at this point, is that I should train the classifier on two separate bag-of-words, one for the subject column and one for the content column.  Is this the right approach?  Is it there a simple way to accomplish this using NLTK or a similar library?

I cannot comprehend the question completely but let me see if I got this right.

You want to train a classifier. You have different columns for subject and body. You have now pre-processed it in a manner such that subject and other meta information have been removed.

Now I shall try to answer your question. Keep in mind that, this is how I would approach the problem (based on experience with similar task in the past) but there could be better ways or approaches.

I would not remove the subject column in the first place. I would also not train the classifier on two separate bag-of-words. I would append both, the subject and the content, into one single document and then classify them accordingly with one unit of bag-of-words instead of two.

Why would I do this? There could be times when the subject could have keywords or something that could lead to better classification than the body of the email. The vice versa could also be true. Hence, without having seen the entire dataset, and to be on the safer side, I would also give this approach a shot. Compare both of your approaches and then see where you have better scores of accuracy, precision, recall and other metrics that aids you in your conclusions.

Ways to accomplish this? Yes, NLTK is one way to get the bag of words. However, I would go ahead with scikit-learn package to train my classifier. I would also run LSA/LDA/HDP (Topic Modeling Algorithms) to see what constitutes my data. To do this, I would use Gensim.

http://scikit-learn.org/stable/

https://radimrehurek.com/gensim/

Additional tutorials to help you get started with text classification if you are new to it (that is what it seems like, pardon me if I am wrong). These will also help you in achieving better results to begin with:

https://towardsdatascience.com/machine-learning-nlp-text-classification-using-scikit-learn-python-and-nltk-c52b92a7c73a

http://zacstewart.com/2015/04/28/document-classification-with-scikit-learn.html

Although both the tutorials I mentioned primarily concentrate on binary classification, you can also achieve fine-grained (multi-class) classification in similar manner.


Trouble understanding the partial differentiation used in reinforcement learning I am studying deterministic actor-critic algorithms in reinforcement learning.

I try to give a brief explanation of actor-critic algorithms before jumping into the mathematics. The actor takes in state $s$ and outputs a deterministic action $a$ based on the distribution policy $u$.

The state and action are fed into the critic. The critic sees how good it is to take a particular action from a given state using the action-value function $Q(s,a,w)$.

The critic is then updated via temporal difference (TD) learning and the actor updated in the direction of the critic

Thus it can be seen that the actor's goal is to try and maximise the state action value function $Q(s,a,w)$ by picking the best actions in the given state.

I am having trouble understanding the mathematics behind the updating of the actor.

The below equation gives how the actor is updated.

\begin{equation}
\frac{\partial l}{\partial u} = \frac{\partial Q(s, a, w)}{\partial a} \frac{\partial a}{\partial u}
\end{equation}

What I understand is that we are taking the partial derivative of $l$ with respect to $u$, and we are backpropogating the critic gradient to the actor.

It seems that $l$ is a differentiable function of the variable $a$, but I am confused when it comes to describing what is happening in the equation above as it seems to consist of two functions multiplied together.

Can someone kindly explain what is really happening in the mathematics above?

Your understanding of what's going on seems to be correct, just one little clarification: $u$ should be the model parameters of the deterministic policy $\mu(s,u)$ and not a distribution itself, same as $w$ are the model parameters of $Q(s,a,w)$, but that's probably what you meant (or I might be unfamiliar with the formulation).

Concerning your actual question, the update step implied by $\frac{\partial l}{\partial u}$ is supposed to make the deterministic policy $\mu(s,u)$ to get closer to the optimal $a$, which maximizes $Q(s,a,w)$. As $a = \mu(s,u)$, we have a composite function on our hands

$$ Q(s,a,w) = Q(s, \mu(s, u), w)$$

When updating the actor paramters $u$ such that $Q$ is maximized, we need to make a step in the direction of the gradient of $Q$ with respect to $u$ which, since it is a composite function, is computed using the chain rule

$$ \frac{\partial Q}{\partial u} = \frac{\partial Q}{\partial a}\frac{\partial a}{\partial u}$$

The notation is a bit sloppy, replacing $\mu$ and $a$ and so on, but that also seems to be the case in the literature. So what is going on intuitively consits of two parts:

moving $a$ in the direction of $\frac{\partial Q}{\partial a}$ will increase $Q$, e.g. in 1D if $\frac{\partial Q}{\partial a} > 0$ increasing $a$ would increase $Q$ and if $\frac{\partial Q}{\partial a} < 0$ increasing $a$ would decrease $Q$
moving $u$ in the direction of $\frac{\partial a}{\partial u}$ will increase $a$, in 1D the example would be the same as above

If you multiply these together and update $u$ according to the product you end up moving $u$ such that $Q$ increases by either increasing or decreasing $a$, which is exactly what you want to do.


CNN for ordinal classification How do we design a CNN for ordinal classification?
I am trying to analyze plant leaf images for diseases. I've got the disease type classification working. Now we need to classify the magnitude of the disease affliction on a scale of 1 to 10 (ordinal scale), where 1 is almost no trace of diseased parts on the leaf and 10 is completely diseased.

Is it possible to construct a CNN for this task?

How do I deal with the imbalance in training samples for each ordinal level?

NN aren't ideal for regression tasks.

I'd train 10 networks, one for each magnitude.

CNN #k would be a binary classifier predicting whether the magnitude is k or less

In other words, CNN #10 is a function that always returns 1, and CNN#0 is the binary classifier you've already trained.


SGD vs SGD in mini batches So I recently finished a mini batches algorithm for a library in building in java(artificial neural network lib). I then followed to train my network for an XOR problem in mini batches size of 2 or 3, for both I got worse accuracy to what I got from making it 1(which is basically just SGD). Now I understand that I need to train it on more epochs but I'm not noticing any speed up in runtime which from what I read should happen. Why is this?

Here is my code(Java)

 public void SGD(double[][] inputs,double[][] expected_outputs,int mini_batch_size,int epochs, boolean verbose){
    //Set verbose
    setVerbose(verbose);

    //Create training set
    TrainingSet trainingSet = new TrainingSet(inputs,expected_outputs);

    //Loop through Epochs
    for(int i = 0; i<epochs;i++){
        //Print Progress
        print("\rTrained: " + i + "/" + epochs);

        //Shuffle training set
        trainingSet.shuffle();

        //Create the mini batches
        TrainingSet.Data[][] mini_batches = createMiniBatches(trainingSet,mini_batch_size);

        //Loop through mini batches
        for(int j = 0; j<mini_batches.length;j++){
            update_mini_batch(mini_batches[j]);
        }
    }

    //Print Progress
    print("\rTrained: " + epochs + "/" + epochs);
    print("\nDone!");
}

   private Pair backprop(double[] inputs, double[] target_outputs){
    //Create Expected output column matrix
    Matrix EO = Matrix.fromArray(new double[][]{target_outputs});

    //Forward Propagate inputs
    feedForward(inputs);

    //Get the Errors which is also the Bias Delta
    Matrix[] Errors = calculateError(EO);

    //Weight Delta Matrix
    Matrix[] dCdW = new Matrix[Errors.length];

    //Calculate the Deltas
    //Calculating the first Layers Delta
    dCdW[0] = Matrix.dot(Matrix.transpose(I),Errors[0]);

    //Rest of network
    for (int i = 1; i < Errors.length; i++) {
        dCdW[i] = Matrix.dot(Matrix.transpose(H[i - 1]), Errors[i]);
    }

    return new Pair(dCdW,Errors);
}
private void update_mini_batch(TrainingSet.Data[] mini_batch){
    //Get first deltas
    Pair deltas = backprop(mini_batch[0].input,mini_batch[0].output);

    //Loop through mini batch and sum the deltas
    for(int i = 1; i< mini_batch.length;i++){
        deltas.add(backprop(mini_batch[i].input,mini_batch[i].output));
    }

    //Multiply deltas by the learning rate
    //and divide by the mini batch size to get
    //the mean of the deltas
    deltas.multiply(learningRate/mini_batch.length);

    //Update Weights and Biases
    for(int i= 0; i<W.length;i++){
        W[i].subtract(deltas.dCdW[i]);
        B[i].subtract(deltas.dCdB[i]);
    }
}

My understanding is that mini-batches are not really for speeding up the calculations... but to actually allow large datasets to be calculated.

If you have 1,000,000 examples, it would be tricky for a computer to compute forward and backward passes, but passing batches of 5,000 elements would be more feasible.

For your case, I recommend you two things

Try different batch sizes.
Make sure you shuffle your batches!!! that will certainly help you a bit.


Autokeras for regression I have 2000 samples with 50 sensors signal every millisecond during 600 milliseconds. This means a 2D record for each sample.

I want to predict a continuous value.

I'd like to try autokeras for this regression problem. Could someone point to a tutorial or some advise to start with ?

It's buggy but this is what I'm trying to use:

https://github.com/nocluebutalotofit/autokeras/blob/2efe5358f89c279ee4a7d615bb2bdca3e0ced3cb/docs/templates/tutorial/structured_data_regression.md


Generate sentences from keywords by adding formal word For example, I have a list of key word like : I , hungry => output: I am hungry or I , author , poem => output: I am the author of this poem.
Can someone please suggest the simplest way to achieve this?
I am newbie, please tell me which knowledge I must have to solve this problems.

I am also fairly new to this, but am working on the same type of problem where I have to predict the sentence based on previous data. Like you want giving X value predict y value.
So, basically you need to create a dataset as per your requirement and once your dataset is ready you can train your model with encoder-decoder or LSTM layers.
Please find below links that may be helpful to you to fulfill your requirements. See as X and Y value input output prediction.

Gmail style smart compose
Neural Machine Translation with Attention
Sentence Prediction using LSTMs


Recurrent Neural Network (RNN) Vanishing gradient problem - Why does it affect earlier timesteps more? I understand the concept of backpropagation in standard neural networks and  backpropagation through time with RNNs, why this causes exponentially smaller gradients at earlier time steps and most of the maths behind it all, but what I don’t understand is why this affects the earlier timestep in particular? Since, the parameters (weights) in the RNN are all shared between timesteps, why is it that the earlier timestep is more affected? Wouldn’t they all be affected since they all share the same badly optimized weight which is never updated due to the many small terms in the multiplicative product which produces $\frac{\partial E}{\partial w} $? I feel like I'm totally misunderstanding something here. Many thanks

Of course that all weights are the same, but the update applied to the weights has a contribution from each of the timesteps, and the contribution associated with the first timesteps is what is more affected by the vanishing gradient problem.


Analyzing A/B test results which are not normally distributed, using independent t-test I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.

I believe the independent t-test requires that the samples be at least approximately normally distributed, which discourages me using the t-test as a valid method of significance testing.

But my question is:
At what point can one say that the t-test is not a good method of significance testing?

Or put another way, how can one qualify how reliable the p-values of a t-test are, given only the data set?

The distribution of your data doesn't need to be normal, it's the Sampling Distribution that has to be nearly normal. If your sample size is big enough, then the sampling distribution of means from Landau Distribution should to be nearly normal, due to the Central Limit Theorem.

So it means you should be able to safely use t-test with your data.

Example

Let's consider this example: suppose we have a population with Lognormal distribution with mu=0 and sd=0.5 (it looks a bit similar to Landau)

So we sample 30 observations 5000 times from this distribution each time calculating the mean of the sample

And this is what we get

Looks quite normal, doesn't it? If we increase the sample size, it's even more apparent

R code

x = seq(0, 4, 0.05)
y = dlnorm(x, mean=0, sd=0.5)
plot(x, y, type='l', bty='n')

n = 30
m = 1000

set.seed(0)
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 30')
x = seq(0.5, 1.5, 0.01)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))

n = 300
samp = rep(NA, m)

for (i in 1:m) {
  samp[i] = mean(rlnorm(n, mean=0, sd=0.5))
}

hist(samp, col='orange', probability=T, breaks=25, main='sample size = 300')
x = seq(1, 1.25, 0.005)
lines(x, dnorm(x, mean=mean(samp), sd=sd(samp)))


String indices must be integers I was trying to encode the string values of the feature 'ProductCategory' into integer values but I got this error. Kindly help.

And I would also like to ask if label-encoding this feature would not force my model to misrepresent the integer values as 0<1<2. Thanks.

LabelEncoding, if not processed again before passing into the model, will impose an order on the categories (namely the 0<1<3) that you mentioned. Here, since it does not seem like you have a lot of categories, one-hot encoding is the easiest to implement and also more effective (the categorical values are nominal rather than ordinal here).

These two articles are a decent starting point for categorical encoding in Python:
https://pbpython.com/categorical-encoding.html

https://medium.com/@contactsunny/label-encoder-vs-one-hot-encoder-in-machine-learning-3fc273365621

As for the TypeError, can you double check what variable type train_df is?


Different methods for clustering skills in text Consider a talent pool in which each member has some set of skills. Some of these talent are submitted to orders as potential candidates of which one is selected. It is reasonable to assume that the submitted talent have some dominant thing in common in their skill sets (let's call it a segment) that qualifies them for the order. Example segments are "front end web-designer" or "brochure / sprint designer".

Given the total set of skills over all of the talent submitted to an order (like 2-5 with say 10 skills each, so 20 - 50 skills total), I am looking for the dominant segment. Then, I am looking for the dominant segment for each individual talent.

My plan is to use latent Dirichlet allocation (LDA) such that the skills of all the talent submitted for an order are a "document" that contains some segments or "topics" with some probability. Likely, there will be one or two dominant topics depending on the total topic number. I will then use this model to predict the dominant segment for each talent where the individual talent skill set is a "document" with some segments or "topics" within.

I am curious if anyone has feedback about my use of LDA or other ideas about how I might go about discovering these segments?

You may want to consider preprocessing - convert different wording for the same type of talent to same wording. For example, a talent in machine learning is called data scientist at Coursera job site, data engineer at Udacity job site, or data analyst. This preprocessing is similar to stemming in concept.


Random Sequence Predictions Can machine learning algorithms predict random number generators.
key
A= 1 2 3
B= 4 5 6
C= 7 8 9
Example to catch a number sequence
4 8 8
I would select B C C
That would give me 27 number combinations
4 7 7
4 8 7
4 9 7
5 7 7
5 8 7
5 9 7
6 7 7
6 8 7
6 9 7
4 7 8
4 8 8 match
4 9 8
5 7 8
5 8 8
5 9 8
6 7 8
6 8 8
6 9 8
4 7 9
4 8 9
4 9 9
5 7 9
5 8 9
5 9 9
6 7 9
6 8 9
6 9 9
Using the example above can a sequence prediction algorithm predict the next sequence
Data
B C C
C C A
A C A
C A A
C B B
C C A
B C C
A C B
C B A
B B B
A A C
C A C
B A C
A C A
B C C
B A A
C B C
A C C
A A C
C C B
B B C
B C C
A A B
B A A
A B B
A B A
C C C
C C A

Can machine learning algorithms predict random number

No, absolutely not.
ML is not a magical process, it works by identifying patterns in the data. In the case of supervised learning (i.e. with training) it means finding in the features the most relevant indications to help predict the target variable.
Randomness is the complete opposite of "patterns", by definition something truly random is not predictible in any way.
Intuitively if a human with a lot of time cannot do a task, then ML cannot do this task either.


gradient descent in n dimensions Gradient descent in $n$ dimensions.

I'm learning about the downward gradient and the youtube videos and books only show a 2d curve as the slope drops to the minimum of the curve.

My question is, does the slope go down a multi-dimensional curve when the characteristics of a data set > 3?

How to see the slope going down in a data set with many characteristics?

Typically, you won’t use plots to show anything larger than 3D. Yes, gradient descent will always go down the curve of convex loss function, I.e, linear regression.

The gradient value is simply the derivative “slope” of the loss with respect to the inputs. It capture how “off” the output is on average for each feature. For example, if the error of an record is high and the feature is lower than optimal, the gradient will point in the direction that increase the feature’s value.


Predicting Sequence based on Tabular Features I have a dataset regarding a specific junction on a piece of road and the cars that will cross that junction. I am trying to predict the order that cars will pass the specific junction given a set of tabular features. My dataset looks similar to the following:

Target      car1_type  car2_type  car3_type      car1_positionId   car2_positionId   car3_positionId ...
3,1,2       1          2          3              3                 6                 8
2,1,3       8          4          9              1                 4                 2

My features inlcude the type of car (car_type) as well as the position of each car (car_positionId). The position represents a region of road that the car is on. Each row above represents an observed case of 3 cars crossing a junction.

I am trying to predict the target column (the order that the cars will pass over the junction) based on the features given. There is additional complexity in that I also do not know how many cars there will be. There could be just one (trivial case) or there could be up to 20.

My question is what machine learning algorithm could I use to help me predict the order.

I think you have 2 different ways to approach this problem. But the variable number of features and targets make it challenging. I am not aware of any well established ways to treat such issues but I find the problem interesting and would like to share my opinion.

The first way is to define this as a classification problem. The order of cars matters, in the sense that it is loss of a problem if you classify car in class 1 as a class 2, compared to identifying it as a class - say - 15 in a 20 car case. This is an "Ordinal Classification" problem and as far as I know, there is no established loss function for this. But someone implemented their own and shared it, but I never used it so I don't know how it works. Of course, this is not a must a you can use a regular classifier loss function.

Now you have to deal with variable number of cars. If you have a maximum possible number of cars present at each time, you can use that to define the number of features in your Neural Network. For the cases where there are less number of cars you can set the features associated with the non-existent cars to a number (for example -1), and put a constraint to the loss function such that cars with such features are ordered after the present cars only, and drop them at the end. This might sound complicated, it certainly is so to type it here; so I hope I was able to explain it.

The second possible way is to define a clustering problem. In order to deal with variable car numbers at each case you can define the total number of cars as an additional feature. In the example you give above they all will have a car number feature = 3. And then use any clustering algorithm you can think of, such as KNN with K=max number of cars. But this also has drawbacks: When 3 cars are present, cars can be clustered as 1,2,3; but they can just as easily be clustered into 4,6,9; or even 1,1,1. And I have no idea how you would handle such an issue. Problems with the classification algorithm defined above is less likely to cause such problems, but it is much more difficult to implement.

I know these options are far from optimal so I really hope someone else comes up with a better answer. Good luck.


For every Bayesian Network, is there a Neural Network that gives the same output? After thinking about a question comparing Bayesian Network to Neural Network, I am now wondering if they may not be one and the same thing!

At this point, my maths (unused for over 20 years fails me!).   But....

If infinite random test data was used
The output of the Bayesian Network was found for each data item
And the Neural Network was then trained on it

Would the Neural Network not tent towards being the same as the Bayesian Network?

We don't have to do anything crazy to prove this, we can simply leverage the UAT.

By the universal approximation theorem, yes, you could concieve of a neural network that precisely matched your Bayesian network.


Python: validating the existence of NLTK data with database search I need to pull the names of companies out of resumes. Thousands of them. I was thinking of using NLTK to create a list of possible companies, and then cross-referencing the list of strings with something like SEC.gov.

I've already been able to successfully pull the candidate's name, and contact info off of the resumes with some RegEx, but this one has me quite stumped.

What I'm thinking is that I could use NLTK to create a list of strings of proper nouns from the resume's, and then search SEC.gov, or some other database.

This is a link to the SEC page I would be searching: SEC company search page

Read Resume1
Get all potential company names as list of strings potentialCompanies
    IF searching for string1 in SEC gets result, THEN add to candidateCompanies
        ELSE remove from potentialCompanies, go to next string

My Questions

To people that have used NLTK, would there be a better way of getting the potential companies from the text besides using proper nouns?

Would there be a better place to search for companies than the SEC site?

I have never done any web scraping before, and don't really know where to start if it is needed.

(I had posted this on Stack Overflow but they told me that it might be better suited for here...)

NLTK has a built-in NER model that would extract potential Organizations from text, you can read about it here (and see examples) NLTK book (look for section "5 Named Entity Recognition").

However, if your input text has organizations in a very specific context that wasn't seen by NLTK NER model, performance might be quite low. In that case you should be looking into training your own NER model, what would extract company names. For that you would require to manually markup a small amount of your dataset.