With unbalanced class, do I have to use under sampling on my validation/testing datasets? I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.

The thing is that, as you may know, the dataset is very unbalanced : Around 1300 negative examples (non click) for 1 positive example (click).

This is what I do:

Load the data
Split the dataset into 3 datasets :
    A = Training (60%)
    B = Validating (20%)
    C = Testing (20%)
For each dataset (A, B, C), do an under-sampling on each negative class in order to have a ratio of 5 (5 negative example for 1 positive example). This give me 3 new datasets which are more balanced:
    A’
    B’
    C’

Then I train my model with the dataset A’ and logistic regression.

My question are:

Which dataset do I have to use for validation ? B or B’ ?
Which dataset do I have to use for testing ? C or C’
Which metrics are the most relevant to evaluate my model? F1Score seems to be a well used metric. But here due to the unbalanced class (if I use the datasets B and C), the precision is low (under 0.20) and the F1Score is very influenced by low recall/precision. Would that be more accurate to use aucPR or aucROC ?
If I want to plot the learning curve, which metrics should I use ? (knowing that the %error isn’t relevant if I use the B’ dataset for validating)

Thanks in advance for your time !

Regards.

Great question...  Here are some specific answers to your numbered questions:

1) You should cross validate on B not B`.  Otherwise, you won't know how well your class balancing is working.  It couldn't hurt to cross validate on both B and B` and will be useful based on the answer to 4 below.

2) You should test on both C and C` based on 4 below.

3) I would stick with F1 and it could be useful to use ROC-AUC and this provides a good sanity check.  Both tend to be useful with unbalanced classes.

4) This gets really tricky.  The problem with this is that the best method requires that you reinterpret what the learning curves should look like or use both the re-sampled and original data sets.

The classic interpretation of learning curves is:

Overfit - Lines don't quite come together;
Underfit - Lines come together but at too low an F1 score;
Just Right - Lines come together with a reasonable F1 score.

Now, if you are training on A` and testing on C, the lines will never completely come together.  If you are training on A` and testing on C` the results won't be meaningful in the context of the original problem.  So what do you do?

The answer is to train on A` and test on B`, but also test on B.  Get the F1 score for B` where you want it to be, then check the F1 score for B.  Then do your testing and generate learning curves for C.  The curves won't ever come together, but you will have a sense of the acceptable bias... its the difference between F1(B) and F1(B`).

Now, the new interpretation of your learning curves is:

Overfit - Lines don't come together and are farther apart than F1(B`)-F1(B);
Underfit - Lines don't come together but the difference is less than F1(B`)-F1(B) and the F1(C) score is under F1(B);
Just right - Lines don't come together but the difference is less than F1(B`)-F1(B) with an F1(C) score similar to F1(B).

General: I strenuously suggest that for unbalanced classes you first try adjusting your class weights in your learning algorithm instead of over/under-sampling as it avoids all of the rigor moral that we've outlined above.  Its very easy in libraries like scikit-learn and pretty easy to hand code in anything that uses a sigmoid function or a majority vote.

Hope this helps!


can machine learning/Deep learning used to minimize an objective function? I have data of construction site and am wondering if i can use machine learning to reduce the cost it takes to build a building. But, as far as i know, Machine learning can only does function approximation.  But, i remember that google used AI to reduce the costs of the data center. This and  This are the article.

We actually wrote this for an engineering company. Basically we modeled the project business processes, then decomposed them to metrics for analysis. I can high level three cost savings areas we found.

Purchasing: use receipts, billing, and time metrics to identify outliers and reconciliation failures (receipts != payments). Outliers/missing may indicate mistakes or fraud/theft/overcharging in billing (ie:$200 hammer). Used filter functions to find.

Contract and Permit analysis: make sure all required permits and contract provisions are in place. Used text classification and boolean (exception reporting, really).

Project length prediction: if a project is shorter or longer than others in it's class, investigate. Used statistical analysis (the client already classified: public/private, plumbing/sewage, location, etc).

This was before TensorFlow, which likely could improve on our results. You will find that there are plenty of features (metrics) to model in these processes.

So hopefully this can help on model definition and  consequently the objective functions.


Correlation between words, then texts I have texts collected on different topics. I would like to study the possible correlation between these.
I have started looking at the word frequency and it seems that in one dataset the word with highest frequency is cat; in another dataset is mice;
in another one is house.

Do you know about some technique or approach (similarity, classification,...) to show the possible correlation between datasets (e.g. between cat and mice)? Should it need only the expert judgement?

One method you can try first is cosine similarity. It works by counting the number of occurrences of each word in the vocabulary for each individual document. Next, you put these counts into vectors and then you take the cosine of the angle between them.

If you have more than one text on a topic, you can combine them into a single text for the purpose of finding the cosine similarity between various topics.

If you have many texts that are examples of each topic, then you can create a supervised machine learning model. However, since your purpose is interpretation instead of prediction, I would recommend a technique like cosine similarity before getting insight from interpreting a predictive model.


I have 50 videos. I ask a customer 10 questions. Based on their answers, I send them a set of videos. How do I do it? This might make you feel like I am looking for a recommender engine, but I am not. A recommender engine works well if accuracy isn't an issue, but in my case, it is.

What I have proposed is to associate tags with certain question's answers, and map those tags to a particular video(s) and discern based on the mapping, whether the person should see it. Personalization. What do you think?

Here's the topic at hand to put this into context. Each video deals with a particular workout and is to be served depending on the answers in the onboarding screen. It then forms part of a program.

Frame this as a classification problem and learn a decision tree to map question responses to video selections.

EDIT: Fleshing this out a bit more:

Collect appropriate data. Get members of your target population to complete the survey and also indicate which videos they think would be appropriate to them, or alternatively have subject matter experts associate surveys with appropriate videos. However you do it, you need to acquire a dataset pairing surveys with "ground truth" video suggestions.
For each video, train a classifier with the survey responses as input to predict the binary target variable of whether or not that video was suggested for that survey.
To construct a suggested set for a new survey, use all your classifiers to score it. Any classifier that outputs a 1 is a video you should send in response to that survey.


Dealing with categorical variables in regression problems which method to use? Usually if I have regression problem and my initial dataset contains categorical variables like :
column 1:
Math
Science
Science
English

I would convert this non-numerical variables to numerical variable such that : Math: 0, Science : 1, English : 2.
However, I recently found a tutorial said that this solution is not performant because there is no favorite class among other means there is no increase between those classes and if it existe we can not quantify it.
Can anyone explain this for me because I usually worked with solution one ?

This solution would be performant only if your values has an order. Some models use as learning function the distance between points, and if you use your method, a student in Math and a student in English (0 and 2 making a 2 distance) will have more distance than a student in Math and a student in Science (0 and 1 making a 1 distance).
Using this method involves a bias, so you'll have to go another way. One well known method is One Hot Encoding, which will create 3 binary-variables Column1_Math, Column1_Science, Column1_English, with values 0 or 1 (for example, if column 1 is Math, then you'll have Column1_Math = 1, Column1_Science = 0, Column1_English = 0). This way, you avoid biaising your model.
I already explained other ways I know to deal with your issue in this answer that I highly suggest you to take a look at


Clustering Customers on transactional behavior Objective:

Segment the accounts on their transactional behavior and find the accounts which are more likely to subscribe for loans.

Dataset:

1)      Account_Number

2-91)   Transaction amount aggregated on days for last 90 days

92)     is_subscriber? (0/1)

My Approach:

1) Normalized the data (x-min(x)) / max(x) - min(x)

2) Clustering on column 2-91

3) Analyze the clusters to check which cluster has the maximum percentage of subscribers (is_subscriber? == 1)

4) Sort the clusters on the basis of maximum percentage of subscribers

5) The remaining non-subscribers (is_subscriber? == 0) of cluster with highest percentage are most likely to get subscribed.

But the results I am getting doesn't look promising.

The data is already prepared. No other variable can be added.

Questions

1) which data normalization to use?

2) Which distance to use in clustering?

3) Which clustering algorithm to use?

4) Considering the data if there is any other approach you can suggest?

Data can be found here: https://drive.google.com/open?id=1ydL1AQmozBlfiqz0nsM3f_E2JK2KB8tD

As you have a target (column 93: "is_subscriber"), why not opt for a regression or classification method? Clustering is more beneficial for situations where you want to extract "common features" out of your dataset but you have no information on that dataset or the dataset implication, i.e., target.

Moreover, if you want to predict future customers as "potential subscribers" or not, a regression will provide you with the prediction formula. Then you can apply that formula to future customers to predict their behavior. You can obtain such result with Logistic Regression or, if you prefer a classifier, a Random Forest.


Custom metrics for unbalanced classes problem in RandomForest or SVM My dataset has highly unbalanced classes ‒ foreground of 30 classes with tens of samples against background set of >100k samples. Classifying foreground class as background is quite OK, while classifying background as foreground should be penalised.

I am using Scikit-learn's RandomForests, and I was experimenting with SVM and OneVsRest classifiers as well. I would like to specify the scoring metrics used for the method fit() of the model, so it will correspond to my goal (I imagine something like fitness function with evolution algorithms). However, API does not allow something like that.

So far I tried:

Use class_weight parameter of the model. If I set it so it represents the real world, then the classifier learns to classify everything as background having accuracy >99 %. If I set class_weight = 'balanced', then it seems better, but it has high false positive rate.
Use scoring method for GridSearchCV, which outputs values specified by me (even F1-score makes more sense than simple accuracy), but it is used only for the parameter selection and the final model is learnt by fit() method, which again ignores my scoring.

Is there a way to provide my own custom scoring function for the fit() method?

I think I understand what you are trying to do. First, let me attempt to obtain clarity on two concepts that may be getting conflated (if not for you, then for other users).

When a random forest is being fit, it typically uses entropy or gini impurity for categorical outcome variables, and mean squared error or mean absolute error for numerical outcome variables -- this is the model building step. However, when the fit model is being evaluated, the scoring method is used to understand the models performance -- this is the step where we learn how accurate the model might be.

I've seen that sklearn allows specifying a custom scoring method, but as far as I know, I don't believe you can specify your own criterion parameter in the RandomForestClassifier or RandomForestRegressor methods/APIs.


calculating average of each session by unique itemID i have a data frame

 > data
session id        item id
  1                 1
  1                 1
  1                 1
  2                 1
  2                 2
  3                 1
  3                 0
  3                 3
  3                 2

and i want to calculate avg of each session id like this uniqueitemsinsession/totalitems= ans (1/3=0.333)and get result like this

>  result
session id       avg
   1             0.333
   2             1.000
   3             1.000

how can i do this??

Assuming you can use dplyr package

data %>% group_by(session_id) %>% summarise(avg = length(unique(item_id))/length(item_id))

You have asked many questions on basic grouping and summarise operations in R. It would be better to go through tutorials like this to learn all these tricks in one go.


Is it acceptable to select a random child node when using a Decision Tree (trained via ID3) to predict if an unknown attribute value is encountered I am writing a decision tree trained with the ID3 algorithm from scratch. I wanted to be able to train on and classify continuous data, so I implemented k-means clustering and reduced the range of values of any input training or predicting data.

However, I ran into the problem where an attribute value that was not encountered during training, in a deep node somewhere in the tree, but that exists further up was encountered. All data points with this specific attribute value probably ended up on a different branch of the tree.

So to 'solve' this, whenever an unknown attribute value is encountered for a node, I sent it down a random existing branch.

I get 95.45% accuracy using randomly split 85%-15% training-test data with iris.

Is this an acceptable approach to take or have I gotten something wrong here?

Here's the code: https://github.com/jamalmoir/ml_components/blob/master/ml_components/models/decision_tree.py

Thanks

Do your leaf nodes return a probability distribution rather than a single class value e.g., the majority class of training instances that arrived at the node? If so, a better approach would be to send the test instance down every child path and average the probability distributions that are returned by the leaves.

Your good score of 95.45% on iris data suggests to me that it's not a very important detail in this example. Try some more challenging datasets and see if it makes a difference.


Neural net only works for small datasets I have a neural network that attempts to solve some regression problem.

The network works fine when the dataset has a small number of training examples, lets say 20. It overfits of course, but the training loss decreases to 0 which is what I want right now.

However, when I take my entire dataset, which has 5000 training examples in it, the loss plateaus at around 375. It won't decrease to zero, no matter how many hidden layers and neurons and activations I add. I tried everything.

What does this mean? Why does this happen and how can I fix it? Is it plausible that it's simply not possible to train a model that fits the training data?

It could mean, that your data is too complex to be learned by the model using a certain number of observations (what is number of features do you have?). On the plot training loss is almost zero for small dataset, but you should rely on the validation loss, not on training one. And validation loss is lower for bigger number of samples, then learning process performes predictable normal


Determining which categorical data is beneficial in predictive modelling I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all independant), and some historic data might look like:
JobID   Manager     City        Design          ClientType      TaskDuration
a1      George      Brisbane    BigKahuna       Personal        10
a2      George      Brisbane    SmallKahuna     Business        15
a3      George      Perth       BigKahuna       Investor        7

Thus far, my model has been relatively basic, following these basic steps:

Aggregate the historical data based on each category, calculating the mean, and counting how many times it occurs. From the previous example, the result would be:

Category        Value           Mean    Count
Manager         George          10.66   3
City            Brisbane        12.5    2
City            Perth           7       1
Design          BigKahuna       8.5     2
Design          SmallKahuna     15      1
ClientType      Personal        10      1
ClientType      Business        15      1
ClientType      Investor        7       1

For each job in the system, calculate the job duration based on the above. For example:

JobID   Manager     City        Design          ClientType
b5      George      Brisbane    SmallKahuna     Investor

Category        Value           CalculatedMean      CalculatedCount     Factor (Mean * Count)
Manager         George          10.66               3                   31.98
City            Brisbane        12.5                2                   25
Design          SmallKahuna     15                  1                   15
ClientType      Investor        7                   1                   7

TaskDuration    = SUM(Factor) / SUM(CalculatedCount)
                = 78.98 / 7
                = 11.283
                ~= 11 days

After testing my model on a few hundred finished jobs from the last four months, I calculated average discrepancies ranging from -15% to +25%.
I think the one of my issues is that I may be taking into account categories that actually have no effect on the build time, and are skewing my results. In reality, I'm taking 15 categories into account from ~400 completed jobs, and some of these categories might have results that only appear once or twice (for example, we might only have a single job in Perth).
How can I determine which categories are actually beneficial to the model, and which should be ignored?
Related question here.

You can try two things -

Try finding the correlation between the Categories and the Target.
Since, It's between Categorial features and a Continuous Feature, you should -
Get the r-square Or Adjusted R-square score of Regression, see which one is best and drop the lowest few and try.
Read more - Kaggle

Calculate Feature Importance using random Forest.
 Read here - MachineLearningMastery


Best metric in imbalanced classification for multi-label classification My test data are imbalanced, i tried to use the precision or the gmean as metrics for a multi-label learning model, but both metrics are not very informative. Is there any way to use for example the precision that gives information about the minor class and the accuracy in a single procedure. Any other solution for this problem is welcomed.

Thanks in advance.

Maybe you should instead try to use some techinique to rebalance your dataset and then use a classical metric for classification problems.
Example of tchniques for rebalancing the dataset for binary classification is given in the following post:
https://stats.stackexchange.com/questions/152823/how-to-balance-my-dataset
If you still want to work on the unbalanced dataset in the following paper some s are suggested:
https://eva.fing.edu.uy/pluginfile.php/69453/mod_resource/content/1/7633-10048-1-PB.pdf


Time Series data: How to convert it in a streaming data? I have a time series data which is available in offline csv format. I am using this data to create anomaly detection model. Although I could create this model to predict anomalies in this dataset, I need to use this model when data is real time. Every second data point will be coming and then if point is anomalous I need model to raise an anomaly.

Any ideas how should I implement this?

Edit -
Data is updated regularly in computer, file is updated as data is streaming in this file only. Predictions are required for the realtime anomaly detection as decision will have to be made to stop the process when such prediction for detection of anomaly is made.

It depends on the answer to these questions:

How is the real-time data provided?
How often and in which format are the predictions required?

It's common to use a database in such a setting. So the ML prediction process would be called regularly, query this database to get the latest instances and generate predictions in the required format (possibly storing the predictions in the database as well).


Distance between very large discrete probability distributions I have 192 countries where each country has some value for 1 million attributes which sum up to 1 (a discrete probability distribution). For any one country most of the values for the attributes are 0.

Now I am trying to find the distance/similarity between those countries using these attributes. I know we can use  Jensen Shannon Divergence between two discrete probability distributions to get a distance measure, but the caveat is that all the values have to be non-zero.

Given that there are zero valued attributes for the countries, is there any other suitable statistical distance measure that can help me to cluster these countries using these 1 million attributes?

Yes, plenty.

Get the book "encyclopedia of distances".

For example, you can use Histogram Intersection distance. Since your data is already normalized, that reduces to Manhattan distance, if I am not mistaken. Yes: this can be appropriate for distributions.


What should return doc.ents if the doc have no entities, in spacy? I want to answer this question: "How many sentences contain named entities given a doc?"
and I have this piece of code as solution

nb = 0

for sent in list(doc.sents):
    if sent.ents:
        nb = nb+1

print(nb)

But I am a little confused about how the if-statement work;
consider that the actual sentence(sent) does not have any entities(ents), so sent.ents will return an empty list I suppose and I can't understand why an empty list is considered as a false statement because the program when executed does not enter the if.

You could just test whether the tuples entities has any elements:

for sent in list(doc.sents):
    if len(sent.ents) > 0:
        nb = nb+1

Edit: For the purpose of evaluating boolean expressions (as in case of if statements), empty lists are evaluated as false, even though they don't explicitly equal False. Expanding your example with some print statements:

nb = 0
spacy_nlp = spacy.load("de_core_news_md")
text = "Nichts. Auch nichts."
doc = spacy_nlp(text)
for sent in list(doc.sents):
    print("type of ents: " + str(type(sent.ents)) + " with " + str(len(sent.ents)) + " elements.")
    if sent.ents:
        nb = nb+1

print(nb)

Results in:

type of ents: <class 'list'> with 0 elements.
type of ents: <class 'list'> with 0 elements.
0


Clustering a set of vectors Provided a set ($m$ no. of) of n-dimensional vectors what would be the correct unsupervised approach to cluster them? The vectors essentially represent patterns.

For example:
Set of vector is represented as $V$. Let a vector $v1$ represents a pattern similar to $y = sin(x)$ curve. The $y$ values are stored in $v1$ and the $x$ intervals are same for all the vectors. Similarly there is a vector $v2$ representing pattern similar to $y=log(x)$.

The problem: Does a group of vectors exist, exhibiting similar (not exactly same) pattern as $v1$, similarly for $v2$ and so on?

Therefore these patterns are required to be clustered appropriately. There are methods such as Vector Quantization, but I am not sure if those methods are appropriate in this case.

Since Neural Networks can learn any non-linear function, I would go for an Autoencoder for dimensionality reduction first, and then run a k-Means clustering on the encoded features. If a regular (non-linear) pattern in the data exist, an Autoencoder that is "deep enough" should be able to account for that.

For the same reasons (i.e. non-linear patterns) I would avoid linear techniques such as PCA.

If this suits you, you can check my Notebooks. I implemented Autoencoders in TensorFlow 1.x here, and in TensorFlow 2.0 here.


What do I do when my column has 50% data missing? I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should i impute this? Do i replace ? with unknown or do i replace it with NaN.

Since my end goal is a classification problem i will be applying various algos on it such as NB classifier, random forest, decision tree, K neighbours. And sometimes it does not matter whether you impute or not since some algos do it for you

It depends on your knowledge of the problem. First, you should classify why is it missing??

Structurally missing data
Structurally missing data is data that is missing for a logical reason. In other words, it is data that is missing because it should not exist.check this

Structurally missing data is data that is missing for a logical reason. In other words, it is data that is missing because it should not exist.
Missing completely at random (MCAR) No pattern in missing data
Missing at random (MAR)An alternative assumption, known somewhat confusingly as missing at random (MAR), instead assumes that we can predict the value that is missing based on the other data.

Then choose your imputing technique based on your knowledge of the problem if you have other cases you can try (from this source):

Encode NAs as -1 or -9999. This works reasonably well for numerical features that are predominantly positive in value, and for tree-based models in general. This used to be a more common method in the past when the out-of-the box machine learning libraries and algorithms were not very adept at working with missing data.
Casewise deletion of missing data. Here you simply drop all cases/rows from the dataset that contain missing values. In the case of a very large dataset with very few missing values, this approach could potentially work really well. However, if the missing values are in cases that are also otherwise statistically distinct, this method may seriously skew the predictive model for which this data is used. Another major problem with this approach is that it will be unable to process any future data that contains missing values. If your predictive model is designed for production, this could create serious issues in deployment.
Replace missing values with the mean/median value of the feature in which they occur. This works for numerical features. The choice of median/mean is often related to the form of distribution that the data has. For imbalanced data, the median may be more appropriate, while for symmetrical and more normally distributed data, the mean could be a better choice.
Label encode NAs as another level of a categorical variable. This works with tree-based models and other models if the feature can be numerically transformed (one-hot encoding, frequency encoding, etc.). This technique does not work well with logistic regression.
Run predictive models that impute the missing data. This should be done in conjunction with some kind of cross-validation scheme in order to avoid leakage. This can be very effective and can help with the final model.
Use the number of missing values in a given row to create a new engineered feature. As mentioned above, missing data can often have lots of useful signal in its own right, and this is a good way to encode that information.


How does graph classification work with graph neural networks I am reading the paper The Graph Neural Network Model by Scarselli et al. I understand how node classification works. I am having trouble understanding how graph classification works however. In particular, in the section titled The Learning algorithm, the authors mention that

  Learning in GNNs consists of estimating the parameter such that w
  approximates the data in the learning data set

  where qi is the number of supervised nodes in Gi. For graph focused
  tasks, one special node is used for the target (qi = 1 holds), whereas for
  node-focused tasks, in principle, the supervision can be performed on
  every node.

The node focused task approach makes sense to me; you would essentially compare the ground truth to each output of the "local output function" for each node, and backprop accordingly. However, based on the above description, I do not understand what you would do to classify the graph as a whole, given its label. What do they mean by "one special node is used for the target (qi = 1 holds)"? Why are they talking about a "special node"? Why is there no mention of the graph's label? Isn't that what we want to predict?

EDIT:

After reading through the entire paper, and specifically looking at the Mutagenesis example, I got a better understanding of how graph classification works (as described in this paper at least). However, my understanding is still not complete. I will explain what I understand, and raise a followup question below.

As the text above suggests, a particular node in the graph is chosen (I believe this can be done at random), and it will be the only node in the graph that is "supervised." All other nodes will be unsupervised (so we will not make any predictions on those nodes). We choose the local output function in such a way as to have it output a number between -1 and 1 (although I'm unsure as to whether or not you could pick a function that instead outputs a number between 0 and 1. I believe you can, and it's just a matter of what activation function you would like to choose i.e sigmoid vs tanh in this example). If the output is < 0, we predict the graph has label -1, and 1 otherwise.

Now we just do what we did with node prediction, except we only backpropagate on this single node that we chose.

This, however, raised a followup question for me. If you are training on multiple graphs (for graph classification), each of which has a different connectivity (which is usually the case, and is the case in the Mutagenesis example), how do you backpropagate? Each graph (in this case, molecule) represents a different neural network ...

The equation you pointed out in the paper has posed the graph learning problem based only on the nodes of the graph. Therefore, to perform graph level tasks like graph classification, one would need a 'special node' which introduces a node that represents the entire graph. This is all just to make the equation hold for graph level tasks which are not dependent on just one node.

The paper which you are reading is quite old and the new GNN formulations are based on the kipf-GCN (https://arxiv.org/abs/1609.02907). It is much simpler to understand, implement and has become the standard GCN. If you want to keep with the current hot topic of GNN, I would recommend you having a look at Kipf's blog (https://tkipf.github.io/graph-convolutional-networks/) for a simple intuition for the modern-day GCNs.


over-fitting with good enough test accuracy Let's make things simple. Imagine an underdetermined linear system with $N$ samples and $p$ features $(N<p)$. Let's say I found one of the possible (among many) solutions of such systems and computed the test accuracy. My train's error is zero in this setting, but not the test's accuracy. Hence, overfitting. Despite that, assume that the test error rate is considered to be good from the perspective of the field experts. Let's also assume that some checks guaranteed that this rather good error rate is not due to test/train separation either.

Is it true to argue that no effective learning has taken place here?
If so, can one conclude from this that a better model with better accuracy in test exists that does not overfit?

There's probably a bit of interpretation involved in the question, here is my take on it.

For the sake of clarity let me start with my definition of the concept: overfitting is when a model takes into account patterns which are present in the training data by chance, i.e. the model assumes that these patterns are characteristic of the distribution even though they're not.

  Is it true to argue that no effective learning has taken place here?

No it would be wrong to say that: the fact that a model overfits does not mean that no effective learning happened at all. In fact it's often the case that a model successfully acquires the patterns that it's supposed to capture (let's call this effective learning), but also acquires patterns that it shouldn't (overfitting). With any complex data it would even be very rare that no overfitting happens at all, and in fact very often it's hard to say exactly where is the line between effective learning and overfitting.

However people usually talk about overfitting when it's actually in excess, that is when the model generalizes too much on the "chance patterns" and not enough on the patterns which actually characterize the distribution. A performance which is significantly lower on the test set than the training set is a typical sign of such "excess overfit". So in this sense (excessive) overfitting is when the "chance patterns" cause the model to be suboptimal. But even a suboptimal model may have learned some relevant patterns.

  If so, can one conclude from this that a better model with better accuracy in test exists that does not overfit?

Not really: as said above overfitting can make the model perform poorly, so naturally it's usually the case that a non-overfit model performs better than an overfit one. However there's no guarantee that a better performance can be reached just by getting rid of the overfitting: as an extreme example, if the features are mostly random and/or not correlated with the response variables then it's very likely that model will overfit, but the performance would be terrible without overfitting anyway.

In the case described by OP I would say that it's always worth trying to avoid (excess) overfitting: first of course because it can only improve performance, but also because at a more general level it means that the model is not very reliable. If later the model is applied "in production" to a large set of instances which happen not to have this particular "chance pattern" which was in the training data, the model is going to go very wrong and it will be too late to detect it.


Predicting descrete value problem in regression or classification In machine learning, regression algorithms attempt to estimate the mapping function (f) from the input variables (x) to numerical or continuous output variables (y).

I have one usecase where I predict shift_id. Shit_Id is ID values given to different city location.

As per my understanding this is regression problem because it predict numerical value. Is this right?

Also precision, recall f1 measure can be calculated for regression problem?

IDs are categorical, not numeric. You should be treating this as a multi-class classification problem. Your IDs are locations, a location is a class. The ID is just a identifier for the class.

Since you have a classification problem you should be using precision, recall and f1. However, if it was regression you would have been using mean squared error, mean absolute error and possibly something else.


Calculating Confidence Interval at a certain confidence level def get_ci(mean, cl, sd):
  loc = stats.norm.ppf(1 - cl/2)
  rng_val = stats.norm.cdf(loc - mean/sd)

  lwr_bnd = value - rng_val
  upr_bnd = value + rng_val

  return_val = (lwr_bnd, upr_bnd)
  return(return_val)

This function takes three parameters which are the following:

mean --> It is the mean
cl --> Confidence Level
sd --> Standard Deviation

Can someone explain how this function is working as for example if we are calculating 95% confidence interval, why can't we just return (-2*sd+mean,2*sd+mean)?

The CI is defined as the interval which contains your mean with a $\alpha$% of probability.

Given that you are using a model which its subjacent assumption is normality (amongst others), the interval is to be obtained comparing the probability in the context of a normal distribution.

The function stats.norm.cdf returns the probability of $loc - \frac{mean}{sd}$ being zero.

The value returned is the $(\bar{X} - Z_{\alpha}\sigma,\bar{X} + Z_{\alpha}\sigma)$.

We can't return $(-2 sd+mean,2 sd+mean)$ because it would not be general in terms of probability, we use tables which say $0.95 \rightarrow 2$ because we (humans) cannot calculate the number for every probability, but the stats.norm.cdf function can.

$Z_{\alpha}$ is difficult for us to have all the possible values (what if we need 90%, 95%, 99%, 99.9% or 70% probability?)


Mining timelines in a long text I am trying to detect timeline of brands histories. For my specific case, I believe it is easy because data is already clustered. For each Wikipedia article I can spot sentences surrounding dates. Here is an example:

McDonald's Corporation is an American fast food company, founded in
1940 as a restaurant operated by Richard and Maurice McDonald, in San
Bernardino, California, United States. They rechristened their
business as a hamburger stand, and later turned the company into a
franchise, with the Golden Arches logo being introduced in 1953 at a
location in Phoenix, Arizona. In 1955, Ray Kroc, a businessman, joined
the company as a franchise agent

From this, it is easy to narrow results programmatically to

McDonald's is founded in 1940
Golden Arches logo introduced in 1953
Ray Kroc, a businessman, joined the company in 1955

This seems easy if documents are clustered. If not, I am thinking of a basic algorithm to mine timelines 'or natural numbers). So I want to discuss existing studies and my intuition here.
Definitions:

Timeline: a logical succession of events on one single subject.
Dates in a timeline are natural numbers, and can be "unordered relatively".
Timelines are continuous (one range like) and cannot intersect.

Let's ignore the NLP related part, and try to figure out timelines in natural numbers ignoring topics (1st definition).
Distance: Initial timeline length. It represents the minimum.
Example:
Step A
1, 4, 2, 5,  3, 8, 7, 9, 20, 21, 23, 24, 1, 5, 7, 9

dist = 4

Becomes:

1, 4, 2, 5/  3, 8, 7, 9/ 20, 21, 23, 24/ 1, 5, 7, 9

Score each set (of 4 elements): Scoring is critical but lets think of bubble sort score, where score = 1 / number exchange ops.
1,4,2,5 => 1/1  | 3,8,7,9 => 1/1

Step B
The reason to score sets is to identify if a set represents a timeline or the combination of two sets represent a timeline, to decide, we score the combined set and divide by two
1,4,2,5,3,8,7,9 => 5/2

We conclude 1,4,2,5 and 3,8,7,9 are two sets, while 1,4,2,5,3,8,7,9 is not.
We move sequentially to process next sets.
The reason I said Distance is minimum is that before comparing scoring initial sets, we first   identify sets of 4, 5, 6 or more elements and score them ((step A)) and only take separate sets with better score (minimum bubble sort score here).
Any thoughts ?

It looks to me like what you propose makes sense, but there has been some research done around these questions of time representation already. I'd suggest you check the state of the art in this domain, if only not to reinvent the wheel or miss important cases.
I'm not very knowledgeable about it but I can at least point you to TimeML and the related publications. There are certainly other recent works building on TimeML, for example this one (disclaimer: I know the author).


How to begin understanding of audio and music analysis I recently have been assigned to do some work with the python libROSA library.  I don't have extensive experience with audio and music analysis and the apis and docs seem to assume a higher level of understanding.  For example, the hello world example says things like:

The example is encoded in OGG Vorbis format,
The variable sr contains the sampling rate of y, that is, the number
of samples per second of audio.
By default, all audio is mixed to mono and resampled to 22050 Hz at
load time.

And I am "why do you need to encode audio?" and "why do you even need to sample? (analog v digital I guess)" and "why do you need to mix to mono?"  "What does that even mean, 'mix'"?

Is there a good book(s) or sites that can help me get a basic understanding of audio and music processing?

Thanks in advance

To learn about the basics and a bunch of advanced topics, check out Meinard Müller's "Fundamentals of Music Processing" (FMP) (Amazon/accompanying website). There's also a website with many  Jupyter Notebooks demonstrating the book's content very well. FMP does not use librosa, but teaches you all the concepts you need to understand librosa. Most demonstrated approaches are signal-processing oriented. For work that relies on machine learning, you probably need to go through the recent research literature—ISMIR papers are a great place to start.

About the points you've raised:

Just like images (think JPEG, PNG, etc.) audio is stored in some format. OGG is just another format, like WAVE or MP3.
After audio has been decoded from some format like OGG, you get the raw samples, pretty much like points in a bitmap for images. One usually uses PCM for this (librosa does).
Most of the time we don't care about stereo, which is why librosa simply mixes stereo channels into one, mono channel by default. Also, most of the time we don't require CD quality, i.e., a sampling frequency of 44.1 kHz, so librosa downsamples the audio to 22.05 kHz by default. In a way, it's similar to reducing the size of an image by reducing its resolution.

Good luck!


How to evaluate data capability to train a model? Bioengineering data comprises of 512 binary features and a single Boolean label: if the particular mixture is worth further research. There are about 1,200,000 results of previous experiments available.

Every "physical" test costs, so it is tempting to train a model to predict if a certain combination of features is worth real testing.

I tried a few models, all failed to render better accuracy than 52% on test data, which is very close to blind guess considering binary classification. Namely, a Feed-forward Multi-layer Perceptron, Random Forest, Naive Bayes models.

For a plain simple XOR example from ML starter courses, for two binary inputs, all 4 cases are required to train the network.

1.2 mln samples are, perhaps, still a too small dataset to train for 512 inputs, considering a 512-bit number of all combinations and complex cross-influence.

Thinking from the opposite, how to prove that it is impossible to train any model given available training data?

1.2M is about 2^16.
You have 512 features plus the concept so the number of possibilities is much larger.
Therefore, you can claim that the number of samples you have is too small.

Though that, machine learning is done almost always in a situation that you have less samples the all permutations.

The VC dimension allow estimate how well a data set fits the underlying distribution. It applies to an hypothesis class (e.g., linear functions) whose size is typically much smaller than the number of permutations.

Note that both the number of permutations and the VC dimension refer to worse case scenarios. It is possible that on of the 512 is identical to the concept. In this case you will be able to learn though the other 511 features.


Machine Learning algorithms and Cross Validation, the best practice I'm new in Machine Learning, and I'm studying the main concepts behind algorithms from the mathematical point of view.
I'm also trying to start implementing some algorithms for regression purposes using sklearn library.
I would like to summarize the main steps in ML and provide the question:

I understand the dataset, clean and prepare, identifying a set of suitable algorithms for such dataset.
Train the algorithms and see the performances, choosing the best one.
Hyperparameter tuning of the best performing algorithm using Random or Grid Search with cross-validation between 5-10
I have the best parameters for training the algorithm, I train the algorithm with such params.

Now the question that arises for which I am not clear what to do.
May I consider to train the algorithm with the best parameters also performing the cross-validation? or Have I  to train the algorithm without the cross-validation?
In short, The model puts into production is the model trained with cross-validation (for instance using KFold for training in step 4) or the model trained without (step 4)?

Cross-validation (CV) is a method meant essentially to accurately evaluate a model using some training data. As a consequence CV doesn't have to be used when training the final model, usually one simply uses the whole training data for that. Note that using one of the CV models would have two disadvantages: there's no reason to select one or the other (the performance on one split is not relevant), and it doesn't exploit the whole training set.
I also notice that you didn't mention any test set evaluation in your steps. A common mistake is to select a model based on parameter tuning (step 3) using CV, and just assume that the CV performance is reliable. It is not because the fact that a choice was made (whether about parameters, models, features...) means that this step is still part of the training process, therefore there needs to be evaluation on a different test set afterwards. In other words: the fact that some parameters perform better than others could be due to chance, so it's important to evaluate the selected model (and only this model) on a fresh test set.


SKLearn KernelRidge memory demand I am fitting a model with 100,000 samples x 10 features (6 ints and 4 floats), using SKLearn KernelRidge:

model = KernelRidge(kernel='linear')

Looking at the task manager, 'Python' process takes ~40GB.

Can you please explain why is there such a high demand?
What kind of matrix is built in the background?

This is rather common.  The algorithm for KernelRidge requires a SVD to be performed.  Sadly, the SVD cannot handle a sparse matrix so the _pre_compute_svd function in sklearn just converts the matrix into a dense matrix and moves on.  This tends to blow up memory rather quickly.

You have a couple of choices.  Rewrite the method to handle sparse matrices or just use a different method.  SVR would be the most similiar alternative.


How to measure confidence in prediction? I built a prediction model and predicted on new data. I now want to specify a value of my confidence in this predicted value, e.g. ranging from 0 to 1.
Three methods come to my mind

Built 100 models on bootstrapped data, predict 100 times on each new observation, then calculate confidence intervals. Smaller intervals mean higher confidence. High computational effort.
use oob predictions of a random forest from every tree
Bayesian methods can give confidence intervals through the posterior

Are there other/better ones?

If you've got enough information on the underlying distributions of your data, classical statistical theory on regression models is an obvious alternative. These models always come with nice derivations of error estimates. See Frank Harrell's "Regression Modelling Strategies" for a comprehensive overview.


Encoding of high cardinality multi-label categorical feature? This is the problem of binary classification:
"1" - the subscriber is a driver (belongs to the segment of drivers),
"0" - the subscriber is not a driver (does not belong to the segment of drivers).
The files tabular_data.csv and hashed_feature.csv  ̶    here are descriptive characteristics for 4084 subscribers ("ID" is the subscriber ID).
The train.csv file is data about the target label (subscriber id and corresponding binary target).
The tabular_data.csv file contains numerical data on the subscriber's activity for 12 periods.
• period - period number (consecutive periods, 1 - the newest);
• id - subscriber ID;
• feature_0 - feature_49 - data on the subscriber's activity in the corresponding period.
The hashed_feature.csv file is a set of hashed values of one categorical variable for the subscriber.
• id - subscriber ID;
• feature_50 - hash from the value of the categorical variable.
I faced the following problems:

How should I encode feature_50 for ML algorithm? This feature contains approximately 5k unique values and each user can have almost 1k of these values. As I think one hot encoding is not very helpful here, but what technique should I use in this case?
NA filling strategy for feature_0 - feature_49 (I've only tried pandas filling strategies)
Make use of period variable?
I would appreciate any help.

When you have high cardinality I suggest you two options (one doesn't exclude the other):

Aggregate least frequent categories into one called other
Use Neural Network with an embedding layer for high cardinality categories. In case you don't know what an embedding is, it is just a table that map each category into a vector of k features (k your choice)


train_test_split() error: Found input variables with inconsistent numbers of sample dataset_url='http://mlr.cs.umass.edu/ml/machine-learning-databases/wine-quality/winequality-red.csv'
wine = pd.read_csv(dataset_url, delimiter = ';')

x_train, x_test, y_train, y_test = train_test_split(wine.quality,wine.alcohol,test_size=0.33, random_state=42)

training_accuracy = []
test_accuracy = []
# try n_neighbors from 1 to 10
neighbors_settings = range(1, 11)

for n_neighbors in neighbors_settings:
    # build the model
    clf = KNeighborsClassifier(n_neighbors=n_neighbors)
    clf.fit(X_train, y_train)
    # record training set accuracy
    training_accuracy.append(clf.score(X_train, y_train))
    # record generalization accuracy
    test_accuracy.append(clf.score(X_test, y_test))

plt.plot(neighbors_settings, training_accuracy, label="training accuracy")
plt.plot(neighbors_settings, test_accuracy, label="test accuracy")
plt.ylabel("Accuracy")
plt.xlabel("n_neighbors")
plt.legend()

Your independent variables need to be passed to the classifier's fit as a 2D array, but yours is 1D; you can use numpy's reshape: https://stackoverflow.com/questions/30813044/sklearn-found-arrays-with-inconsistent-numbers-of-samples-when-calling-linearre

Separately, I find it odd that you are trying to use quality to predict alcohol content, for which you should be using regression instead of classification.  Perhaps you meant to switch the inputs to train_test_split?


Finding an appropriate binary classification algorithm for time series data intervals Maybe someone here has experience in this matter and can point me in the right direction. I want to classify parts of an interval of numerical movement data as either resting or not resting. I have training data of what resting intervals look like. And am looking for the right algorithm to tackle this problem. I don't need the code, just a friendly push in the right direction.
Here is a rundown of the expected process.

I have an array of values that represent measurement over 5 minutes.

testData = [0, 4, 3, 5, 2, 4, 3, 4, 6, 11, 9, 10, 3, 15, 21, 5]

I have training data

trainingDataResting = [[1, 2, 1, 6, 4, 2], [3, 2, 2, 0, 4, 1, 5, 2], [3, 6, 2, 1, 0, 4, 5, 2]]
trainingDataActive = [[10, 4, 5, 9, 19, 13], [12, 8, 20, 9, 14], [13, 22, 19, 21, 11, 7, 9]]

A resting interval has to be at least 6 measurements long, and the longest possible resting interval is of interest. Is there a way to classify portion of the testData as resting, based on the trainingData? Something like:
testData = [0, 4, 3, 5, 2, 4, 3, 4, 6, 11, 9, 10, 3, 15, 21, 5]
           '-------------------------' '----------------------'
                     Resting                 Not Resting

testResting = [0, 4, 3, 5, 2, 4, 3, 4, 6]
testActive = [11, 9, 10, 3, 15, 21, 5]

I get that there will be many solutions of possible resting intervals, but I am particularly interested in the longest and most resting one.
Of course I am working with a lot more training data than I provided here.
I was working on a decision tree -> Decision Tree, before I was informed that there is prelabeled data that I could use for training.
I was thinking about long short term memory networks, do you know whether this applies here?
Thank you for the help.
Best,
Jo

You could do logistic regression combined with an additional moving average/ clustering step.
Combine your data into training rest and active data into a single array X, and have an additional array y which would represent the labels e.g. 0 - active / 1 - resting for each row in your training dataset.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(random_state=0).fit(X, y)

Then  predict for each step whether it is resting or active.
The predictions of the test set would look something like:
[0,1,1,1,0,1,1,0,0,0,0,1,0,0,0]

The additional step would be to find the groups. You could do this with a moving window like "if 5 active states in a window of 7, label it as active". You could also solve that final step with clustering, e.g. "if 0 is surrounded by 1s it is 0", or find local clusters of binary values.


From most frequent words how to extract technical skill words I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description.

# each item is a list of tokenized job_description
tok = [nltk.word_tokenize(job.lower()) for job in job_desc]

# ignore stop words, bullets, etc. And put it into one list
from nltk.corpus import stopwords
stop = stopwords.words('english')

def clean_token(what_to_clean):
    cleaned_tok = []
    for lists in what_to_clean:
        for item in lists:
            if len(item)>2 and (item not in stop):
                cleaned_tok.append(item)
    return cleaned_tok

After cleaning job description I've found most frequent words using:

freq = nltk.FreqDist(clean_token(tok))
most_freq_words = freq.most_common(100)

Which outputs:

 [('data', 211),
 ('experience', 78),
 ('learning', 70),
 ('business', 65),
 ('team', 53),
 ('science', 51),
 ('machine', 48),.....

From here I only want to extract words like machine, python, C+, technical skills. How can I go about it?

Also you can see there is word "machine" showing up 48 times and I am not sure whether it is talking about machine learning how can I go about this, I know if I want to make predictions I could've used CountVectorizer and n-grams.

From what I understand, you need more than 1-word terms. Thus, it's better to go for n-grams.

Start with creating n-grams

import re
from nltk.util import ngrams

s = s.lower()
s = re.sub(r'[^a-zA-Z0-9\s]', ' ', s)
tokens = [token for token in s.split(" ") if token != ""]
output = list(ngrams(tokens, 2))

You should get 1-gram, 2-grams and 3-grams at least to cover all possible terms.

Then you can use a Computer Science Ontology (CSO) which includes about 14k technical terms.

An alternative would be to use their own Classifier that gets a text as input and annotate terms on it from the same ontology. You can find a demo here https://cso.kmi.open.ac.uk/classify/

When I try to classify the abstract of Machine Learning from the Wikipedia article, I get the following results

  Machine learning (ML) is the scientific study of algorithms and
  statistical models that computer systems use to effectively perform a
  specific task without using explicit instructions, relying on patterns
  and inference instead. It is seen as a subset of artificial
  intelligence. Machine learning algorithms build a mathematical model
  based on sample data, known as "training data", in order to make
  predictions or decisions without being explicitly programmed to
  perform the task.1:2 Machine learning algorithms are used in a
  wide variety of applications, such as email filtering, and computer
  vision, where it is infeasible to develop an algorithm of specific
  instructions for performing the task. Machine learning is closely
  related to computational statistics, which focuses on making
  predictions using computers. The study of mathematical optimization
  delivers methods, theory and application domains to the field of
  machine learning. Data mining is a field of study within machine
  learning, and focuses on exploratory data analysis through
  unsupervised learning.[3][4] In its application across business
  problems, machine learning is also referred to as predictive
  analytics.

artificial intelligence
computer systems
computer vision
correlation
analysis
data mining
email
inference
machine learning
optimization
unsupervised learning


Machine Learning on financial big data Disclaimer: although I know some things about big data and am currently learning some other things about machine learning, the specific area that I wish to study is vague, or at least appears vague to me now. I'll do my best to describe it, but this question could still be categorised as too vague or not really a question. Hopefully, I'll be able to reword it more precisely once I get a reaction.

So,

I have some experience with Hadoop and the Hadoop stack (gained via using CDH), and I'm reading a book about Mahout, which is a collection of machine learning libraries. I also think I know enough statistics to be able to comprehend the math behind the machine learning algorithms, and I have some experience with R.
My ultimate goal is making a setup that would make trading predictions and deal with financial data in real time.

I wonder if there're any materials that I can further read to help me understand ways of managing that problem; books, video tutorials and exercises with example datasets are all welcome.

There are tons of materials on financial (big) data analysis that you can read and peruse. I'm not an expert in finance, but am curious about the field, especially in the context of data science and R. Therefore, the following are selected relevant resource suggestions that I have for you. I hope that they will be useful.

Books: Financial analysis (general / non-R)

Statistics and Finance: An Introduction;
Statistical Models and Methods for Financial Markets.

Books: Machine Learning in Finance

Machine Learning for Financial Engineering (!) - seems to be an edited collection of papers;
Neural Networks in Finance: Gaining Predictive Edge in the Market.

Books: Financial analysis with R

Statistical Analysis of Financial Data in R;
Statistics and Data Analysis for Financial Engineering;
Financial Risk Modelling and Portfolio Optimization with R
Statistics of Financial Markets: An Introduction (code in R and MATLAB).

Academic Journals

Algorithmic Finance (open access)

Web sites

RMetrics
Quantitative Finance on StackExchange

R Packages

the above-mentioned RMetrics site (see this page for general description);
CRAN Task Views, including Finance, Econometrics and several other Task Views.

Competitions

MODELOFF (The Financial Modeling World Championships)

Educational Programs

MS in Financial Engineering - Columbia University;
Computational Finance - Hong Kong University.

Blogs (Finance/R)

Timely Portfolio;
Systematic Investor;
Money-making Mankind.


Does BERT use GLoVE? From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated.

What I don't know is if the original embedding - the original thing you put into BERT - could or should be a vector. People wax poetic about how BERT or ALBERT can't be used for word to word comparisons, but nobody says explicitly what bert is consuming. Is it a vector? If so is it just a one-hot vector? Why is it not a GLoVE vector? (ignore the positional encoding discussion for now please)

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably cannot make much sense of them either.

Anyway, BERT learns its custom word-piece embeddings jointly with the entire model. They cannot carry the same type of semantic information as word2vec or GloVe because they are often only word fragments and BERT needs to make sense of them in the later layers.

You might say that inputs are one-hot vectors if you want, but as almost always, it is just a useful didactic abstraction. All modern deep learning frameworks implement embedding lookup just by direct indexing, multiplying the embedding matrix with a one-hot-vector would be just wasteful.


Is there a possible intersection of mathematical logic and machine learning? Why there is so comparatively little work in machine learning using  mathematical logic? Most of the research uses methods in (Bayesian) statistics, linear algebra, numerical analysis and so on. However, almost no use of, for instance, relevant logics is made.

My background is in philosophy, not computer science. But given that logic is in the business of analyzing  the notion of inference I was rather surprised to learn that machine learning and other branches of AI research do not rely on mathematical logic as much as I thought they would. What about non-monotonic logics in drawing inferences or modal logic? It seems to me that it is mostly philosophers and some, dare I say, older people in cs departments that work in logic?

Are the reasons practical (easier implementation of methods from other areas), sociological (machine learning/AI was started by non-logicians), or theoretical (the bulk of machine learning is mostly about categorising and analyzing data, because that is the hard part, and less drawing inferences).

As a closer: Is there any chance that mathematical logic will have greater influence on machine learning in the future?

All the best!

I think the reason why formal logic is not widely used is its strictness. Also, it has quite restricted area of application and requires the domain to be formalized first. In contrast to that, ML methods can be fuzzy, robust to noise and work well for real-life data.

Here is an interesting discussion on the topic:
https://www.quora.com/Whats-the-relationship-between-mathematical-Logic-and-Machine-Learning


Difference between SVM and GD/SGD? My colleague mentioned that a data science project is using SGD classifier.

So I started reading about GD/SGD and came across a nice article about Text classification using SVM and GD.

In the end of the article the author mentions:

  We begin with a bunch of documents to train on. We then represent each
  documents as numerical vectors, and you can choose to split them into
  n-grams and weigh these n-grams with TF-IDF. Now that we have a
  numerical representation, we can give the program training data and
  make it guess each time. Based on how wrong the program was, the
  program adjusts to its mistakes to make better prediction the next
  time using Gradient Descent. It keeps track of its learning using SVMs
  and make predictions with the SVM model by “plotting” the new
  datapoint in the graph and seeing if it is on the right or left by the
  decision boundary.

The article does not mention about SGD, but from internet research I understand that SGD is a variation of GD where-in the parameter is tweaked immediately at each iteration, rather than at the end of the entire training set.

So to me it appears that SVM is the actual classifier and GD (or SGD) is the function that tells the classifier how correctly it has done the prediction based on which the SVM classifier improves itself.

Coming back to my original question: Is my understanding right that SGD is not the classifer?

Yes,gradient descent (GD) and Stochastic Gradient Descent(SGD) are Optimization functions.Objective of a Optimization function is  to give the best combinations of parameters for our algorithm which minimize the error.Support Vector Machine is the Classification algorithm and GD/SGD are the optimization functions use to find best optimal hyperplane(separator).

Gradient Descent’s cost-function(total error of the training set) iterates over ALL training samples

Stochastic Gradient Descent’s cost-function only accounts for ONE training sample, chosen at random


How to explain decision tree algortihm in layman's terms? I have a task at hand, where I have to explain decision tree algorithm to a person who has not much understanding of machine learning. I have been looking around, but find it difficult to explain the algorithm in layman's terms, so that a person will understand what is happening in the process.

What is the best way to describe the algorithm, possibly using some very simple basic example, so that the whole process of the algorithm is broken down to simple steps for understanding?

I learn well by examples, so perhaps this will be helpful:

Lets say you are a resort considering your next advertizing campaign. You will be sending out 1,000 vouchers that give two free days at the resort, with the hope that the customer will decide to extend it to a full week (who want's a two day vacation?!).

You have a rather extensive database of customer data, including where they are from, income range, duration of stay(s) and money spent at each stay, number of children, marital status, and employment status (working, unemployed, retired). You would like to create a decision tree to help select the customers to send the vouchers to (specifically, those who will spend at least $200 per day and stay at least one week.

To create the tree, a computer will perform the following steps:

From the above list of customer data, find the property that best separates the customers into two groups.
Repeat step one for each of the above groups using the remaining properties.
In the end, you will have a tree where at each point, you can make one of two decisions. Following a path leads to a decision. The split points will be chosen to maximize the probability of a correct classification.


Machine learning algorithm for ranking I am working on a ranking question, recommending k out of m items to the users. The evaluation metric is average precision at K.

Both R and Python have xgboost can be used for pairwise comparison and can be adapted for ranking problems.

Are there other algorithms or approaches that can be applied to ranking problems?

Your problem sounds like the classical top-N personal recommendation to me. There're lots of possibilities in the literature, for example:

User-based collaborative filtering
Content-based collaborative filtering
Matrix factorisation

You'll need to read the literature and figure out which one is better for you.


Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass? I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the source language and partial/incomplete translation in the target language, the Transformer is tasked with predicting the next token in the translation. For example:

English (source): I love eating chocolate

Spanish (target): Yo amo comer ...

The next token in the translation should be chocolate (thus, the full translation would be "Yo amo comer chocolate"). So, in this example, the encoder would process the sentence "I love eating chocolate", the decoder would process the partial translation "Yo amo comer" and the final output would be a softmax over the whole vocabulary (hopefully with chocolate being the token with the highest score).

The issue is, during training we want the Transformer to learn the full translation. This means that if the target sentence has length N, we want the transformer to predict the first word, the second word, the third word, and so on all the way up to the N-th word of the target sentence. One way to do that is by generating N training instances (one for each word of a target sentence of length-N). However this approach is computationally quite expensive, because instead of having a single training instance per source-target pair, we now have as many training instances as there are words in all the target sentences. So I was wondering if it's possible make all predictions for all words in the target sentence in a single forward pass. Is that possible?

In "Transformer model", during the training, all the predictions are computed in one step. That's one of its strength points.

It is during inference that the target sentence is generated sequentially invoking a beam search (usually with a small beam width).

During training (emphasizing on training) just use target word indexes as inputs to the decoder, you just need to trim the index of  token from end of target sequence.


Getting feature vectors from CatBoost pool I have a dataset with some numerical and categorical features and I am trying to apply CatBoost for categorical encoding and classification.

Since my dataset is highly imbalanced, with a large number of data samples with label 0 compared to those with label 1, I'm also trying to use SMOTE to synthesize label 1 data samples before CatBoost classification.

My code -

# train_categorical_features is a list of columns that have categorical values
train_pool = Pool(data = X,
                  label = y,
                  cat_features = train_categorical_cols)

X_enc = train_pool.get_features()
print(X_enc)
y_enc = train_pool.get_label()
print(y_enc)

smote = SMOTE()
X_res, y_res = smote.fit_resample(X_enc, y_enc)
print('Resampled dataset samples per class {}'.format(Counter(y_res)))

predictions = []
for i in range(10):
    clf = CatBoostClassifier(learning_rate=0.08,
                         depth = 10,
                         loss_function='Logloss',
                         l2_leaf_reg = 4,
                         iterations=1000,
                         task_type="GPU",
                         random_seed=i,
                         logging_level='Silent')
    clf.fit(train_pool, plot=True,silent=True)
    predictions.append(clf.predict_proba(test[inputcols])[:,1])
    print(clf.get_best_score())

I get an error on X_enc = train_pool.get_features() that says -

  CatBoostError: Pool has non-numeric features, get_features supports
  only numeric features

My questions are -

Is my approach towards applying SMOTE with CatBoost correct?
I've diligently followed the catboost documentation, and I am not able to understand or fix the error I've mentioned above. Would appreciate any help.

The reason you are getting the error is that your pool contains categorical features. If all of your features were numerical, it would work fine.

SMOTE is irrelevant here.


Epochs or Loss Convergence I was training a Deep Neural Network using transfer learning and the loss value as of now is close to 1.7. I am using a dataset of 31,000 images and I do apply data augmentation techniques to help the model generalize better. However, I had confusion regarding epochs and loss values. Should I be training the model for say 50 epochs for good accuracy or would it be better if I trained until the loss value is as low as possible which will be very close to zero. Which one is recommended to get the best accuracy? Will more epochs cause the model to perform better than the lowest loss value? Any help would be greatly appreciated. Thanks in advance!

I think it is better to use early stopping  method.Too many epochs will lead to overfitting the model.Early stopping allows you to stop the training once the model performance stops improving on a hold out validation dataset.

refer this link to the concept

https://en.wikipedia.org/wiki/Early_stopping


What approaches could be used to teach a model to make appropriate brush strokes to reproduce a painting? I'm pretty new to ML and most of the supervised techniques I've discovered so far are along the lines of

Pass model training set inputs and get it to generate predictions
evaluate predictions (predicted 'y') against real results (actual 'y') for the training set
Perform some kind of gradient descent to tweak the model parameters to give outputs that are closer to the real results

The precise method of training can vary, but the common theme in the examples I've seen so far is that the outputs of the model are the actual things that we want to get out of the system. For example, we might be trying to predict house prices - in this case, an output of the model would actually be a predicted price, and it's that that I would care about as part of my cost function and as a user of the system.

I'm imagining a scenario where we have a robot that's able to hold and use a palette and paintbrush, and we want it to copy existing paintings. (Strictly theoretical - I'm not trying to forge anything here!).

What I imagine I'd want is to

present robot with example input (existing paintings)
have the robot run a model where the output is some brush motions
evaluate the painting resulting from the brush strokes against the original painting

The difference here is that the output of my model is 'brush strokes', but that output (from the model) isn't ultimately what I care about - it's what's rendered by those brush strokes. I don't mind if the robot makes the same strokes that the original artist made, but I do mind about the similarity of what's rendered by those strokes to the original.

So I am wondering what areas of ML I need to look into to find something that works this way, where our cost or fitness is evaluated based on some transform of the model output, rather than the model output itself. I tried searching for examples of where a transform of the model output was incorporated into the cost function, but I couldn't see that it was a common technique.

Main Idea:

The main idea is the you could measure how good is the output and the input of your model simply representing it by some mathematical object, for example a vector in a high dimensional space and then using some error function to measure the fitness (how good your model is performing).

A possible formalization

The scenario your describing fits in a more general optimization function, the most general optimization scenario is;

$$ min_{x \in X} f(x) $$

where $X$ is the domain. In the case of image processing and computer vision $x$ is a matrix of pixels, this matrix of pixels can be represented by a vector $x \in \mathbb{R}^{m \times n}$. In the case you describe you could also represent the input (painting) as a vector $y\in \mathbb{R}^{m \times n}$.
Then $f$ coul be RSME between some $x$ and $y$, i.e. $f(x) = {\| y - x \|}_2$. Notice that $y$ is given and it does not change.

Now the key question here is how to represent a stroke, or more formally a function the values of $x$. A very basic idea can be represent the stroke by the following tuple $(o, e, t, c)$ where $o$ and $e$ are two dimensional points, the origin and the end of the stroke; $t$ is the thickness of the stroke and $c$ is the color of the stroke. Let $S$ be the space of all the strokes, assuming the thickness and colors are finite, S is finite. The possible pairs of $o$ and $e$ are finite namely all the pairs of possible points in $m \times n$. Now let $S^k$ be the composition of $S$ with itself $k$ times.

Finally the problem can be represented the following way:

$$\min_{s \in S^k}{\| y - t(s, x) \|}$$

Where $t$ is the function that applies $s$, that is a number of different strokes to an empty canvas $x$, by empty canvas I mean a matrix full of zeros.

Summarizing the ideas expose above are a possible formalization of your problem, of course is not the only one. The main issue is that the robot does not "learn" is simply computes the optimal set of strokes to apply.

Other ideas

Another possible approach is to use reinforcement learning, this learning paradigm does not needs labeled training samples, it relies in a reward function $R$ to guide the learning process. Also if defines a set of states and actions the agent can be or do. The states could be the strokes and the actions add another stroke, the reward function can be a variation of the error function. Also I am far from being an expert in reinforcement learning I guess is worth looking into it.

A proven approach

The problem you describe has been demonstrated in real life. See this paper: Feedback-guided Stroke Placement for a Painting Machine. The paper demonstrates a system that describes your scenario, also see this site for more work of the authors or this for the site of the project.


How to use a loss functions closed-form gradient even when it does not have a closed-form itself I have a loss function without a closed analytic form that I want to implement in tensorflow that has a nice clean gradient (i.e. i have some nice gradient function $f(x)$ that i want to use for backprop but $\int_x f(x) \ dx$ is with no closed form)

Ofcourse i wouldnt be able to evaluate the loss, but i want to use $f(x)$ in the optimization process. How could this be accomplished?

TensorFlow allows for custom gradient functions with tf.custom_gradient.

You could write a decorator that would return specific gradient values for specific values of x. The loss function would not need to be evaluated.


How to calculate growth function for a threshold function I'm working on a homework problem but don't fully understand it. The problem and solution:

I don't understand the definition of the threshold function.

Does it mean to pick one feature and classify the point based on that one feature?

It's the only way I can think of to explain the solution, $N$ ways to pick a feature, for each feature there are $m+1$ ways to select the threshold.

Yes, your interpretation is correct.

Each member of $H$ is one such function, they are parametrized by $i$, the feature selected and $\theta$, the chosen threshold. Different $\theta$ might corresponds to the same function but effectively there are only $m+1$ such function for each $i$. Hence, the set $H$ consists of at most $(m+1)N$ elements.


Distractor Generation for Multiple Choice Questions I'm currently working on generating distractor for multiple choice questions. Training set consists of question, answer and 3 distractor and I need to predict 3 distractor for test set. I have gone through many research papers regarding this but the problem in my case is unique. Here the problem is the questions and answers are for a comprehension(usually a big passage of text story) but the comprehension based on which is not given nor any supporting text is given for the question. Moreover, the answers and distractor are not a single word but sentences. The research paper I went mostly worked with some kind of support text. Even the SciQ dataset had some supporting text but the problem im working is different
This research paper was the one which I thought closely went by what I wanted and I'm planning to implement this. Below is an excerpt from the paper which the authors say worked better than NN models.

We solve DG as the following ranking problem: Problem.
Given a candidate distractor set D and a MCQ dataset M = {(qi , ai , {di1, ..., dik})} N i=1, where qi is the question stem, ai is the key, Di = {di1...dik} ⊆ D are the distractors associated with qi and ai , find a point-wise ranking function r: (qi , ai , d) → [0, 1] for d ∈ D, such that distractors in Di are ranked higher than those in D − Di.

My questions are a) From what I understood, The above lines says we first create a big list containing all the distractors in the dataset and then we create a pointwise ranking function with respect to all distractors for every question? So if we have n questions and d distractors. We will have a (nxd) matrix where pointwise function values range between o and 1. Also, a question's own distractors should be ranked higher than the rest. Right?

To learn the ranking function, we investigate two types of models: feature-based models and NNbased models.
Feature-based Models: Given a tuple (q, a, d), a feature-based model first transforms it to a feature vector φ(q, a, d) ∈ R d with the function φ. We design the following features for DG, resulting in a 26-dimension feature vector:

Emb Sim. Embedding similarity between q and d and the similarity
between a and d.
POS Sim. Jaccard similarity between a and d’s POS tags.
ED. The edit distance between a and d.
Token Sim. Jaccard similarities between q and d’s tokens, a and d’s tokens, and q and a’s tokens.
Length. a and d’s character and token lengths and the difference
of lengths.
Suffix. The absolute and relative length of a and d’s longest
common suffix.
Freq. Average word frequency in a and d.
Single. Singular/plural consistency of a and d. This
Wiki Sim.

My questions: Will these feature generation idea applies to both word distractors and sentence distractors? ( As per the paper, they claim it will).
Apart from all of these, I have other simple questions such as should I remove stopwords here?
I'm new to NLP. So any suggestions about which SOTA implementation would work here would be very helpful. Thanks in advance.

How are you planning to create the distractor in the first place?

I can't comment because I don't have the repo to do so.


XGBoost: Quantifying Feature Importances I need to quantify the importance of the features in my model. However, when I use XGBoost to do this, I get completely different results depending on whether I use the variable importance plot or the feature importances.

For example, if I use model.feature_importances_ versus xgb.plot_importance(model) I get values that do not align. Presumably the feature importance plot uses the feature importances, bu the numpy array feature_importances do not directly correspond to the indexes that are returned from the plot_importance function.

Here is what the plot looks like:

But this is the output of model.feature_importances_ gives entirely different values:

array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00568182,  0.        ,  0.        ,  0.        ,
        0.13636364,  0.        ,  0.        ,  0.        ,  0.01136364,
        0.        ,  0.        ,  0.        ,  0.        ,  0.07386363,
        0.03409091,  0.        ,  0.00568182,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.00568182,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.00568182,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.01704546,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.05681818,  0.15909091,  0.0625    ,  0.        ,
        0.        ,  0.        ,  0.10227273,  0.        ,  0.07386363,
        0.01704546,  0.05113636,  0.00568182,  0.        ,  0.        ,
        0.02272727,  0.        ,  0.01136364,  0.        ,  0.        ,
        0.11363637,  0.        ,  0.01704546,  0.01136364,  0.        ,
        0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
        0.        ,  0.        ,  0.        ], dtype=float32)

If I just try to grab Feature 81 (model.feature_importances_[81]), I get:0.051136363. However model.feature_importances_.argmax() returns 72.

So the values do not correspond to each other and I am unsure about what to make of this.

Does anyone know why these values are not concordant?

In xgboost 0.7.post3:

XGBRegressor.feature_importances_ returns weights that sum up to one.
XGBRegressor.get_booster().get_score(importance_type='weight') returns occurrences of the features in splits. If you divide these occurrences by their sum, you'll get Item 1. Except here, features with 0 importance will be excluded.
xgboost.plot_importance(XGBRegressor.get_booster()) plots the values of Item 2: the number of occurrences in splits.
XGBRegressor.get_booster().get_fscore() is the same as XGBRegressor.get_booster().get_score(importance_type='weight')

Method get_score returns other importance scores as well. Check the argument importance_type.

In xgboost 0.81, XGBRegressor.feature_importances_ now returns gains by default, i.e., the equivalent of get_score(importance_type='gain'). See importance_type in XGBRegressor.

So, for importance scores, better stick to the function get_score with an explicit importance_type parameter.

Also, check this question for the interpretation of the importance_type parameter: "weight", "gain", and "cover".


Data Annotation: "labeling" target vs features I understand how one would use a data annotation tool to label targets for a given sentence, for example though, I'm not clear on how placing labels on features can be used to improve model performance.  For example, in this text annotation tool , you can add "labels" to a body of text like person, location, event ...etc .  Given that you must create Word Embeddings to work with the data, and the vector representation is not human-readable, how would you be able to improve model performance by annotating feature variables?

Those labels are not primarily for features, those labels are primarily for targets. Person, location, and event for targets for named-entity recognition (NER).


How can different classification algorithms expressed as neural networks? I have heard that each of the different classification algorithms can be expressed as neural network architecture. How can the different algorithms like Logistic Regression, SVM(Support Vector Machine), ELM(Extreme Learning Machine) be expressed as a neural network? Is there a way to convert the vector equation of a classification algorithm into neural network architecture?

I have heard that each of the different classification algorithms can be expressed as a neural network architecture.

The Universal Approximation Theorem
guarantees that neural networks can approximate any function on a closed subset of $\mathbb{R}^n$.  So we are theoretically guaranteed that any function specified by a logistic regression model, SVM, ELM, or any other model can also be expressed by a neural network.

  Is there a way to convert the vector equation of a classification algorithm into neural network architecture?

This part may be hard depending on the model.  Although the universal approximation theorem guarantees that an equivalent network exists, it offers no information about the network's weights or number of hidden units.  Nonetheless, let's give it a shot for the three models you mentioned:

ELMs

This one is easy.  ELMs are already neural networks, so there's nothing left to be done.

Logistic Regression

This one is also pretty straightforward.  For simplicity, let's consider a binary classification problem.  A logistic regression model can be written as
$$
p(Y=1) = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}}
$$

Where the expression $p(Y=1)$ means "the probability that the class is positive".  The input features are ($x_1$, $. . .$ $x_n$) and they have corresponding weights ($w_1$, . . . $w_n$).

We can represent this function with just a single neuron!  Suppose we have a neuron with $n$ inputs, weights ($w_1$, . . . $w_n$), and bias term $w_0$.  The output of the neuron (with no activation function) is given by
$$
z = w_0 + \sum_{i=1}^{n} w_ix_i
$$

Applying a sigmoid activation function to $z$ results in the exact same expression as we had above for the logistic regression model:
$$
S(z) = \frac{1}{1+e^{-z}} = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}} = P(Y=1)
$$

Here's a diagram showing the neuron:

The situation would be more complex for multinomial logistic regression, but you get the idea.

SVMs

This one is probably the trickiest of the three, since an SVM's kernel function can be non-linear.  Again, let's consider a binary classification problem.

A linear SVM is specified by the maximal-margin hyperplane between the two classes, which can be written as $\vec{w} \cdot \vec{x} - w_0 = 0$, where $\vec{w} = \langle w_1, . . . w_n \rangle$ and $\vec{x} = \langle x_1, . . . x_n \rangle$.

Of course, the actual classification is determined by which side of the hyperplane a point lies, so the classifier can be written $f(\vec{x}) = sign(\vec{w} \cdot \vec{x} - w_0)$, with 1 denoting the positive class and -1 denoting the negative class.

As with logistic regression, it's pretty easy to represent the decision boundary with a single neuron.  A neuron with inputs $\vec{x} = \langle x_1, . . . x_n \rangle$ with weights $\vec{w} = \langle w_1, . . . w_n \rangle$ and with bias term $\hat{w_0} = -w_0$ describes the decision boundary of a linear SVM.  This is the same neuron as the one associated with logistic regression, except there is no activation function and the bias term has its sign flipped:

However, it's exceedingly common for SVMs to use the kernel trick to find non-linear decision boundaries.  When the kernel is non-linear, the translation of an SVM to a neural network is not so straightforward.  I imagine you could find an equivalent network by using the kernel function as the activation function for each neuron, but off-hand I can't describe exactly how it would work.

Hopefully that helps!


Methods for ensembling ranked lists? I was wondering if there's a good way to use ensembling when I have two or more algoritims producing ranked lists.  That is, suppose I have the following datasets consisting of ordered lists (higher to the top means more relevant):

Method1_Rankings  Method2_Rankings GoldStandard_Rankings
item1             item2             item1
item3             item1             item3
item2             item10            item5
...

Is there a way to optimally combine methods 1 and 2 (e.g., give the rankings some weights or similar)?  Thank you.

In this case, you might want to convert these into pair-wise relationships (e.g. item1 < item3), put together what you get from the different methods, and find a ranking which agrees with them the most.

You can look at the answer in the comment for some ideas - I would also suggest another paper that might help you: Aggregating inconsistent information: ranking and clustering


Why we use Pickle instead of saving the weights in HDF5 format in Deep Learning I have read that, Pickle library is used to save trained model into a .pkl file to use it later.
Also, we can save the weights of a trained model into a hdf5 format using
model.save()

and use those weights afterwards.
So why we use pickle instead of hdf5 files?

Avoid pickle, there are a couple of problems
Even in keras documentation they say
"It is not recommended to use pickle or cPickle to save a Keras model."
Also look at this solution.


Detecting synchronization cascades in time series I am researching delayed synchronisation in a system of coupled oscillators. There is a one-way causal connection between the oscillators, which leads to the synchronisation events occurring in a rough sequence.
When I plot this data a very clear pattern emerges:

I am looking for a way of identifying and fitting a line to the highlighted areas of the graph. Do you have any ideas on how to find these patterns?

One option is:

Threshold at ~10 to remove the baseline values.
Fit a clustering algorithm to identify groups.
For each cluster, fit a linear regression. If the linear regression has certain a slope label cluster as a synchronization event.


comparison of t-SNE and PCA and truncate SVD How to compare the trucate SVD ,PCA, and T-SNE?

What we can say about features if  t-SNE and PCA and truncate SVD digaram is in this figure?

PCA and truncate SVD do not differ much, since they are based on
the same theory that the eigenvectors with the less eigenvalue are
discarded. As mentioned here the difference:

TruncatedSVD is very similar to PCA, but differs in that it works on
sample matrices  directly instead of their covariance matrices. When
the columnwise (per-feature) means of  are subtracted from the feature
values, truncated SVD on the resulting matrix is equivalent to PCA. In
practical terms, this means that the TruncatedSVD transformer accepts
scipy.sparse matrices without the need to densify them, as densifying
may fill up memory even for medium-sized document collections.

T-SNE attempts to find the underlying structure of the data by taking
into account the neighbors of a sample. By prioritizing the neighbor
points gives an advantage when data structure is a manifold.
A way to understand the meaning of your figure clusters is described here,
using radar charts. From my point of view, what i would say is that

The information, which predicts the labels Fraud/not Fraud, is not
on    a structure manifold
The samples seems to be linear separated and and that the main information is given by only two features. If you wanted, you could check how much of the eigenvalues energy is retained when keeping only two features