What is the MapReduce application master? From Hadoop The Definitive Guide

  The whole process is illustrated in Figure 7-1. At the highest level,
  there are five independent entities:

  • The client, which submits the MapReduce job.

  • The YARN resource manager, which coordinates the allocation of
  compute re‐ sources on the cluster.

  • The YARN node managers, which launch and monitor the compute
  containers on machines in the cluster.

  • The MapReduce application master, which coordinates the tasks
  running the Map‐ Reduce job. The application master and the MapReduce
  tasks run in containers that are scheduled by the resource manager and
  managed by the node managers.

What is the MapReduce application master?

In a MapReduce program written in Java, we need three things: a map function, a reduce function, and some code with main() function to run the job. Is  the MapReduce application master the code with main() function to run a map reduce job?

Thanks

MapReduce Application Master coordinates the tasks running the MapReduce job. It is the main container for requesting, launching and monitoring specific resources. It negotiates resources from the ResourceManager and works with the NodeManager to execute and monitor the granted resources.

For MapReduce program, you do not need map and reduce functions. You can have map-only jobs and reduce-only jobs.

MapReduce Application Master appropriately implements the code in the main().


Best way to build a wave classification system I want to make a classifier for waves such as following:

Above image is from: http://www.invisiblesbook.com/equal-temperament-tuning/

I believe, I will have to extract features from raw input using FFT or some other method and subject that to a CNN. What is the best way to go about building classifier for such waves? I am not particularly asking about neural-networks; any other method which is accurate and fast will also be of help.

I don't work in this field, however, recently I have read a magazine article (in German) about this. The article explained how  one of the most well-known companies who does this, Shazam, works.

Building a spectrogram, hashing, ???, indexing.

Video, 8:50: How Shazam Developed Their Signature, Awesome Algorithm | Shazam Founder Chris Barton
Whitepaper, from 2003, 7 pages, very technical
github repo from an open-source implementation: https://github.com/itspoma/audio-fingerprint-identifying-python See the READme file for more links


What are the limits of capping ratios to [0-1]? I work with some financial ratios. By construction those variables should be between 0 and 1. However, for some reasons I end up with values <0 and >1. These reasons include :

reporting problems (fat finger errors, unit problems...)
discrepencies between databases
discrepencies due to accounting reassessment

What would be the reasons for me to not cap my ratios to [0,1] ?

In essence, you risk biasing the model by making the values 0 and 1 more likely than they actually are. This may or may not be a significant problem; really only cross-validation will tell you.

I would go ahead and try it, but compare your results to a different method of filling invalid values, like using the mean() of the valid values plus an extra feature flagging the samples that were invalid. Make sure you cross-validate using a hold out dataset that's been transformed in the same way; you can then select the method that results in the best results of the cross-validation process.


depth/complexity of decision trees I have used the same methods/parameters to create two decision trees. The trees classify the presence or absence of a medical condition using the presence or absence of various symptoms. There is a tree for Medical Condition #1 and another tree for Medical Condition #2. Both trees are based on the same set of symptoms, rated by patients. If Medical Condition #1 resulted in a much simpler tree than Medical Condition #2, can that suggest Medical Condition #2 is a more complex disease? If so, can anyone point me to a reference that states the complexity/depth of a tree can be representative of a complex condition?

The goal of my analysis is separate from this question. I am trying to figure out all the conclusions I can draw or suggest from my analysis. Yes, I am interested in saying something about the complexity of condition A to B.
When is a condition complex ? When there are many symptoms as the disease is diagnosed by the symptoms.
If it's hard to diagnose ? YES
if the symptoms are severe ? NO
can s.o. have both conditions ? NO

No, you can't infer that.
I assume you have the same training set with the same predictors (symptoms). The only difference in the training set is the binary class label for each patient.

The smaller tree just means:
- with the given symptoms, it might be easier to distinguish between the people who have the condition 1 and the ones not having it. (since you have the same certainty with fewer information)
This distinction seems harder with medical condition 2, that's way you have to consider more symptoms to be pretty sure about your classification. So if condition 2 is a very mild condition where it is hard to diagnose even for an expert if someone has it then it would result in a deep tree.


Adding more layers decreases accuracy I have my ANN trained on MNIST dataset. Hidden layer has 128 neurons and input layer has 784 neurons. This gave me an accuracy of 94%. However when I added one more layer with 64 neurons in each then the accuracy significantly reduced to 35%. What could be the reason behind this.

Edit : Activation function : sigmoid. 521 epochs.

The reason is that by adding more layers, you've added more trainable parameter to your model. You have to train it more. You should consider that MNIST data set is a very easy-to-learn dataset. You can have to layers with much less number of neurons in each layer. Try $10$ neurons for each to facilitate the learning process. You can reach to $100%$ accuracy.


Supervised learning for variable length feature-less data I have data in following form:

a: 1,2,3,2,3
a: 2,4,5,6,7,8,0,9,7,6,5,6,2
a: 7,8,9,3,4
b: 4,5,3,5,6,3,5,1,2
b: 1,6,3,2,4,5
b: 2,4,5,6,7,8,0,9,7,6,5,6,2
c: 7,8,9,3,4
c: 4,5,3,5,6,3,5,1,2
...

(in reality, each case has about 100-200 numbers, though the length is variable)

Here, a, b and c are groups (their number is fixed - taken as 3 here) and the numbers indicate a vector associated with each case. How can I apply supervised machine learning with such a data so that if I get a new series of numbers, e.g. following:

3,2,3,4,1,5,6

I should be able to determine which group (a, b or c) does this case belongs to.

Following features of each list of numbers may be important:

length of series
mean value of series
variance of series
maximum of series
minimum of series
type of distribution of series (normal or non-normal)

How can I apply machine learning methods to such data. Thanks for your insight.

This is a typical scenario in language processing, e.g. if you want to read a product review representing each word with a number and output a 1-5 star rating.

You can use a recurrent many-to-one RNN/GRU/LSTM network: After feeding each word it generates a feature vector which can be fed into a classifier.
https://en.wikipedia.org/wiki/Recurrent_neural_network


Load keras model in Java What are the requirements to load the trained model by Keras in Java?

I checked that DeepLearning4J supports Keras models, network architecture and weights can be easily loaded. The only cons are probably that we need to use ND4J backend or it does not matter?

If there is a created model using keras and tensorflow, what is the best way to load it in Java ecosystem?

I tried to use frozen graph script to save tensorflow model, but it cannot be loaded with SavedModel.load. Loading model as a Tensor Graph object seems to be common way, but available examples in Java are chaos especially when input is an image.

Giving you some links which have worked:

deploying-keras-deep-learning-models-with-java :  this uses DeepLearning4J, ND4J which supposedly you have tried already.

tensorflow-keras-java

And same question is asked here - converting-keras-in-python-to-java . It uses KerasModelImport.importKerasSequentialModelAndWeight

You should give HDF5 file operation java examples: hdf-java . It would be helpful if you want to change the weights yourself

Please comment if you have already tried all these already.


Owen Zhang's slides: what does the "time" mean? Here is one of Owen Zhang's slideshares. At page 12, what does the time mean?

Here's my understanding:

I think it's the explanation of page 10, and time is the same as the "time" in page 10, which means the time axis in real life.

In page 10, it shows that:

hold out data should be out-of-time, we can not do a random shuffle because it will not represent the real situation our model will be faced.
unless data is extremely small

As the slide has said before:

  When you looked at your validation result for the Nth time, you
  are training models on it

In this case: validation data = hold out data = Public LB (if my understanding is correct)

The first figure of page 12 is the normal condition, we shouldn't check our Public LB to tune our model(or we're training model on it). So the answer is no.

The second figure is the small data condition, we have to train model using the feedback from Public LB, because of the data limitaion. So the answer is yes.

The third figure shows that Public Data is the same as Private Data, they both represent something under same time period, sharing the same "reality", thus we can use the feedback from Public LB.

The forth figure shows that Public Data is reality, and Private Data is more "realistic" than Public Data. It's the "hold out data should be out-of-time" case, the answer is yes.


What's the correct reasoning behind solving the vanishing/exploding gradient problem in deep neural networks.? I have read several blog posts where the solution to solve the vanishing/exploding gradient problem in a deep neural network is suggested to be using Relu activation function instead of tanH & sigmoid.

But, I have encountered an explanation by Prof. Andrew NG lecture that explains that a partial solution to the vanishing gradient problem is a better or more careful choice of the random initialization of weights in your neural network.

i.e the solution is:

To set the variance of Wi to be equal to 1/n, where n is the number of input features that are going into a neuron. Along with the assumption that the input features of activations are roughly mean 0 and standard variance 1. So, what it's doing is that it's trying to set each of the weight matrices w so that it's not too much bigger than 1 and not too much less than 1, therefore, it doesn't explode or vanish too quickly.

So, if you are using a ReLu activation function then setting the variance of Wi to be equal to sqrt(2/n) works better**.
and if you are using a TanH activation function then setting the variance of Wi to be equal to sqrt(2/n) works better.
or in some cases, it's being suggested to use Xavier initialization
Also, if we need we can tune of variance parameter as another hyperparameter by multiplying into the above formula and tune that multiplier as part of your hyperparameter search.

Therefore, choosing a reasonable scaling for how to initialize the weights helps weights not to explode too quickly and not decay to zero too quickly, which in turn could help in training a reasonably deep network without the weights or the gradients exploding or vanishing too much and not simply using ReLu!.
Please correct me if my understanding is wrong or incomplete!

I think the two aspects you mention are two faces of the same medal: if your weights are too high or low, the activation of a layer ends up being too high or low. If you use $tanh(z)$ or $sigmoid(z)$ as activation functions, you'll end up with values that will have a derivative almost to zero, and the gradient will vanish as it is back-propagated into the network.
By using $relu(z)$, instead, you use a linear function for $z>0$ and therefore the derivative is constant as the activation grows. This helps the gradient which does not approach zero while it is back-propagated.

This works in every case: both if you use weights that are too high or too small.
Tanh and sigmoid are similar in the sense that they are basically linear near $z=0$, but almost flat when $|z|>>0$, ReLU solves the problem in the latter case, because it becomes linear for any $z>0$.

On the other hand, another strategy is to prevent $z$ from being too large or too small, and therefore you attempt to keep it near the origin ($z=0$). In this way, both tanh and sigmoid are linear, as ReLU would have been for any $z>0$.

The expected result is the same: the derivative of the function is almost linear, and therefore the gradients can traverse the network without vanishing or exploding.

Of course, there are other factors here that must be considered (e.g. ReLU clamps to 0 every negative value, which has other sort of effects), but beside these, in both cases you get a similar outcome (non-vanishing gradients) with different approaches (by tweaking the activation function or by tweaking the weights).


Monitoring machine learning models in production I am looking for tools that allow me to monitor machine learning models once they are gone to production. I would like to monitor:

Long term changes: changes of distribution in the features with respect to training time, that would suggest retraining the model.
Short term changes: bugs in the features (radical changes of distribution).
Changes in the performance of the model with respect to a given metric.

I have been looking over the Internet, but I don't see any in-depth analysis of any of the cases. Can you provide me with techniques, books, references or Software?

The changes in distribution with respect to training time are sometimes referred to as concept drift.
It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this.
In terms of research, you may want to take a look at the scientific production of João Gama, or at chapter 3 of his book.
Regarding software packages, a quick search reveals a couple of python libraries on github, like tornado and concept-drift.
Update: recently I came across ML-drifter, a python library that seems to match the requirements of the question for scikit-learn models.


Question about Logistic Regression I'm studying Logistic Regression, there is a thing that I'm not able to understand.

I know what the Logistic Function is, I studied it. But I don't understand why the logistic function is equal to the dot product between $w$ and $x$.

$$ \log \left(\frac{p}{1-p}\right)=\bar{\omega }\cdot \bar{x} $$

For me these are two completely different concepts, I don't understand why are related in this manner. Can you help me?

It is just a matter of modelling.

Note that $w \cdot x = \sum_{i=1}^n w_i x_i$.

For example in linear regression (ignore intercept term for now, we can always insert the intercept term by including $1$ as a feature), we try to fit a model of the type of $y(x) = \sum_{i=1}^n w_ix_i$, however, the problem with this when the dependent variable is that $y$ can take values beyond $0$ and $1$.

Hence, we consider another model $F(p) = \sum_{i=1}^n w_ix_i$ where $F(p)$ can take any real value and $F$ is a function which takes value from $(0,1)$ and be able to map it to $\mathbb{R}$. $F(p)$ can be chosen to be $\log \left( \frac{p}{1-p} \right)$, which is an increasing function which map $(0,1)$ to $\mathbb{R}$.


Train object detection without annotated data/bounding boxes From what I can see most object detection NNs (Fast(er) R-CNN, YOLO etc) are trained on data including bounding boxes indicating where in the picture the objects are localised.

Are there algos that simply take the full picture + label annotations, and then on top of determining whether an image contain certain object(s) also indirectly

learn to understand the appropriate bounding box(es) for objects?

Yes, there are models that do this. This link points to one of the first papers I believe. The main idea is called weakly supervised object detection.

The paper essentially makes three modifications.

They treat the typical hidden fully connected layer as a convolutional layer. This works because convolutional layers can be thought of as convolving the same fully connected network about the image.
They add a global max pooling later at the end of this convolutional layer. This is the operator that will "highlight" the area of this final conv layer that has learned the pattern of objects it is trying to classify. Using a threshold on the weights of this global max will ensure a region is significant. Then, they use an algorithm to create a bounding box from this region.
They suggest a new loss function that lends itself to an object existing or not. I think they assume a Bernoulli for each class which lends itself to multiple logistic regression instead of softmax.

Take a look because it's pretty sweet and has been cited by a lot of other new exciting papers.


Determine useful features for machine learning model I am working with a dataset with hundreds of features. I wish to create a simple machine learning model using 7-10 features from the original dataset. My question is this:

What quantitative metrics can I use to determine that a feature will be useful to the learning model?

I have been comparing the distribution of the target mean over the feature groups, to the target mean of the dataset as a whole. For example, take a binary feature X and a binary target. Let's say the target has a mean of 0.10 when taken over the entire dataset.

To analyze the feature X, I take the target mean for each group within feature X.

mean (X=0) = 0.07
mean (X=1) = 1.15

In this way, I can observe the effect of a feature on the target.

I know that there must be some stronger metrics which people use to determine the strength of a feature. In school I used p tests to determine the statistical significance of a variable. Is there an analog in DS/ML?

I suggest taking a look at this page for some more ideas:

Feature Selection

That being said a couple of ideas that come to mind quickly, is to:

use a tree based method (like Random Forest) and look at your feature importances. Scikit Learn has a handy class for doing just that see the link above.
Use some sort of regularization/penalty like L1 or L2 regularization. That will force non-useful features to have parameters close to zero.
Recursively remove variables and see what the resulting output is and cross-validate. Again sklearn has a method for this.

Generally, these methods will be "expensive" as you are fitting multiple models to get you where you need to go.


Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript? This question is from An introduction to RL, page 78. In the formula below the page, both

  $\mathbb{E}$ and $\mathbb{E_\pi}$

are mentioned. Could you help me understand the difference between these two in this page and in general?

In general, the expectation is taken with respect to some random variable X. Often, when dealing with a single random variable, it can be implicitly inferred over which random variable it is being integrated and thus writing $\mathbb{E}$ suffices. However, when dealing with multiple random variables, this is not the case anymore. Then, a subscript denotes with respect to which random variable the expectation is taken.

However, the subscript can also denote on which random variable to condition on. This seems to me to be the case on the page you are referring to. Here, $\mathbb{E}_{\pi'}$ means that you are conditioning on actions which are distributed according to $\pi'$.

(Just as a side note: sometimes you might see something along these lines of: $\mathbb{E}_{a\sim \pi'}$, the subscript here is the actual notation for "$a$ distributed according to $\pi'$")

For a more technical answer, have a look at this question on Cross Validated.


Clustering of multi-label data The dataset consists of

1) a set of objects and

2) a set of labels, which are used to describe the objects.

For the moment, for simplicity sake, each label can be marked as either true or false (In a more complex setup, each label will have a value of 1-10).

But, not all the labels are actually applied to all the objects (in principle, all the labels can and should be applied across all the objects, but in practice, they just are not). Also, when a label isn't applied to an object, one cannot simply assume that the label's value for that particular is false. Therefore, the missing labels will be ignored in the model.

I need to cluster the objects based on their labels.

Any tips on how and what algorithms to use will be appreciated.

It is possible to cluster the objects based on their labels by treating the labels as features. Typically, labels are treated as targets which would frame the problem a supervised machine learning problem.
Since labels are nominal valued, you will need to use an appropriate distance metric. Jaccard index is one option.


Compute parameters of a PDF (probability density function) for which no closed form expression is available I would like to compute parameters such as mean, variance, quantiles, etc. for a PDF which is only given as a piece of code. That is, it can only be evaluated numerically at given points; no closed-form expression.

For example, after using scikit-learn for kernel density estimation, I would like to compute parameters of the resulting PDF.

I feel that popular libraries such as numpy/scipy, scikit-learn, or pandas probably provide such functionality. However, I couldn't find it.

Can anybody please point me to it?

Your problem could be solved either by direct numeric integration or by MCMC.

Numeric integration can be performed most easily by scipy:

import numpy as np
import scipy.stats
import scipy.integrate

def weird_density(x):
    """ The function I want to sample """
    return scipy.stats.lognorm.pdf(x, 1.0)

def quad_quantile(fun, q, precision=1e-10, minus_inf=-10e10, lower=-10e10, upper=10e10):
    """ Use bisection to evaluate a quantile """
    while upper - lower > precision:
        med = (upper + lower) * 0.5
        cdf_med = scipy.integrate.quad(fun, minus_inf, med)[0]
        if cdf_med < q:
            lower = med
        else:
            upper = med
    return (upper + lower) * 0.5

print('true mean  :', scipy.stats.lognorm(1).mean())
print('integrated mean:', scipy.integrate.quad(lambda x: weird_density(x) * x, 0, 100)[0])
print('true median  :', scipy.stats.lognorm(1).ppf(0.5))
print('integrated median:', quad_quantile(weird_density, 0.5, 0.001, 0, 0, 100))

The output is

true mean  : 1.6487212707
integrated mean: 1.6484641126903046
true median  : 1.0
integrated median: 0.9998321533203125

However, if the distribution is multidimensional, direct integration may be intractable. In this case,  MCMC methods (Markov chain Monte Carlo) can help.

What they do is just make a correct (in some sense) sample from the distribution for which you know the PDF. When you have a sample, you can calculate all your parameters from it as classical sample statistics, just like from any observed data.

MCMC algorithms may be implemented manually, like in the example below. Another option is to use the PyMC3 library or its analogs.

One of the best known MCMC algorithms, Metropolis-Hastings, works as follows:

def sample_next(x):
    """ Generate random next point"""
    return scipy.stats.norm(loc=x).rvs(1)[0]

def metropolis_hastings(density, generator, n_samples, starting_point=1, random_state=None):
    """ Generate sample from the given density function """
    np.random.seed(random_state)
    result = [starting_point]
    for i in range(n_samples):
        current_point = result[-1]
        next_candidate = generator(current_point)
        acceptance_ratio = density(next_candidate) / density(current_point)
        if np.random.uniform() <= acceptance_ratio:
            result.append(next_candidate)
        else:
            result.append(current_point)
    return np.array(result)

sample = metropolis_hastings(weird_density, sample_next, 1000, random_state=1)

print('sample mean:', np.mean(sample))
print('sample median:', np.median(sample))

This code prints

sample mean: 1.2872857165
sample median: 0.932826883898

You see that the results are not very close, but with longer sampling (1000 points is too few) they will converge. And the shape of distribution is already well matched:

import matplotlib.pyplot as plt
plt.hist(sample, bins=30, normed=True)
idx = np.linspace(0, sample.max())
plt.plot(idx, scipy.stats.lognorm(1.0).pdf(idx))
plt.legend(['true density', 'sample density'])
plt.title('convergence of MCMC sample')
plt.show()


How to use ADWIN with multiple columns I want to perform drift detection on data with multiple input values (x0, x1, x2, x3). I'm using an adaptive window algorithm found from sci-kit found here.

Doing this

from skmultiflow.drift_detection.adwin import ADWIN
adwin = ADWIN()
adwin.add_element(np.array([1, 2, 3, 4]))

Results in ValueError: setting an array element with a sequence.

But doing this

from skmultiflow.drift_detection.adwin import ADWIN
adwin = ADWIN()
adwin.add_element(np.array([1])

works just fine.

Any idea how to do the first thing?

My current solution is to just use 4 different drift detectors, but I would like to use one.

This is not currently implemented. So while you can add any number as a concept value, you're probably best served by adding 1s and 0s depending on whether a not misspecification occurred.

The reason it is not implemented is probably that ADWIN essentially performs a statistical test (or a heuristic approximation) whether the mean of two (large enough) windows is significantly different. And comparison of multivariate mean vectors is very tricky, in particular if one cannot make any assumptions about normality etc.

Otherwise, your workaround of using 4 drift detectors seems like the best solution!


Evaluating Model Accuracy on a testing data set for a DecisionTreeReegressor Model I am trying an exercise where I have been asked to "Evaluate each model accuracy on testing data set for a max_depth parameter value changing from 2 to 5".

The model here is DecisionTreeRegressor. I just wanted to know what is the metric for calculating the Accuracy for a DecisionTreeRegressor model.

My understanding is that it's same as Score which can be calculated simply as regressor.score(X_test, Y_test)

Please let me know what should be used to calculate the Accuracy of the DecisionTreeRegressor Model.

Accuracy in ML vocabulary is used mostly for Classification problem i.e. Count of correct prediction out of total.
In a common speaking language, it will mean the predictive correctness of the model esp. on test data.

  My understanding is that it's same as Score which can be calculated simply asregressor.score(X_test, Y_test)

I am assuming that you are using SciKit-Learn,
score method for DecisionTreeRegressor will return R-square coefficient.Offical Link

  score(self, X, y[, sample_weight]) Return the coefficient of determination R^2 of the prediction.

What should you do -
You should calculate two metrics - R-square and MAE/MSE.
Reason being - for an end-user/business person, MAE would be useful e.g. saying that model's prediction will be ~250$ away from the correct value on an average.

Challenge with MAE/MSE is that it doesn't say if it is good model unless you have an idea of the underlying data. e.g. Creating two models on pricing data of 2 different city - Boston/Tokyo and the MSE is 1000$/$1500. You can't conclude that the former is a better model from this data.
R-square helps here.
Adjusted R-square (Another regression metrics) - If your feature set is fixed, then you need not check this metrics. It was devised to fix an issue with R-square when the feature set is different for different models.

 Snippet to get RMSE, R-square, Adjusted R-square

#https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics

def reg_metrics(y_test, y_pred, X_train):
    from sklearn.metrics import mean_squared_error, r2_score

    rmse = np.sqrt(mean_squared_error(y_test,y_pred))
    r2 = r2_score(y_test,y_pred)

    # Scikit-learn doesn't have adjusted r-square, hence custom code
    n = y_pred.shape[0]
    k = X_train.shape[1]
    adj_r_sq = 1 - (1 - r2)*(n-1)/(n-1-k)

    print(rmse, r2, adj_r_sq)

Links to study -
Statistics by Jim
Wikipedia


ZeroR as performance baseline for binary classfication model? It is known that ZeroR model is used predict the majority class in a given data set.
Having said that, is ZeroR a suitable performance baseline provided one has a balanced data set (50/50)?
If not, what would be a good baseline for a Naive Bayes classification model, used for binary classification (positive/negative)?

Sure, ZeroR is a perfectly fine baseline. In this case I think it's better to call it a random baseline rather than a majority baseline, since it's just what it is.
To my knowledge this is the only basic baseline usable with any classification task. Other baselines would involve something more sophisticated based on the specific task. The standard way to have a more competitive baseline is to used a state of the art method for the task.
Btw the baseline or evaluation method doesn't depend on the learning algorithm, it doesn't matter if it's NB or any other classification method.


Why RNNs necessary for time series? I got it that when using time series data, I have to use a RNN

The highlight here is that the neurons receive their output again as an input, so they can take into account the previous step.

But what I do not understand is: Why cannot regular NNs do the same? For example, when I provide an order like

0, 1, 2, 3, 4, 5, 6, 7, ..

the NN should register that there is a pattern in it. If these numbers represent days or seconds, it is not important.
So why are self-looped neurons now necessary to do so?

To be clear, RNN are not necessary as there exist other options. However, RNN are useful since they implicitly deal with temporal relationships between inputs.

Regular neural networks are idempotent, meaning that the same input can be passed multiple times and the same output will be computed (obviously if the network is not updated in the meantime).

This idempotence means that regular neural nets cannot implicitly "see" temporal patterns. To do so, they would need to be provided with temporal information as you suggest. This is actually what transformers achieve! By providing temporal embeddings on top of the input, it always a regular neural net to learn from temporal information.


Are calibrated probabilities always more reliable? EDIT: Based on the answer below, I have updated the question and added more detail.

I have applied Dirichlet calibration to my fast-bert sentiment classification model, and I am struggling to really understand why/ if it is actually more reliable.

Application:

def dirichlet_scaling(train, y_train, test):
    logreg = LogisticRegression(multi_class='multinomial', solver ='newton-cg')
    logreg.fit(np.log(train), y_train)
    return logreg.predict_proba(np.log(test))

train --> validation set, size ~ 11000

test --> test set, size ~ 7000

These are the uncalibrated probability reliability diagrams:

The classwise diagrams below are based on this recent article https://arxiv.org/abs/1910.12656.

Evaluation metrics:

Confusion-Matrix:

After applying Dirichlet scaling, as per the article, I get these classwise-reliabilities:

And overall confidence calibration:

Dirichlet Evaluation Metrics:

Dirichlet Calibrated Confusion Matrix

After calibration, the minority class - mixed sentiment - is less accurately predicted. The sample below is of mixed samples with high uncalibrated probabilities (> .9).

These cases are all undoubtedly mixed (even the incorrect label). However, after calibration they are less confidently predicted.

My main question is:

The estimator over-estimates the minority class and under-estimates the majority class, which balances the class imbalance - calibrating the probabilities seems to reverse this to a certain extent.

After calibration the class imbalance appears more pronounced in the predictions. I may very well be wrong, but the miscalibration in the classwise reliability diagrams appears to be the classifier incorporating the class distributions (class priors). Is it not usually the case that when imbalanced datasets are artificially balanced, the class priors are reintroduced to balance the probabilities? Does this technique also reduce calibration?

Is it possible the classifier was already well calibrated? And that the over/under-estimation seen in the classwise-reliability diagrams is perhaps not always indicative of miscalibration if overall confidence is well calibrated?

The results seem quite reasonable to me, but I cannot be sure based on your given information. In your table of results, you have shown 10 instances where the uncalibrated probability for the mixed class is between 0.97 and 0.99. It means that on average one would expect only about 2% of misclassifications on these instances. In the small sample that you are showing, the misclassification rate is 10% (1 out of 10), since one of the instances has the negative label. Maybe you have chosen these 10 instances randomly, and actually in most other samples of size 10 there would be 2 or 3 misclassified instances? If that is true, then 70% or 80% probability would indeed be better calibrated.

Looking at the misclassified instance, however, I am not sure whether the negative label is even correct for it? Maybe it should be mixed as well? So maybe there are labelling errors also in the data? Calibration is taking the labels as the ground truth, so if your model's predictions are even better than the labels, then the confidence is still reduced, because calibration is trusting the labels.

The above was my short answer, but below I will try to interpret your figures in more detail.

In your task, you have 3 classes: positive, negative, and mixed. In an attempt to make it easier to follow my answer, let me first give names to your figures:

Figure 1a - confidence histogram of uncalibrated predictions;
Figure 1b - confidence-reliability diagram of uncalibrated predictions;
Figures 2a, 2b, 2c - classwise-reliability diagrams of uncalibrated predictions for classes Neg, Pos, Mixed;
Figures 3a, 3b, 3c - classwise-reliability diagrams of Dirichlet-calibrated predictions for classes Neg, Pos, Mixed;
Figure 4a - confidence histogram of Dirichlet-calibrated predictions;
Figure 4b - confidence-reliability diagram of Dirichlet-calibrated predictions.

Fig. 1a shows that the vast majority of predictions have confidence above 0.9. Fig. 1b shows that in the highest-confidence bin (0.9 and above) there is a tiny bit of under-confidence, as the blue bar (actual accuracy within this bin) seems to be higher than the predicted (above the diagonal) - this is indicated by the red gap color mixing with the blue actual color to obtain dark red. Since the gap is small, the predictions seem quite well confidence-calibrated.

However, looking at the classwise-reliability diagrams it can be seen that the negative class is quite well-calibrated (Fig. 2a), whereas the positive class predictions are under-confident (Fig. 2b), and the mixed class predictions are under-confident (Fig. 2c). Figures 3a, 3b, 3c show that after applying the Dirichlet calibration, the probabilities are better classwise-calibrated.

The confidence-reliability diagram in Fig. 4b shows better calibration in the lower confidence bins, compared to Fig. 1b. However, most of the predictions are still in the highest confidence bin with confidence above 0.9 (Fig. 1a). In that bin, there is perhaps a tiny bit more under-confidence than before calibration. A possible explanation for this is that the reduction of loss in lower bins is more than the tiny increase of loss in the highest bin.

Based on the diagrams and table of results alone I cannot be sure whether Dirichlet calibration has been applied correctly, but it is possible that everything is correct. There seems to be improvement in classwise calibration and probably in confidence calibration also. I can try to interpret the results further, if you could give more information:
 - the classwise histograms of class probability to go with figures 2 and 3;
 - the learned parameter values for Dirichlet calibration;
 - size of the dataset on which you calibrate and on which you show the reliability diagrams, and perhaps the class distribution as well.

Here are some more thoughts based on the confusion matrices and numeric results for evaluation measures that were added to the question after my above answer.

I don't think that the original model was already calibrated, given your reported numeric evaluation information and the classwise calibration plots. Dirichlet calibration has certainly improved calibration.

It seems that all the measures are improving applying after Dirichlet calibration, with the single exception being the accuracy on the mixed class. I can think of two reasons for this, which probably together result in this effect.

First, the classes are imbalanced, and thus 1% of increase in accuracy for the majority class is for the classifier and for the calibrator worth more than 1% decrease in accuracy for the minority class. You haven't specified your exact class distribution, but for example, if there is more than 3.5 times difference in size of positive and negative classes vs the mixed class, then increase of 1% both in positive and negative class accuracy is worth more than 7% (3.5%+3.5%) decrease in the mixed class accuracy. Well, Dirichlet calibration is optimising log-loss rather than accuracy, but the point remains roughly the same.

Second, there are misclassified instances in the validation set. This forces the calibrator to lower the confidence, as discussed in my earlier answer above.

Maybe there are further factors relating to different distributions of confidences for different classes. This could be checked if you would provide the histograms of predicted class probabilities for each class separately (maybe both before and after calibration).

If you are particularly interested in having a better accuracy in the mixed class, then you need to upweight the mixed class during calibration. Ideally, calibration should not push towards the overall class distribution, but to some extent this can happen due to the limitations of parametric modelling. I suggest you to try oversampling the mixed class to be of the same size as the positive and negative classes during calibration.


What possible data products can be built using this dataset I have a dataset which contains information about when do people enter and leave a premise. I have the following information in the dataset :

Person Id
Time of Entry
Time of Leaving

The dataset has around 50 unique persons. Each person will have multiple entries corresponding to multiple visits. The data spans over a year so I have quite a lot of entries (around 1 million).

These people can be classified on the basis of the department they work under (2 departments - mutually exclusive) or on basis of  their role (4 possible roles - all mutually exclusive)

I was wondering what kind of data analysis can be done with this kind of dataset. I am not looking for straight-forward things like "who spent the most time in building". However things like finding correlation between visits of 2 people would be interesting. So if person A visits the premise, what is the probability that person B would also visit. Since I have only around 50 unique visitors, I think such an analysis is feasible.

Another line of thought was to apply some interval-pattern mining techniques but I am not much familiar with them.

Can someone give me some pointers/ideas about what kind of data products can be build using this or what kind of techniques can be used with such data.

Edit : As discussed in comments, I call it a product in the sense that I do not want some simple or trivial analysis. And I am not looking for any commercially viable idea - just some cool fun idea :)

You might be able to find individuals that act as anchors.  That if Abe is here, then everybody else is more likely to stay longer. Interesting to know if it was always the same people who stayed or not.

There might also be some interesting patterns with groups that arrive, are present, or leave together.

As a business owner you could look for individuals who not just work shorter hours, but arrive at their own time, and leave with a group.


What is the best way to store images in python for machine learning I am currently working on a classification problem that requires me to classify whether an image contains cancerous tissue cells or not. Each image is 50x50x3 pixels, the 3 is for RGB values.

So far I have a pandas dataframe that contains the target value, patient id, image id and the path to the corresponding image.

I can access the image by using

io.imread(df['path'])

So it is possible for me to loop through all the images to access them. The question now is, where do I store the images so that I can apply principle component analysis on them?

If I were to simply store it in a dataframe it would contain 7500 columns; 1 for each pixel value. My dataset contains 280,000 images. That means my my dataframe would need to be 280,000x7500. I Feel that there is a better way to approach this problem.

Your input to this matter would be highly appreciated.

This might be a bit more complicated.
I normally reuse computer vision and deep learning software to do that. Even if I don't do Deep Learning.
Particularly I use Pytorch, for its bridge with Numpy and pandas.  Here is a tutorial.
This allows me to use a GPU if wanted, and to reuse a lot of code since for deep learning and images there is tons of code snippets out there.


What does the collate function in pytorch (geometric)? I am creating a message passing neural network and have some issues with the dataset creation. In pytorch (geometric) it is recommended to create a dataset with the following class. I wonder what is the meaning of the collate function that is called at the end of the process method? In what cases should I use my own collate function? My graphs have mostly different sizes.

import torch
from torch_geometric.data import InMemoryDataset

class MyOwnDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(MyOwnDataset, self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return ['some_file_1', 'some_file_2', ...]

    @property
    def processed_file_names(self):
        return ['data.pt']

    def download(self):
        # Download to `self.raw_dir`.

    def process(self):
        # Read data into huge `Data` list.
        data_list = [...]

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])

The error I get is:

  RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 4422 and 4032 in dimension 1 at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/THC/generic/THCTensorMath.cu:71

What collate does and why:

  Because saving a huge python list is really slow, we collate the list into one huge torch_geometric.data.Data object via torch_geometric.data.InMemoryDataset.collate() before saving . The collated data object has concatenated all examples into one big data object and, in addition, returns a slices dictionary to reconstruct single examples from this object. Finally, we need to load these two objects in the constructor into the properties self.data and self.slices.

Source


Extremely negative r^2 I use a linear regression to predict house prices (https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview). My linear regression sometimes works great with R^2 of 0.8 and sometimes really sucks with R^2 of - 20000000000000 (Yes, really that bad). My data is scaled (with Min-Max) but my target value isn't. The problem appeared before scaling as well. Apparently my predictions are only a few times off, but then they are really off.
Here is my code:
df = pd.read_csv('data/results/house_prices_advanced_regression/house_prices_advanced_regression_.csv')
target = df['SalePrice']
df = df.drop('SalePrice', axis=1)
x_train, x_test, y_train, y_test = train_test_split(df, target, test_size=0.2)

# model
lr = LinearRegression().fit(x_train, y_train)

predictions = lr.predict(x_test)
rmse = mean_squared_error(y_test, predictions, squared=False)
mae = mean_absolute_error(y_test, predictions)
print('RMSE: {}'.format(rmse))
print('MAE: {}'.format(mae))
x = range(len(y_test))
plt.plot(x, predictions, 'r')
plt.plot(x, y_test, 'b')
print("R^2: {}".format(lr.score(x_test, y_test)))

Output:
'RMSE: 4.2834860573491624e+16'
'MAE: 3483821398471256.5'
'R^2: -3.358286170039772e+23'

Difference between prediction (red) and real value(blue):

Coefficients:

Does anybody know what to do?

There are a number of approaches to try and improve the model but more importantly to understand what is happening.
First of all, you've displayed a plot with over 200 'features' and despite scaling the input the coefficients are in the order of 1E17!!! Edit: house prices (in dollars are in the range 1E5-E7, ie less than $10M), yet the coefficients are at least 7 orders of magnitude greater or smaller.
Therefore, two immediate steps to take either together or in combination is conduct:

dimensionality reduction either using a formal method e.g. PCA or just dropping terms that have coefficients close to zero backed with some plots to help you decide.

regularisation - at at the moment you have some extreme and opposite coefficients around +7.5E17 and -7.5E17.  See models here https://scikit-learn.org/stable/modules/linear_model.html#linear-model that implement ways to suppress extreme coefficients. Edit: If your input data is between 0-1 ask yourself why for some parameters are at opposite ends, in effect cancelling each other out.

Use some personal domain knowledge and evaluate the individual columns. Is there anything that has no bearing on the sale price? Some silly examples might the name of first person who bought it, which has no relevance to house price. Or a telephone number etc.

As we're now entering the world of 'hyper-parameter' tuning don't forget to split your data into 3 parts instead of 2.


How do I know the best pruning criteria for decision trees? Right now,I am working on decision trees on python,how do I know what would be the best pruning criteria based on my data?

Experimentally: using cross-validation on a subset of your training data, compute the performance of every option that you want to consider. Then select the best option and train the final model using this option.

// different settings for hyper-parameters,
// for instance different pruning criteria:
hpSet = { hp1, hp2, ...}

trainSet, testSet = split(data)

for each hp in hpSet:
    // run cross-validation over 'train' using hyper-parameter 'hp'
    // and store resulting performance
    perf[hp] = runCV(k, trainSet, hp)

bestHP = pick maximum hp in 'perf'
model = train(trainSet, bestHP)
perf = test(model, testSet)


Why will the accuracy of a highly unbalanced dataset reduce after oversampling? I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while after balancing (i.e., 100 samples in both the classes)  it is 60%. What are the possible reasons for this?

Imagine that your data is not easily separable. Your classifier isn't able to do a very good job at distinguishing between positive and negative examples, so it usually predicts the majority class for any example. In the unbalanced case, it will get 100 examples correct and 20 wrong, resulting in a 100/120 = 83% accuracy. But after balancing the classes, the best possible result is about 50%.

The problem here is that accuracy is not a good measure of performance on unbalanced classes. It may be that your data is too difficult, or the capacity of your classifier is not strong enough. It's usually better to look at the confusion matrix to better understand how the classifier is working, or look at metrics other than accuracy such as the precision and recall, $F_1$ score (which is just the harmonic mean of precision and recall), or AUC. These are typically all easy to use in common machine learning libraries like scikit-learn.


Does anyone use Julia programming language? Is anyone using Julia (http://julialang.org/) for professional jobs?

Or using it instead of R, Matlab, or Mathematica?

Is it a good language?

If you have to predict next 5-10 years: Do you think it grow up enough to became such a standard in data science like R or similar?

I personally have used Julia for a good number of professional projects, and while, as Dirk mentioned, this is purely conjecture, I can give some insights on where Julia really stands out. The question of whether or not these reasons will prove enough to have Julia succeed as a language is anyone's guess.

Distributed Systems: Julia is the easiest language I've ever dealt with in terms of building distributed systems. This is becoming more and more relevant in computing, and will potentially become a deciding factor, but the question of whether or not Julia'a relative ease decides this is up for debate
JIT Performance: Julia's JIT compiler is extremely fast, and while there is a lot of debate as to how accurate these benchmark numbers are, the Julia Website shows a series of relevant benchmarks
Community: This is an area where Julia just isn't quite there. The community that is there is generally supportive, but not quite as knowledgable as the R or python communities, which is a definite minus.
Extensibility: This is another place where Julia is currently lacking, there is a large disconnect between the implies code patterns that Julia steers you toward and what it can actually support. The type system is currently overly bulky and difficult to use effectively.

Again, can't say what this means for the future, but these are just a couple of relevant points when it comes to evaluating Julia in my opinion.


Why softmax in YouTube’s DNN recommender I am confused about the softmax layer of YouTube’s DNN candidate generation. A user may interact with many videos. Softmax is assuming classes are exclusive. For example, logits = [[4.0, 4.0, 1.0]], labels = [[1.0, 1.0, 0.0]], the sigmoid cross entropy loss is 0.45 while softmax cross entropy loss is 1.43.
Is it because in the candidate generation stage, the relative order of items does not matter?

"Is it because in the candidate generation stage, the relative order of items does not matter?"

Yes this is exactly what appears to be going on, although it seems that youtube is using softmax in a untraditional way. The Candidate Generation model simply selects the few hundred candidate videos that are subsequently ranked by the ranking model.
Section 3 of the paper you referenced I think does a good job explaining whats going on:

"At serving time we need to compute the most likely N
classes (videos) in order to choose the top N to present
to the user... Since calibrated
likelihoods from the softmax output layer are not needed
at serving time, the scoring problem reduces to a nearest
neighbor search in the dot product space for which general
purpose libraries can be used."

As far as I can tell, this kind of recommender architecture is only beneficial at the scale of which an organization like youtube is operating, and has more to do with the practicality of organizing computational infrastructure rather than model performance. I'm sure their model performance vs. a more "traditional" architecture is negligible as far as things like map@k are concerned.
EDIT: Found the same question already answered with much more detail than shown here.


What pre-processing of the image is needed before feeding it into the convolutional neural network? I can't figure out what preprocessing of the image is needed before feeding it into the convolutional neural network. For example, I want to recognize circles on a 1000 by 1000 px photo. The learning process of a neural network occurs on 100 by 100 px (https://www.kaggle.com/smeschke/four-shapes/data). I'm having a little difficulty wrapping my head around the situation when the circle in the input image is much larger (or smaller) than 100x100 px. How then the convolution neural network determines that circle if it was learned on a dataset of a different picture's size.

For clarity, I want to submit a 454 by 430 px image to the network input:

Example of the dataset for the learning process (100 by 100 px):

Finally, I want to recognize all the circles on the input image:

If you have a trained network that accepts input of 100 * 100 you can either scale your image accordingly or slice it into smaller parts.

Scaling the image

Scaling your image down is really straight forward. It will lower the information and might distort it too if your aspect ratio doesn't match. In your case it is a little off, but most networks are trained with a little distortion.

You will have to ask yourself if the details you are interested in are still discernable, and whether they (still) match the examples the network is trained on.

Slicing smaller parts

If you are interested in features rather than the larger scene, you could slice your image into bits of 100 * 100. I recommend you use a sliding window, wherein you tile the subsamples as if they were shingles. That way you avoid that a feature is only availble in bit's and doenst get recognized.


Reason for generally using RMSE instead of MSE in Linear Regression In linear regression, why we generally use RMSE instead of MSE? The rationale I know is that it's easy to minimize the error in RMSE instead of MSE by Gradient Descent, but I need to know the exact reason.

However RMSE seems similar to MSE and is the root of it, gradient of RMSE with respect to $i^{th}$ prediction differs from that of MSE.
$$\frac{\sigma{RMSE}}{\sigma{y_i}} =\frac{1}{2}\frac{1}{\sqrt{MSE}}\frac{\sigma MSE}{\sigma y_i}$$
Gradient of RMSE is equal to the gradient of MSE multiplied by this $\frac{1}{2}\frac{1}{\sqrt{MSE}}$ value which is constant and is called learning rate. And it shows that RMSE and MSE cannot be interchangeably used when using gradient based methods like when it comes to use linear regression with gradient descent optimization algorithm.
Further, in cases when it is better to give more weight (higher error) to large errors, RMSE can be a better error measure. I believe it depends on your data distribution which one of them to pick to use. But in linear regression, maybe not caring about the points which are considerably off the actual prediction line leads to being biased to considering these points and having a bad model. So, it might be better to give a higher error value to these points.
This and this might also help.


Including validation set in my code for a Linear SVM classifier returns a Type error I'm using a predict function for a linear SVM classifier:
def predict(self, train_file: str, test_file: str, lower_case: bool) -> pd.DataFrame:
"Train model using sklearn pipeline"
train_df = self.read_data(train_file, lower_case)
dev_df = self.read_data(dev_file, lower_case)
learner = self.pipeline.fit(train_df['text'], train_df['truth'])
# Fit the learner to the test data
test_df = self.read_data(test_file, lower_case)
# assuming dev_df is validset
dev_df['pred'] = learner.predict(dev_df['text'])
test_df['pred'] = learner.predict(test_df['text'])
return test_df, dev_df

And the predict function refers also to:
    def run_classifier(files: Tuple[str, str, str],
               method: str,
               method_class: Base,
               model_file: str,
               lower_case: bool) -> None:
"Inherit classes from classifiers.py and apply the predict/accuracy methods"
train, dev, test = files  # Unpack train, dev and test filenames
result = method_class.predict(train, test, lower_case)
method_class.accuracy(result)
# Plot confusion matrix
make_dirs("Plots")
print(result)
fig, ax = plot_confusion_matrix(result['truth'], result['pred'], normalize=True)
ax.set_title("Normalized Confusion Matrix: {}".format(method.title()))
fig.tight_layout()
fig.savefig("Plots/{}.png".format(method))

Where the dev set is not really taken into consideration. When I run predict, I receive this error: predict() missing 1 required positional argument: 'lower_case but I don't understand how to solve it.

Based on the repo you linked in the comment section (github.com/prrao87/fine-grained-sentiment), method_class is meant to be a Class not an instance. Therefore method_class.predict is meant to be a static method not an instance method, Static method are declared without specifying a self (as they are not linked to an instance but a class) declaring your function as def predict(train_file: str, test_file: str, lower_case: bool) should fix your problem.
Some reference on the topic : https://www.makeuseof.com/tag/python-instance-static-class-methods/


One-liner:

result = data.loc[data.x == 4, "time"]    # assuming you expect just the time as output

Explanation

To describe your problem, first in English (then code):

  You want the rows of the data frame, where the value of the X column is 4, and all columns should be included in the result.

To check where coilumn X is 4, we can do this:

data.X == 4

This returns True or False for every single row. Now we can use this as a filter, by passing it to the pandas DataFrame's .loc accessor (loc = "locate").

result = data.loc[data.X == 4, :]
# Notation: dataframe[rows_we_want, columns_we_want]

Because we want all columns, we have left the colon : in there after the comma. Pandas assumes you want all columns by default, so we can actually leave it out to get the one-liner given above.

So if you just want to time to be given as output, you can filter for that by putting the name of the column after the comma:

result = data.loc[data.X == 4, "time"]


How to feed data for ngram model? I want to train an ngram language model

Let's say I have the following corpus:

The sliding cat is not able to dance
He is only able to slide
Because obviously he is the sliding cat

I am planning to use tf.data.Dataset to feed my model, which is fine

But I don't know if it is better to use a sliding window to iterate through my copus or simply feed my corpus n words at a time

Using a sliding window, my model (assuming a bigram) will see:

The sliding
sliding cat
cat is
is not
...

Going n word at a time:

The sliding
cat is
not able
...

I'd appreciate any recommandation, thanks

You should definitely use a sliding window.

An n-gram language model represents the probabilities for all the n-grams. If it doesn't see a particular n-gram in the training data, for example "sliding cat", it will assume that this n-gram has probability zero (actually zero probabilities are usually replaced with very low probability by smoothing, in order to account for out-of-vocabulary n-grams). This would result in a zero probability for a sentence which was actually in the training case (or a very low probability with smoothing).

Also it's common to use "padding" at the beginning and end of every sentence, like this:

#SENT_START# The
The sliding
sliding cat
cat is
is not
...
to dance
dance #SENT_END#

This gives the model indications about the words more likely to be at the beginning or end (it also balances the number of n-grams by word in a sentence: exactly $n$ even for the first/last word).


Is it possible to apply PCA on different subsets independently? I need to apply PCA on a rather big set of data, but my machine is not able to handle the workload. So I was considering to split randomly my original set into 4 subsets, apply PCA independently on each subset and finally join the 4 subsets to have the original one with the PCA.

For my understanding PCA looks for correlated variables so they can be combined into one, which somehow will represent the values of the original variables. So I believe this operation happens on a row level. However, I guess the algorithm needs to analyse all the set as a whole to determine the correlation between features, since correlation among features row by row may differ, and some rows may even have NaN values.

So I would like to know if this approach with the subsets is correct, or if I may end up with one subset which after PAC combined features a and b and another subset which combined b and c.

You can use mini batch PCA. One formulation is available in sklearn.

Alternatively, you can run PCA on a carefully selected subset of your data. It's very time consuming and I'm not sure it's possible, and its feasibility is task specific.


Automation of finding a starting point of measurement in a large dataset I am looking for a way to automatically find a starting point of rising in my signal in Python. The data are collected with the frequency 10k (0.0001 s each) so the differences between each point are very small, lost in the noise. I found this point (black dot) manually using data analysis software before but I have multiple files and the manual process is not gonna work well. I was trying to think of something to do with derivative (red dots) or rolling variance (green dots) but it's a dead end for me now. Here's how manual point was chosen:

I pick a point that looks to me that is the closest one to rising signal but is still in the middle of noise before rising. Chosing it manually is just my rough estimation but I don't mind being one or two points wrong from the "correct" starting rising point. I will use it to offset my signal so that rising starts more or less at X = 0.
And now I wanted to find it using python. The full signal looks like this:

The derivative:

The rolling variance:

So they're all close to the interest point (black dot) but I don't know what to do with them next. If I change the limits it all looks like this:

Any ideas how to solve my problem? The simple code sample is below (plotting excluded)
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from scipy.optimize import curve_fit
import scipy.signal as sig

#reading dataset
signal = pd.read_csv('dataset.txt', delimiter=' ' )
signal.columns = ['time','current']

#calculating derivative, finding max and min indices of derivative
signal_derivative = np.gradient(signal,axis=0)
signal['derivative'] = pd.DataFrame(signal_derivative[:,1])
index_derivative_max = signal['derivative'].index[signal['derivative'] == signal['derivative'].max()]
index_derivative_min = signal['derivative'].index[signal['derivative'] == signal['derivative'].min()]

#calculating rolling variance, range 50 points, finding indices of peaks
signal['rolling_var'] = signal['current'].rolling(window=50,center=False).std()
index_rolling_max = signal['rolling_var'].index[signal['rolling_var'] == signal['rolling_var'].max()]
index_rolling_2nd_max = signal['rolling_var'].index[signal['rolling_var'] == signal['rolling_var'][:100000].max()]

Well ... I would do exactly what you did. The derivative on original signal is very noisy. I would probably take derivative out of moving-averaged smoothed signal, however it brings some delay into your detection. See this answer for more info and python code.
The other approach is to detect the point in time-frequency domain. Simply plot the STFT of your signal and see if that helps the accuracy of detection over derivative.


Estimating sales volume of new drugs to a therapeutic class I work with pharmacy claims data for an insurance company and wish to estimate how many members on existing products in a therapeutic class will shift to the new product in the same class i.e. insulin once the drug has been covered by the insurance company.

New entrants/products are very common but each one has their own nuances, some are blockbusters while others are flops, some drugs have many indications/uses while others are very specific.

I have access to lots of demographic, historical utilisation, diagnosis code data and formulary information that I could incorporate into a model. I would be keen to hear people’s thoughts and suggestions about how one would tackle such a problem.

The first approach is a supervised machine learning model. If you have historical data, you can label members who transfer to the new drug when a new drug came out as label one, otherwise label zero. Then you have a supervised machine learning setting. Just use your other information as features. In the end, you count how many members are predicted to take the new drug.

The second approach is an unsupervised machine learning model. Using clustering to classifies members into two groups. You then count the group whose members will take new drugs. The problem of this method is that you need to have couple samples's label determined so that you can know which group is the group you need.

The above two methods all require that you have historical data.

I am thinking if there are other methods that can give an expected value and don't require training on the historical data. I will update once I figure out one.


Can I download Twitter data via web scraping for research? I want to do a sentiment analysis using twitter data. Was thinking about hardcoding a cURL script to download data, from a Google Cloud service (I'll run the data on a neural network on the server, to label each tweet), but I have this question:

Am I allowed to do this? I know twitter sells the data, so I am not sure if I can get in trouble for downloading it directly (I have to disclose the data gathering methodology on the paper).

Last time I checked it was not allowed to store the contents of the tweets, instead one is supposed to store the tweet id and retrieve the content of the tweet dynamically.

Afaik this is because users are allowed to delete their tweets at any time, and keeping a tweet that they chose to delete would be against Twitter terms of use (and possibly illegal in some jurisdictions). Using the tweet id solves the problem since the content will simply not be available anymore if the tweet was deleted.

Since you plan to write a paper I assume that you're in academia? If yes in case of doubt it's always safer to ask the data protection and/or legal office in your institution. In this case you're using a secondary source (i.e. you're collecting data which already exists) so it should be straightforward (I think).


Correct learning algorithm for grouped examples I need to learn which item will be (most probably) picked from a group. The data I have consists of groups by 10 items and which one was picked(or if neither was picked).

My current approach is to ignore the groups and to use flat list with all items classified as picked or not picked and use some classification algorithm, but I wanted to know if there is a way to use the knowledge that the item was picked from that specific group.

Let $M$ be the size of each group. You want to learn the function $f$ that takes $M$ vectors $v_{i}$, $i \in [1,M]$  and yields a vector $\hat{v}$ that can be 0 or one of the $v_{i}$.
You can train a DNN to learn $f$ by organizing your training data as follows:

Input              Label
[batchSize, M, 1]  [batchSize,1]

Optimize on the reduced RMS error between the predicted vectors and the label vectors. The group information will then be learned by your DNN.


clustering with heterogeneous (quantitative and qualitative)data? I'm a Phd student and I have the results of some approaches (algorithms) that I would like to analyze.  Data (results) are stored in csv files as follows: - the lines describe each algorithm with its parameters and the result obtained. - Some columns (characteristics)  are quantitative and  others are qualitative (enumerated types) and the last column is the results which presents the efficiency of the algorithm and it is a numerical value that I can convert in classes (of intervals of values).  I would like to highlight  at first what is the tendency of the good and worse algorithmes. Then, what are the features that contributed to obtain these results I'm beginner in machine learning and I searched a lot on the net but i didn't find the appropriate method . Could you point me to a clustering/ learning method for this purpose.
I would be very greatful.
Best regards.

K-means algorithm cannot be directly used for data with both numerical and categorical values because of its objective function. K-means uses Euclidean distance, which cannot be defined for categorical data.

However, there is a simple method which combines Euclidean distance and Hamming distance for the finding the similarity between instances that include both numerical and categorical features. In this method Euclidean distance is used as a metric for computing the similarity between numerical data and hamming distance for computing the similarity between categorical data. Both of them with the appropirate weights form an objective function that can handle mixed data.

For more information check the relevant paper

https://grid.cs.gsu.edu/~wkim/index_files/papers/kprototype.pdf


Vanishing Gradient vs Exploding Gradient as Activation function? ReLU is used as an activation function that serves two purposes:

Breaking linearity in DNN.
Helping in handling Vanishing Gradient problem.

For Exploding Gradient problem, we use Gradient Clipping approach where we set the max threshold limit of Gradient, similarly to ReLU which set the minimum Gradient limit which is 0.
So far I read, ReLU is considered as an activation function. In a similar fashion, can we use Gradient Clipping also as an activation function? If yes, any pros and cons of using it?

ReLU is considered as an activation function, on similar fashion can we use Gradient Clipping also as an activation function?

ReLU is an activation function. Gradient clipping is a technique to keep the problem of exploding gradient at bay.

I wish also to stress that the best technique to control for vanishing/exploding gradients is, at the moment, batch normalization. Dropout (a technique born to fight overfitting) also has a similar regularization effect - by forcing the model to distribute weights more evenly through the layer. That's why you don't see gradient clipping that often as it used to.

EDIT:

I forgot to mention that a proper scaling of your variables and appropriate weight initializations make the problem of vanishing/exploding gradient not very frequent. This of course is purely based on personale experience. It's still very important to take it into account


Errors changing column dtype to numeric I am working with data from the government regarding COVID immunization numbers and would like to convert object columns into numeric. I am using Python in Jupyter Notebook. I tried using to_numeric, but the dtype is still an object. Any suggestions?
Thanks in advance :)

Set the errors parameter of the to_numeric function to 'coerce'. The value 'ignore' will leave any invalid value as is while 'coerce' will set it to NaN.


Do I need to encode samples during inference? I recently started saving (pickling) my fitted encoders. The thinking was that I would need them to encode previously unseen samples during inference.

Encode training features and labels.
Train model on encoded data.
Export fitted encoders alongside model.
New/ unseen data arrives.
Encode features of new data.
Run model prediction.
Decode prediction output.

Is this mandatory?

Yes the encoders need to be rerun during inference time. You basically need to run the exact same data preprocessing steps prior to training on the unseen observations. Without them your model will not be to recognize unseen data and hence fair poorly.


How to improve loss and avoid overfitting I'm trying to build a 2 class image classifier using the architecture suggested in first part of this blog https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

My dataset has 1500 images of class1 and 500 images of class2. I created 4 copies of class2 to make the number of images same in both classes.

I also use ImageDataGenerator to augment images

datagen = ImageDataGenerator(
        featurewise_center=True,
        featurewise_std_normalization=True,
        shear_range=0.2,
        zoom_range=0.5,
        rotation_range=45,
        horizontal_flip=True,
        vertical_flip=True)
datagen.fit(train_imgs)

I'm using AdaDelta,RMSProp, SGD, Adam, AdaGrad optimizers and tried adding/removing Conv2D and Dense layers. Tried BatchNormalizationa and Dropout. The results are coming out almost same:

For first few epochs(about 20) training and validation errors keep reducing until log loss reaches about 0.4 (best I have got till now) after that the model starts to overfit and validation loss keeps increasing.

I know I can prevent overfitting by reducing the network complexity and adding dropouts but that reduces the training accuracy too.

Please suggest some tips to improve the accuracy and avoid overfitting.

I am not familiar with the software you are using but keep in mind: You EXPECT accuracy to drop if you reduce over fitting. It is not a bad thing. Over-fitting is essentially "fake accuracy".

Some good approaches in general to avoid over-fitting though: Use cross-validation, normalize your features, increase size of data-set and dont just increase your data-set by copying data.


Reason for adding 1 to word index for sequence modeling I notice in many of the tutorials 1 is added to the word_index. For example considering a sample code snippet inspired from Tensorflow's tutorial for NMT https://www.tensorflow.org/tutorials/text/nmt_with_attention :
import tensorflow as tf
sample_input = ["sample sentence 1", "sample sentence 2"]
lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
lang_tokenizer.fit_on_texts(sample_input)
vocab_inp_size = len(lang_tokenizer.word_index)+1

I dont understand the reason for adding 1 to the word_index dictionary. Wont adding a random 1 affect the prediction. Any suggestions will be helpful

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.
Then, why adding 1 ?
Because Tokenizer.word_index is a python dictionary that contains token keys (string) and token ID values (integer), and where the first token ID is 1 (not zero) and where the token IDs are assigned incrementally. Therefore, the greatest token ID in word_index is len(word_index). Therefore, we need vocabulary of size len(word_index) + 1 to be able to index up to the greatest token ID.
Update: note that adding 1 to the vocabulary size has nothing to do with out of vocabulary words: the words that are not pretrained are encoded as the out-of-vocabulary token (oov_token) if it was provided when building the tokenizer, or ignored if not. The oov token, if provided, has index 1.


Using training data generated with pure regular expressions - Can machine learning surpass the accuracy of your regular expression? For text classification with machine learning - If your training data was generated purely with regular expressions, is it possible to train a machine learning model with this training data which will be more accurate than your original regex?

By 'accuracy' I mean to say the percentage that are correctly labelled as compared to the human truth of the question.

(E.g. we have a set of Tweets that we've labelled as happy or sad with regex. I'm wondering if a machine learning classifier trained on this data will be able to surpass the accuracy of the regex that generated the training data)

No- not if your regex is actually generating the data. If this is the case, the regex (or a complementary one) should be able to classify the resulting text perfectly. That is, you know what the rules for creating a 'happy' or 'sad' piece of text are within the regex, and so you simply apply those rules backwards to classify it.

So, if you take what the regex says when it creates the data as ground truth, then there's no reason to create a machine learning classifier to classify it- the creation mechanism itself already does so.

On the other hand, if you have created a regex that will produce "happy" or "sad" tweets, and you want to know if the regex itself or a machine learning algorithm will better conform to the opinions of, say, human observers, you would have a very interesting question to evaluate and a relatively straightforward test case for sentiment analysis.


Extracting data, sorting and removing duplicates from an array using python 2.7 I have a text file data set in the form of
[['09', '07', '02', '11'], ['04', '09', '11', '04'], ['06', '07', '02', '02'], ['06', '06', '09', '09'], ['04', '11', '04', '09'], ['09', '04', '11', '04'], ['06', '09', '09', '04'] .....]

I want to extract the complete data from it, sort it and remove duplicates of the same (i.e., the same row should not appear again).
The output should look like (in rows):
11090702
11090404
07060202
09090606
09090604

I want this data for machine learning algorithm implementation.

I think with the volume of data you have, you can try something like this (using the file provided above)

import json
with open('data1.txt', 'r') as f:
    f_json = f.read().replace('\'', '"')
    d = json.loads(f_json)
sorted_d = list(set([''.join(sorted(x, reverse=True)) for x in d]))


How do I force specified coefficients in a Linear Regression model to be positive? Looking for a way to do this in Python. scipy.optimize.nnls forces all coefficients to be positive.

Some additional context: I have a data frame with a some explanatory variables and a response variable. When I run a regular linear regression, the coefficients of some explanatory variables become negative. This is okay for some variables, but not for all. I want to prevent the coefficients of specific variables from becoming negative.

I want to force these coefficients to be positive because they should have a positive contribution to the response. The variables I want to have positive coefficients are investment dollars in a few different channels. The response is revenue. I don't want my model to say that investing more dollars in a certain channel lowers revenue (even if that creates a more accurate model).

Sorry, but on the surface, this sounds like a terrible idea to me: if linear regression gives you negative coefficients for some explanatory variables that you think should be positive, then it means that either your data is "wrong" (typically noisy or too small) or your intuition is misguided.

I can't see any good reason why one would use a data-driven approach if the goal is to manually force the model in a particular way. This is the equivalent of breaking the thermometer to hide the fever.

I'd suggest the following instead:

In general an unexpected outcome is arguably a good thing, in the sense that it tells us something we didn't know about the data. That's a cue to investigate what happens in the data. Linear regression is simple enough to analyze: one can look at the correlation, plot the relation between the variables etc.
If there's really something suspicious going on with some variables, maybe some errors in the data which make them behave in a way they shouldn't, then it's much better to discard them altogether from the model rather than fixing their coefficient, because this way the model won't rely on them at all.


How can I detect partially obscured objects using Python? I'm building a computer vision application using Python (OpenCV, keras-retinanet, tensorflow) which requires detecting an object and then counting how many objects are behind that front object. So, objects are often overlapping.

For example: How many are in this queue?

Here, the aim is to detect the person in the front (foreground) of the queue, and then detect the amount of people behind the front person, despite those behind being obscured.

I have built an object recognition model using keras-retinanet to find the object in the foreground, but now I need to count how many are behind that object. My training data is thousands of images of objects ins straight lines, similar to the image linked above.

However, I have been unable to find a viable way to count the objects behind the front object when they are partially obscured (the rest of the people in the queue).

So far I have tried counting edges (using auto-thresholding), lowering the confidence threshold of my object detection model, and training a new object detection model using only obscured objects as training. These methods have all had very low accuracy.

Do you have any advice or direction on how I can attack this problem?

You can try CSRNet model.  The "official" repo is here.
There are a number of other implementations floating around GitHub.


Item based and user based recommendation difference in Mahout I would like to know how exactly mahout user based and item based recommendation differ from each other.

It defines that

User-based: Recommend items by finding similar users. This is often harder to scale because of the dynamic nature of users.

Item-based: Calculate similarity between items and make recommendations. Items usually don't change much, so this often can be computed off line.

But though there are two kind of recommendation available, what I understand is that both these will take some data model ( say 1,2 or 1,2,.5 as item1,item2,value or user1,user2,value where value is not mandatory) and will perform all calculation as the similarity measure and recommender build-in function we chose and we can run both user/item based recommendation on the same data ( is this a correct assumption ?? ).

So I would like to know how exactly and in which all aspects these two type of algorithm differ.

Item Based Algorithm

for every item i that u has no preference for yet

  for every item j that u has a preference for

    compute a similarity s between i and j

    add u's preference for j, weighted by s, to a running average

 return the top items, ranked by weighted average

User Based Algorithm

for every item i that u has no preference for yet

 for every other user v that has a preference for i

   compute a similarity s between u and v

   add v's preference for i, weighted by s, to a running average

 return the top items, ranked by weighted average

Item vs User based:

1) Recommenders scale with the number of items or users they must deal with, so there are scenarios in which each type can perform better than the other

2) Similarity estimates between items are more likely to converge over time than similarities between users

3) We can compute and cache similarities that converge, which can give item based recommenders a performance advantage

4) Item based recommenders begin with a list of a user's preferred items and therefore do not need a nearest item neighborhood as user based recommenders do


Anomaly detection on multidimensional time series I have relatively little knowledge of unsupervised machine learning.

I'm working on a project that aims to find anomalies in a set of n data, measured every x ms, and we have no label on all these data.
So I'm here to ask you which algorithms/approach should be the best?

Edit : I'm working in an industrial context, and my data are sensor readings (temperature, pressure, power consumption, etc.) all from the same equipment.

One option is to treat it as a "supervised learning" problem.

Since

  a set of n data, measured every x ms

You have a time series. This can be treated as a supervised problem.

Say that you have observations for 10 minutes. In supervised learning, each sample would be :

X : Set of last t-1 , t-2,  t-3 and t-N observations (N is fixed, say 300)
Y : Observation t

You can train a model to predict next observation, given last 300 observations. If actual value is beyond a  certain % from the prediction, it is an anomaly.

Example : https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


How to update bias in CNN? How do we find the gradient and the back propagation error if we had a bias which just added a scalar value after the convolution calculation?

Like the update rule for bias terms in dense layers, in convolutional nets the bias gradient is calculated using the sum of derivatives of Z terms:

$$ dJ / db = \sum_h \sum_w dZ_{hw} $$

which J is the cost function, w is the width of the activation after convolution and h is the height of the activation after convolution.

db is computed by summing  dZs. It means you are summing over all the gradients of the conv output (Z) with respect to the cost.

Calculating the error of the net depends on the cost function that you have used. Depending on using cross entropy or mean-squared-error or other justified cost functions, you may have different update rule for other parameters of your net. But if you use cross entropy which is common for variants of classification tasks, the above update rule is used for updating bias terms.


How to get weightage of each factor for a prediction in a neural network classifier? I am working on a model to predict which employee is going to resign from a firm. The dataset has columns like Date of Birth, Date of Joining, Department, Gender, Marital Status, Years at company etc.

Using Tensorflow, I created a neural network classifier which is able to give the predictions (Going to leave/Not going to leave) and the probability that an employee is going to leave.

Let's say that the model gives 87% probability that a particular person is going to leave. I want to know which factor is contributing how much to this person's resignation (i.e. to find out the reason for the particular person's leaving)

How can I do that?

I'm using Jupyter notebook for the code and Keras for the neural network.

It's very difficult to understand the effect of each variable on the final output, since the weights of an input variable propagate their effect to all the units of the following hidden layers. Truth is, Neural Networks are very powerful predictors, but they are not very good when it comes to estimate feature importance.

One (very time consuming) way of doing it would be: substitute each variable with random noise that has the same mean and variance, and observe how the model's performance changes. Repeat this for all your variables. If performance gets significantly lower once you substituted a variable with random noise, then the contribution of it to the final accuracy has to be relevant.

As a more viable alternative, train a tree-based classifier (Random Forest or XGBoost) on the output. This class of models returns importance scores. You can use Neural Network for prediction, and the tree-based model to estimate what variable is more relevant for your task.


What mu and sigma vector really mean in VAE? In standard autoencoder, we encode data to bottleneck, then decode with using initial input as output to compute loss. We do activate matrix multiplication all over the network and if we are good, initial input should be close to output. I completely understand this.

But within a Vae, I have few problems, especially in understanding latent space vector. I think I know how to create it, but I am not really sure what is its purpose. Here I will write a few ideas, please let me know if I am correct.

We first create mu and sigma matrices, which are just matrix multiplication of previously hidden layer and random weights.
To create Z (latent matrix), we use parameterization trick. mu+log(0.5*sigma)*epsilon, which is a random matrix with 0 mean and 1 std. I have seen this Z (latent matrix) always produces Gaussian distribution, no matter what is the distribution of mu and sigma vectors. And here is my question. Why, why do we want to feed forward latent matrix with Gaussian distribution?
When we come to the end of decoding. We compute loss function and we penalize the network with KL divergence of Mu and Sigma Matrix. I don't understand why is it important in the first place to compare a distribution of Mu and Sigma matrices? I assume, that when we do back-propagation, mu and sigma become more close in term of distribution, but my question is why is this important, why those two matrixes must be close to each other in term of distribution?

I would really appreciate very simplistic answers with simple examples if it is possible.

In a variational antoencoder (VAE), the output of the encoder, $\mu$ and $\sigma$, is a deterministic function of input data $x$ (you put $x$ into your encoder neural network, and it generates $\mu$ and $\sigma$, there is nothing random here).
Then the hidden representaition $z$ (posterior) is sampled from the Guassian distribution parameterized by $\mu$ and $\sigma$. So notice that we don't talk about what 's the distribution of $\mu$ and $\sigma$ (since they are deterministic based on $x$, not random variables), but we do talk about the distribution of $z$.
"why do we want to feed forward latent matrix with Gaussian distribution?"
To be specific, we feed a Gaussian distributed variable $z$ into the decoder network. This is the basic model assumption of VAE regarding how samples are generated. Notice that one of the main difference between VAE and standard autoencoder is that VAE is a generative model, in the sence that you can randomly generate samples, instead of using a input $x$ to reconstruct something similar to $x$.
If we imagine a VAE trained on the MNIST hand written digits dataset, we want to randomly sample $z$~$p(z)=\mathcal{N}(0,1)$, and feed this $z$ into the decoder network which give us the output distribution (which is again a Gaussian), and sampling from that output distribution results in an image of a digit.
"Why compare a distribution of Mu and Sigma matrices? "
We do not compare the distribution of $\mu$ and $\sigma$. As explained above, we are talking about the Gaussian distributed hidden variable $z$ (posterior $q_{\theta}(z|x)$) parameterized by $\theta=(\mu,\sigma)$. We compare the posterior distribution of $z$ with its prior ($p(z)=\mathcal{N}(0,1)$) and the difference (measured by their KL-divergence) is used as a regularization term in the loss function.
The reason for that is we want the latent representation $z$ to be close to its prior (standard Gaussian). This is relevant to how we generate samples from the VAE (we first sample from the standard Gaussian prior). If we encode a $x$ to somewhere very far from this prior, it will be unlikely that we can generate such samples.
For example, a well trained posterior (first two dimensions) for MNIST looks like this (photo from here)

Notice that the posterior for each digit (e.g. dark blue for digit "0") is distributed like a Gaussian and not far from $\mathcal{N}(0,1)$, but difference digits do occupy different area.
Many online tutorials (e.g. here or here) have more detailed explanation about the concepts in VAE. I would suggest to read them.


MR images segmentation for feature extraction I have datasets of brain MR images with tumours, the tumours are already selected manually by a physicist using Image J.

I have read about segmentation, but I still couldn't understand how do they extract features from a segmented image.

should the images have only the tumor with a black background as shown in the below images, so the feature extraction will be processed on the whole image?
or do they extract features only on the region of interest using overlay, or layer that specifies the ROI?

and would Discrete wavelet transform DWT descriptor be a good choice for a descriptor?

"Feature extraction" is basically a process of reducing the input data (an image of X pixels) to a lower dimension (that is still enough to perform the model's task). So you have 2 options, to use a man-made features (for this you need to be the expert that determines what is a good and relevant feature) or train a feature extractor.

For a man-made feature extractor you need to understand what is the difference between the tumor area and the rest of the image (is it just a matter of the grayscale, where you can use some threshold value? maybe texture or spatial frequency where discrete wavelets can help?). This is not an easy task (there is a reason why the tumor was segmented by some physicist?? {I would think he should be some sort of a doctor}).
I am not an expert in MRI imaging but I've found some papers that talk exactly about your question:

"Feature Extraction for MRI Segmentation - Wiley Online Library"

"Feature extraction and selection from MRI images for the brain tumor classification"

If we are talking about training a feature extractor, then you need to define a model with an input and an output and some quantifiable training goal (loss/score function).
From my understanding, your final goal is to segment just the tumor part of the image, right?

Which means that your model's output should be a binary image with the same size as the input image. The pixels of the output image will be 0 for not-tumor and 1 for tumor pixels. This is similar to tasks of "Salient Object Detection".

The only difference with segmentation is that your output image is binary. In segmentation you will get an output image where each pixel is classified to one of k segments. This might also be relevant to you if you are trying to both segment the tumor and classify it as one of several known tumor-classes. Or if you want to divide the image into different segments, with the tumor being one of them.

If we are talking about a deep-learning model, both tasks share similar architectures known as Fully Convolutional Networks (with many different variations). You can read more about it at a paper named: "Fully Convolutional Networks for Semantic Segmentation".
In these models, the output of the second-to-last (refereed to as the "embedding layer") can usually be used as a feature extractor.


Is there a standardized way to do data analysis? Is there a standard way to do data analysis?
So for example, something like this:
 1. Data mining
 2. Data cleaning
 3. xx
 4. Data and result interpretations

I ask this because I'd like to do it for a paper which analyses a model of Data Science and, therefore, a common method would be appreciated. For a Literature Review, its f.e. (Transfield et. al 2003).
I actually could not find any reliable source, neither an approach to do it standardized.

If you only consider the process of making something from your data (excluding data extraction etc), I'd say something like :

Visualising your data to understand it
Cleaning your data, removing useless features etc
Feature engineering : keeping only relevant features, creating new features from existing ones (combining a few one, making ratio of 2 features for example)
Preprocesses (including simplifying your data if it's too precise, like too many classes in a category variable; encoding your variables so it can be taken by a model)
Testing and chosing the most adapted model
Tuning your model (using cross validation to have the best view of what your model does)
Try your model in brand new data (validation set) so you have the best estimation of how good it works, gather your results

This is pretty hard to give 'precise steps' since it differs from one project to another, depending on constraints, type of data, ...