Increasing minNumObj increasing accuracy in decision tree I have been using a J48 classifier in weka and have noticed that increasing
minNumObj -- The minimum number of instances per leaf leads to a small accuracy increase.

-M      Result.     Size    Num Leaves
2       73.8281 %   39      20
3       74.2188 %   39      20
4       74.4792 %   37      19
5       74.6094 %   25      13
6       74.2188 %   23      12
7       74.2188 %   23      12
8       74.349  %   23      12
9       75.2604 %   29      15
10      75.5208 %   29      15
11      75%         23      12
12      76.3021 %   23      12

However in several examples such as :

http://ww.samdrazin.com/classes/een548/project2report.pdf

The opposite its shown , which minNumObj increasing lowering the accuracy. The confidence factor was held constant at 1.0 to minimize post-pruning. Cross validation folds for the testing set (crossValidationFolds) was held at 10.

My results were made with a confidence factor of 2.5 but the difference between 2.5 and 1 is minimal.

https://pdfs.semanticscholar.org/9984/a7d06e04a347718cb8c7f645b72195bb11ce.pdf

See section 5.3. Measuring Performance: Precision and Recall

Why is the accuracy in my data going up and not down with an increase of minnumobj?

Your results are more in line with my experiences.  Increasing the minimum number of observations per leaf decreases the capacity of the model, increasing bias and decreasing variance.  Tree models are often overfitted without some kind of complexity reduction, so this usually is a beneficial tradeoff in net performance.

I'd speculate that the papers' results are different because of their data's small size: 683 and 4601... that second one doesn't strike me as too small, but...


How to improve results in classification problems (SVM, Logistic Regression and MultiNaive Bayes)? I am new on Machine Learning and building models but a lot of tutorials has given me the chance to learn more about this topic.
I am trying to build a predictive model for detecting fake news.
The percentage of data with labels 1 e 0 is the following:
       T
0    2015
1     798

It is not well balanced, unfortunately, as you can see.
I split the dataset as follows:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, stratify=y)

i.e. 70% train and 30% test.
I hope it makes sense, though I have unbalanced classes.
Then, after cleaning text by removing stopwords and punctuation (should I have done something else?), I ran different models, specifically MultiNaive Bayes, SVM and Logistic Regression, getting the following results:
MNB : 84%

  precision    recall  f1-score   support

           0       0.88      0.90      0.89       476
           1       0.45      0.40      0.42        95

    accuracy                           0.82       571
   macro avg       0.66      0.65      0.66       571
weighted avg       0.81      0.82      0.81       571

SVM:  Accuracy: 0.8336252189141856
Precision: 0.5
Recall: 0.2736842105263158 (Terrible results!)
Logistic regression: 0.8546409807355516
All the tutorial show that the steps for building a good model when you have some text, are removing stopwords and punctuation and extra words.
I have done all these things, but probably there will be something that I could do more to improve the results.
I read that, in general, who gets results above 99% met problems like overfitting: however, I would really have liked to get a 92% (at least).
What do you think? How could I improve further the models? Do you think that having unbalanced classes could have affected the results?
Any suggestions would be greatly appreciated it.

If you have a lot of data - down-sample your negative class to achieve 50/50 split on your fake news/real news classification. If you don't have much data - you can use techniques like SMOTE to up-sample the lesser class.
You seem to have better accuracy than randomly choosing fake/real which is a good sign. Your probability of a negative class based on your data split is 71.6% - and you are able to achieve 85.4% accuracy with LogReg. Don't get too down on that (especially if you are new to ML).
I would recommend checking out Gradient Boosting or Bagging algos if this is an NLP problem - these usually yield the best results for me when I'm encountered with sparse text data in classification.
As for the punctuation and stop words this is a common first step - however it's not good general advice for any problem. Do you think the presence of exclamation points might weed out some fake news in your data? If so I would include punctuation. If not - you're probably already on the right track. Removing stop words and punc only makes sense if the context of your specific problem calls for it.
More generally - your desire to reach 92% accuracy might not be possible given the difficulty of your problem. This is not to say it's not possible but keep in mind that the tutorials you may follow online are pre-determined to show that you can get good results. Some projects are simply harder than others (and some are not even possible given the context).
Good luck!


Predictive power of a dataset I am reading a book on machine learning for undergraduate. I am actually confused on linear regression flexibility as the say:

Occasionally, linear regression will fail to recover a good solution
for a data set. While this may be because our data doesn’t actually
have predictive power, it might also just indicate that our data is
provided in a format unsuitable for linear regression.

I read some questions here related to predictive power and I noticed that it's all about the model produced. What do we mean when we talk about the predictive power of a dataset?
I think of it as there is no relationship between features (as linear regression is intented to learn the relationship between the inputs $X$ and the output $Y$). But I am not convinced of this answer yet.

Usually predictive power refers to the model, rather than the data. I've occasionally seen some people use it in the way that the author of your book uses it (see this for example).
In the context of your book, yes, predictive power refers to whether input can be mapped to target output $X\rightarrow Y$. We can infer a dataset's "predictive power" by trying to model it (e.g. linear regression). If the model performs poorly, then there are two possibilities as the book says: either the dataset is not predictive (i.e. it does not offer a clean mapping from input to target output) or the methods we are using are unsuitable to model the mapping.

Some examples of both situations:

If you generated random data for $X$ and $Y$, the resulting dataset would (probably) have no predictive power as no model could reasonably generalize the mapping $X\rightarrow Y$.

If you have a nonlinear mapping, then linear regression would not fit it well. For example, if our dataset was such that $y_1$ is mapped to by $||\vec{x}||<\alpha $ and all other inputs map to $y_2$, then our dataset is extremely predictive, but our linear regression model cannot fit it (since the mapping is nonlinear). In this toy example, it's easy to see the predictive power of the dataset, particularly if the input is in 2D/3D since we could just plot it. However, manually observing such trends in highly dimensional space using actual data can be very difficult, hence we use the tools that you are learning to help interpret the data. Also, when there's nonlinearity, it's difficult to statistically evaluate the dataset itself. Variables with linear relationships are simple to correlate (e.g. Pearson's correlation coefficient) but nonlinearities can make correlation difficult. I assume that this is why your book defers to vague terminology as it's probably for pedagogical, rather than pedantic, purposes. After all, it gets the point across without needing to discuss the ongoing research into quantifying nonlinear correlations.


Estimating Variance Reduction Resultant from Additional Data I couldn't quite think of how best to title this, so recommendations are welcome. Same goes for the tags (I don't have the reputation to use the tags that I thought were appropriate). The question is this:

"Suppose you have N pairs of observations, (x,y), and you have a model with some unknown parameters, B, that estimates the relationship between x and y, F(x,B) -> y. Now suppose you determine B using the method of least-squares (and, implicitly, that all the assumptions of least-squares are satisfied). The parameters, B, are themselves random variables, each with its own variance. Is there any way to estimate the reduction (or increase) in the variance of B that would result from applying the same method of least-squares to N+1 pairs of observations?"

The question is asked in the context of experimentation. If each data point costs $X, an affirmative answer to the question would go a long way in determining whether or not to continue testing.

I would think about starting with a power analysis: i.e. how many data points do you need to measure the effect (or parameter) that you are interested to a specified level of confidence, ceteris paribus? Then, you estimate a cost.


When does boosting overfit more than bagging? If we consider two conditions:

Number of data is huge
Number of data is low

For what condition does boosting or bagging overfit more compared to the other one?

I read your question as: 'Is boosting more vulnerable to overfitting than bagging?'

Firstly, you need to understand that bagging decreases variance, while boosting decreases bias.

Also, to be noted that under-fitting means that the model has low variance and high bias and vice versa for overfitting.

So, boosting is more vulnerable to overfitting than bagging.


Are LDA clusters identical across different runs? for a given corpus are the Latent Dirichlet Allocation clusters for it is unique in general?
How about the gensim  multi-process implementation of LDA? are there unique or they will be different for every run of code?

No, as there is randomness in the method implementation, for example here (in LdaModel of the gensim library). Hence, it can affect your final result in each run. Therefore, if you want to keep the result reproducible, you can set the random_state property of the model to a constant seed (see the documentation for more details).


How are ANN's, RNN's related to logistic regression and CRF's? This question is about placing the classes of neural networks in perspective to other models.

In "An Introduction to Conditional Random Fields" by Sutton and McCallum, the following figure is presented:

It shows that Naive Bayes and Logistic Regression form a generative/discriminative pair and that linear-chain CRFs are a natural extension of logistic regression to sequences.

My question: is it possible to extend this figure to also contain (certain kinds) of neural networks? For example, a plain feedforward neural network can be seen as multiple stacked layers of logistic regressions with activation functions. Can we then say that linear-chain CRF's in this class are a specific kind of Recurrent Neural Networks (RNN's)?

No, it is not possible.

There is a fundamental difference between these graphs and neural networks, despite both being represented by circles and lines/arrows.

These graphs (Probabilistic Graphical Models, PGMs) represent random variables $X=\{X_1,X_2,..,X_n\}$ (circles) and their statistical dependence (lines or arrows). They together define a structure for joint distribution $P_X(\boldsymbol{x})$; i.e., a PGM factorizes $P_X(\boldsymbol{x})$. As
a reminder, each data point $\boldsymbol{x}=(x_1,..,x_n)$ is a sample from $P_X$. However, a neural network represents computational units (circles) and flow of data (arrows). For example, node $x_1$ connected to node $y$ with weight $w_1$ could mean $y=\sigma(w_1x_1+...)$. They together define a function $f(\boldsymbol{x};W)$.

To illustrate PGMs, suppose random variables $X_1$ and $X_2$ are features and $C$ is label. A data point $\boldsymbol{x}=(x_1, x_2, c)$ is a sample from distribution $P_X$. Naive Bayes assumes features $X_1$ and $X_2$ are statistically independent given label $C$, thus factorizes $P_X(x_1, x_2, c)$ as $P(x_1|c)P(x_2|c)P(c)$.

In some occasions, neural networks and PGMs can become related, although not through their circle-line representations. For example, neural networks can be used to approximate some factors of $P_X(\boldsymbol{x})$ like $P_X(x_1|c)$ with function $f(x_1,c;W)$. As another example, we can treat weights of a neural network as random variables and define a PGM over weights.


What is the difference between "equivariant to translation" and "invariant to translation" I'm having trouble understanding the difference between equivariant to translation and invariant to translation.

In the book Deep Learning. MIT Press, 2016 (I. Goodfellow, A. Courville, and Y. Bengio), one can find on the convolutional networks:

[...] the particular form of parameter sharing causes the layer to have a property called equivariance to translation
[...] pooling helps to make the representation become approximately invariant to small translations of the input

Is there any difference between them or are the terms interchangeably used?

Equivariance and invariance  are sometimes used interchangeably in common speech. They have ancient roots in maths and physics. As pointed out by @Xi'an, you can find previous uses (anterior to Convolutional Neural Networks) in the statistical literature, for instance on the notions of the invariant estimator and especially the Pitman estimator.

However, I would like to mention that it would be better if both terms keep separate meaning, as the prefix "in-" in invariant is privative (meaning "no variance" at all), while "equi-" in equivariant refers to "varying in a similar or equivalent proportion". In other words, one in- does not vary, the other equi- does.

Let us start from simple image features, and suppose that image $I$ has a unique maximum $m$ at spatial pixel location $(x_m,y_m)$, which is here the main classification feature. In other words: an image and all its translations are "the same".
An interesting property of classifiers is their ability to classify in the same manner some distorted versions $I'$ of $I$, for instance translations by all vectors  $(u,v)$.

The maximum value  $m'$ of $I'$ is invariant: $m'=m$: the value is the same. While its location will be at $(x'_m,y'_m)=(x_m-u,y_m-v)$, and is equivariant, meaning that is varies "equally" with the distortion.

The precise formulations given (in mathematical terms) for equivariance depend on the class of objects and transformations one considers: translation, rotation, scale, shear, shift, etc. So I prefer here to focus on the notion that is most often used in practice (I accept the blame from a theoretical stand-point).

Here, translations by vectors  $(u,v)$ of the image (or some more generic actions) can be equipped with a structure of composition, like that  of a group $G$ (here the group of translations). One specific $g$ denotes a specific element of the translation group (translational symmetry). A function or feature $f$ is invariant under the group of actions $G$ if for all images in a class, and for any $g$,
$$f(g(I)) = f(I)\,.$$

In other words: if you change the image by action $g$, the values for feature or function $f$ are the same.

It becomes equivariant if there exists another mathematical structure or action  (often a group again) $G'$ that reflects
 the
transformations  (from $G$) in $I$  in a meaningful way. In other words, such that for each $g$, you have some (unique?) $g' \in G'$ such that

$$f(g(I)) = g'(f(I))\,.$$

In the above example on the group of translations, $g$ and $g'$ are the same (and hence $G'=G$): an integer translation of the image reflects as the exact same translation of the maximum location. This is sometimes refered to as "same-equivariance".

Another common definition is:

$$f(g(I)) = g(f(I))\,.$$

I however used potentially different $G$ and $G'$ because sometimes $f(.)$ and $g(.)$ do not lie in the same domain. This happens for instance in multivariate statistics (see e.g. Equivariance and invariance properties of multivariate quantile and related functions, and the role of standardisation).
But here, the uniqueness of the mapping between $g$ and $g'$ allows to get back to the original transformation $g$.

Often, people use the term invariance because the equivariance concept is unknown, or everybody else uses invariance, and equivariance would seem more pedantic.

For the record, other related notions (esp. in maths and physics) are termed covariance, contravariance, differential invariance.

In addition, translation-invariance, as least approximate, or in envelope, has been a quest for several signal and image processing tools. Notably, multi-rate (filter-banks) and multi-scale (wavelets or pyramids) transformations have  been design in the past 25 years, for instance under the hood of shift-invariant, cycle-spinning, stationary, complex, dual-tree wavelet transforms (for a review on 2D wavelets, A panorama on multiscale geometric representations). The wavelets can absorb a few discrete scale variations. All theses (approximate) invariances  often come with the price of redundancy in the number of transformed coefficients.
But they are more likely to yield shift-invariant, or shift-equivariant features.


Regression: Should "known outputs" be also activated? Okay so, I understand that inputs are sent directly to the network (basically being multiplied to weights of the nodes, receive a bias, and gets activated) and then the network produces an output through this feed forward method.

Now, in order for the network to "learn," we determine the error by comparing the predicted output versus the "known output / answer," and then back-propagate. Right?

What I don't get it is if the output goes through some activation function, say Sigmoid (results to a value between 0.0 - 1.0), how can it ever learn if the known output is a continuous value such as "Salary", "Number of Something", or in the use-case I have, "Volume of Orders" which ranges from anywhere between 300 to 1500.

// Example I have 7 inputs
// They go through a hidden layer that consists 16 hidden neurons
// and my output is a single neuron that's supposed to be "Volume of Orders"
//
// MyInputLayers[] --> HiddenLayers[] --> Output=0.123
//
// Since I learn from tutorials that Output layers go through an activation,
// I think it's impossible for the network to learn because a
// prediction of 0.123 compared to the "known output" that is, say 615
// is almost ridiculous

Should I apply Sigmoid to the "known output" as well? If so, how can I remap it when the network has already learned and is ready to do some predictive magic?

I must be missing / not understanding something here. Thank you in advance.

p.s. My dataset is composed of 6 months of records; an Excel file that has 200,000+ rows and 8 columns (7 leading, 1 lagging).

The sigmoid function is usually used for classification tasks. Basically, it converts a continuous value into a binary variable (0 or 1) based on a certain threshold. An example is predicting breast tumor as malignant (1) or benign (0) based on whether or not the output is greater than the threshold value (typically, 0.5).

If you want to use ANN for regression, first you'll have to change your cost function into something more appropriate, like mean squared error (MSE). Then adapt the network to provide continuous output, by using a linear activation function, for example.

Source: https://www.quora.com/Can-Deep-Learning-and-Neural-Networks-be-useful-for-regression-problems-where-the-output-variable-has-an-unknown-or-varying-in-real-time-upper-bound


How do I setup a server in the cloud for machine learning? As datasets and the number of parameters get larger, it becomes increasingly difficult to run validation locally because of limited disk size and computing power.  As a result, people might use an old laptop or a server in the cloud.  In particular, I'm interested in learning more about the second option.

For the setup, it sounds like I will need a storage system and a computing system.  Is this S3 and EC2?  Redshift and Amazon ML?  Combinations from some other vendor?  At the moment, I'm asking as an amateur participant for Kaggle, but I'd also be curious to know what the pros use.

On the machine itself, is there any way to interact with a GUI rather than a command line to set this up? Would I have to install Anaconda or some other Python distribution before getting started?  Is there a specific file structure to use? What are the main pitfalls to watch out for?

Ultimately, I am looking for the practical advice to set everything up.  If you're able to just offer links to documentation, that would be immensely helpful as well.  Thanks!

If you are looking for an external storage, then I would suggest you Redshift.

Redshift is a central warehouse and a columnar data store. It allows complex and huge data aggregations and joins; so it is a nice bet for serious Kaggle participants. (I personally use Redshift as data science architecture for kaggle.)

Combinations from some other vendor?

No, a combination of Redshift and S3 is enough. The data is stored in S3, and then loaded into Redshift using the COPY command.

Is there any way to interact with a GUI rather than a command line to set this up?

Yes, there is a CLI interface for Amazon AWS. However, you can use the boto library on Python for handling this stuff.

Would I have to install Anaconda or some other Python distribution before getting started?

Not a compulsion.  Anaconda is a wrapper of numerical and scientific libraries. You can install them on your own or intall Anaconda.

Check out PySpark, which allows you to do handle BigData in Python.


Pytorch convolution input reshaping I am new to CNNs, and I'm trying to follow along with a Pytorch DCGAN tutorial by reimplementing it in Keras. Clearly there are some differences in the frameworks, but in particular I am struggling to understand how the input noise vector is handled per the following image of the generator architecture:

The code goes on to define the first layer of the network as:

# input is Z, going into a convolution
nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
nn.BatchNorm2d(ngf * 8),
nn.ReLU(True),
# state size. (ngf*8) x 4 x 4

Where nz = 100 (size of noise vector) and ngf = 64.

How does that work? Is the noise vector implicitly reshaped by that first convolutional layer to a (?, 4, 4) tensor, as suggested by the diagram? How would that even work? Is there an implicit dense connection between the noise vector and the convolutional layer? How does that first layer result in a tensor of shape (64*8, 4, 4) per the comment?

Turned out to be a dumb question in the end - the noise vector is simply treated as a 1x1 array with "depth" (i.e. number of channels/features) of 100, which can of course be processed by the ConvTranspose2d layer. This reshaping to 1x1x100 just has to be explicitly specified as the first layer in keras:

model.add(Reshape((1, 1, 100), input_shape=(100,)))


Is there potentially data leakage during imputation for time-varying sensor data? I have a time-varying dataset that contains some missing data. I have sensors that continuously monitor some properties at evenly-spaced intervals and I would like to impute the missing values using basic interpolation for both the training and test set. This is a time-series binary classification problem (e.g., based on the entire time-series present, classify as either 1 or 0). I am concerned that taking data from the future to interpolate the missing value is a form of data leakage.
My reason for believing it is not is primarily based on the fact that I am not doing forecasting. I am not trying to predict future values of these sensors, just impute the missing dynamic variables with its most likely values (in fact, based on my domain knowledge and some experiments simple interpolation is very accurate at predicting the true values). If I were trying to predict future sensor values (e.g., forecasting) this would certainly be data leakage, correct?

If you are using information from the future to impute missing data would be data leakage as you would not have this extra information when the model is in production and trying to predict future values. To prevent data leakage, make sure to only use values that are available at the date/time you want to predict. If you were to impute the missing data based on historical data you would not be leaking data since you have this data available at the moment of prediction (as the term "historical" implies).


Updating the weights of the filters in a CNN I am currently trying to understand the architecture of a CNN. I understand the convolution, the ReLU layer, pooling layer, and fully connected layer. However, I am still confused about the weights.

In a normal neural network, each neuron has its own weight. In the fully connected layer, each neuron would also have its own weight. But what I dont know is if each filter has its own weight. Do I just have to update the weights in the fully connected layer during back propagation? Or do the filters all have a separate weight that I need to update?

In a normal neural network, each neuron has its own weight.

This is not correct. Every connection between neurons has its own weight. In a fully connected network each neuron will be associated with many different weights. If there are n0 inputs (i.e. n0 neurons in the previous layer) to a layer with n1 neurons in a fully connected network, that layer will have n0*n1 weights, not counting any bias term.

You should be able to see this clearly in this diagram of a fully connected network from CS231n. Every edge you see represents a different trainable weight:

Convolutional layers are different in that they have a fixed number of weights governed by the choice of filter size and number of filters, but independent of the input size.

Each filter has a separate weight in each position of its shape. So if you use two 3x3x3 filters then you will have 54 weights, again not counting bias. This is illustrated in a second diagram from CS231n:

The filter weights absolutely must be updated in backpropagation, since this is how they learn to recognize features of the input.  If you read the section titled "Visualizing Neural Networks" here you will see how layers of a CNN learn more and more complex features of the input image as you got deeper in the network. These are all learned by adjusting the filter weights through backpropagation.


Server log analysis using machine learning I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and Sparks MLlib(or PredictionIO).An example of the desired result would be to be able to predict based on the exception logs gathered to be able to predict which user is more likely to cause the next exception and at which feature(and bunch of other stuff to keep track and improve optimization of the application).

I have successfully been able to ingest data from ElasticSearch into spark and create DataFrames and map the data needed. What I would like to know is how do I approach the Machine Learning aspect of my implementation. I've been through articles and papers that talk about Data preprocessing, training the data models and creating labels and then generating predictions.

The questions I have are

How do I approach transforming the exiting log data into numerical vectors which can be used to datasets to be trained.
What algorithms do I use to train my dataset(with the limited knowledge i've gathered the past couple days, i was thinking bout implementing linear regression, please suggest which implementation would be best)

Just looking for suggestions on how to approach this problem.

Thank You.

I don't think you necessarily need to convert the individual log entries into vectors for use in an algorithm.  I would guess that what you are interested in is a sequence of log entries, which represent a series of events, ordered in time, which together make up a series of 'cases'.  Here the relationship between a series of collected log entries is important.

If this is the case then you might want to consider using Process Mining techniques.  This allows you to build models of your process (the use of your application) and determine patterns of process steps, along with errors and rework steps.

There is a good introduction course on Coursera, here.  There are even some developed, commercial packages like 'disco' to help you with the analysis and visualisation


Recommender System for mostly unique user and items I am trying to develop a recommender system for a job matching problem. My data consists of past matched candidate profiles and job profiles as well as if there was a success such that both, candidate and employer, accepted the offer.
Now, what I want is to build a recommendation system for future candidate and jobs which will recommend a fitting candidate to an open job posting based on past successes.
The big problem here is, that each candidate and each job is unique. So, if a matching was successful, both candidate and job won't be available anymore. As for recommender systems, this is a huge problem, since both, collaborative filtering and content-based filtering, rely on user ratings/interaction and similar users, who already rated items for which a recommendation should be calculated. But these items don't exist anymore. What I have here is basically a cold-start Problem for user and items.
One way I could tackle this problem would be to user a knowledge-based recommender system, since I have candidate and job profiles describing skill, position, experience and so on. I could construct common feature vectors and simply calculate similarity between them, recommending the top n most similar candidates to a job posting. But I want to use also the knowledge of the past job-matchings since they contain information about what kind of candidate was suitable for a specific job.
Does anybody have an idea how I could do this or had a similar problem?

There are a lot of different ways you could approach this problem. Sort of depends on what kind of results you're hoping to get for your particular situation. I'm going to break up my answer into sections, as this will be a longer reply because I want to provide as much context as possible to the problems you will face with building a job/candidate recommender as they are quite numerous.
Some suggestions on how to deal with your current problem:
Your biggest challenge is data sparsity, so a few suggestions:

If you're not already, use a Factorization Machine as your core model. At least until you're ready to implement a more complicated neural net or something. FMs deal with sparse conditions really well and can take in the wide variety of data & embeddings that you're going to be using in this kind of task. Just doing this alone I think will get you on track to solving the problem you're talking about.
You need to define features to make impressions more dense. An example could literally be consolidating companies with sparse histories into segments:

['Netflix', 'Facebook', 'ComputerVisionCompaniesUnder10Employees', ...]
Then from now on any company that falls under the category of ComputerVisionCompaniesUnder10Employees will only be represented as that label.
You then need to one-hot-encode normalized job title as a separate feature from company and candidate.
Given how FMs work, you shouldn't need to do this, just including the feature should accomplish the same thing, but I've noticed that sometimes explicitly replacing companies labels like that does boost performance quite considerably.
So now variance in recommendation for companies under the same company label would depend on the job title and contextual features about that particular job.
If two different companies under that same label had similar job postings, then they would get recommended similar (if not the same) candidates and that's OK.
As far as candidate abstractions go, you're probably going to want to find some way of normalizing and grouping skills as the skills matrix can get quite sparse on its own.
Additionally adding in aggregations of features of companies that a candidate previously worked for could help. Like how many "Large companies" has the candidate has worked for as an example.
The more categorizations you can feed to the FM the better, the FM will calculate  interaction embeddings between every set of features. The important bit is to separate the different features about the jobs/candidates and abstract separately. Don't calculate abstractions for job postings, calculate abstractions for companies, job titles, and skill descriptions, and let the FM figure out how all those things relate.
Data issues in Job/Candidate Recommendation:

Data sparsity of impressions.

As you implied in your question, the only knowledge you currently have on candidates' job preferences is their past jobs and content related features. Getting a measure of whether or not the candidates liked their past jobs is also difficult.

Lower than expected information entropy in job-related data points.

Almost all resumes, LinkedIn profiles, and job postings are all designed with brevity in mind. It's a well known fact that recruiters spend very little time reading resumes, so it's become a standard that people should aim to keep their resume under 1 page. This makes information content lower than one may think. The same is often true of job postings, in fact most candidates don't even read job postings due to the prevalence of "1 click" apply features.

Non-trivial normalization of different job related entities

There aren't really any good open source standard job title sets out there last time I checked. Good skill ontologies are also hard to come by. These hurdles are a big pain, but you can overcome them with a lot of hard work. I'd start by picking some standard sets as truth (I personally recommend checking out the data referenced here). There's a couple of ways to try and do this automagically, but I personally haven't found a great way of dodging a lot of manual cleanup.
Notes on candidate recommendation:
Candidate recommendation is really only something that recruiter platforms like LinkedIn have had to think about all that much, as most job platforms recommend jobs to candidates and then candidates hit apply and that's when they show up in the employer's applicant pool.
From my experience, the features that recruiters care the most about when being recommended candidates is whether the technical skills match, whether or not they went to a prestigious university, and years of experience. As mentioned earlier, it's difficult to get any measure of what one might call "personality" from resume/profile data, so they just want to get candidates that match on a call to further vet them. Another note here is that most candidates don't actually read the JD, so getting candidates that are genuinely interested in the position beyond the paycheck/title saves recruiters a lot of time.
Notes on job recommendation:
There are actually very few platforms that do true job recommendation in the way that's been described here. Most just do job search. When you go to indeed and search for a job, indeed knows very little about the visitor on it's site beyond what they're typing in the search box. Indeed's success hinges on its search ranking far more than its recommender performance. Another note is that in my experience senior candidates like to have a lot of control of what jobs get thrown their way and junior candidates like to shotgun apply to every job that vaguely resembles what they're looking for as fast as possible. The junior case is at odds with what recruiters want, so the goal is make sure that candidates are only shown jobs that they are fit for.
My experience:
I've actually started a recruitment company before and have had to build a job recommender for candidates. Now this company didn't work out for reasons unrelated to the recommender performance. The job-search market is a very saturated one and surviving/innovating in it given the problem's constraints is very difficult but that's a whole other subject on it's own.


Why prior in MAP could be ignored? A posterior  $p(\theta\vert x) = \frac{p(x \vert \theta)p(\theta)}{p(x)} $

Many materials say that since the $p(x)$ is a constant, the $p(x)$ can be ignored. Thus, $p(\theta\vert x) \propto p(x \vert \theta)p(\theta)$

My question is why $p(x)$ is a constant and ignored. Is this because even though we don't know the distribution x, there is a corresponding true distribution for $x$? So, $p(x) $ is a constant (we don't know but already determined) and thus, can be ignored?

The maximum a posteriori definition usually goes like this:

$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} $$

and given that $p(x)$ is independent of $\theta$, it's not needed for finding the $\arg\max_{\theta}$, then you have

$$ \arg\max_{\theta} p(\theta|x) = \arg\max_{\theta} \frac{p(x|\theta)\cdot p(\theta)}{p(x)} =  \arg\max_{\theta} p(x|\theta)\cdot p(\theta)$$


How can I determine the relationship between spam and weekdays? I am trying to check if there is a correlation between spam emails and weekdays.
My dataset looks like as follows:
  Spam? Day
0   1.0 Saturday
1   1.0 Saturday
3   0.0 Saturday
5   1.0 Saturday
7   0.0 Friday
... ... ...
346 0.0 Friday
348 1.0 Friday
361 0.0 Saturday
383 1.0 Thursday
387 1.0 Friday

where 1 means spam and 0 not spam.
I have tried as follows
corr = (numpy.corrcoef(df['Spam?'],df['Days']))

I do not know how to explain a possible relationship between these two variables and if a plot could help to better visualise data and relationship.

(started as a comment but it turned out to be longer than expected)
With a dataset like this a simple barplot could be very insightful: on the X axis the days of the week, on the Y axis the frequency, with two bars (spam/not spam using different color) for each day. A slightly more advanced version: two boxplots, one for weekdays the other for weekends. A boxplot is kind of overkill for only 5 (mon-fri) and 2 (sat-sun) values but it's easy to do and shows the big picture.
In order to test whether any difference (e.g. weekdays vs weekends) is significant I think this is a good case for a chi square test.


Model to predict coronavirus (covid19) spread im new in data sience and machine learning but i have some mathematical and statistics backgroud. I really just want some information about models (like papers or raw models). So if you have any information please share them with me.

thank you

For a beginner, I would say the SIR model is a great place to start:

https://idmod.org/docs/general/model-sir.html

Numberphile did a great video on using SIR to predict Covid-19:

https://www.youtube.com/watch?v=k6nLfCbAzgo

Hopefully this can get you started on your journey!


Pre-processing - Removing outliers I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data:

It looks like outliers are 4,5,6 and should be removed.

Now plotting the field "A" in test file also shows somewhat similar results to the image shown. In that case, should I consider 4,5,6 as outliers, and remove them from train data, or keep them? Any pointers are appreciated.

In general, its not acceptable to drop an observation just because it is an outlier.

Sometimes outliers give us very important information about data.
Removing outliers is legitimate if:

it is obvious that's incorrect data
not change the results and assumptions.

How to handle outliers? Try other models or use data transformation.


Data Scientist Consulting Interview Guide Does anyone have any books or blogs that specifically sheds light on questions to ask your organization (from a consulting POV) as a data scientist? I am a new data scientist, which I have a background in consulting and predictive analytics.

There are several good reads such as "The Data Scientist's Field Guide" by Booz Allen, "The McKinsey Mind" from McKinsey, or maybe "Business Analytics for Managers" from SAS. I have come across interview guides to understand and execute strategy, but none that specifically structure questions for a preliminary assessment of analytic capability.

Any recommendations would be appreciated.

Thank you.

This might be a resource that you would be interested in. It's a preliminary assessment for health centers, but a lot of the structure can be used in any industry.


Permutation feature importance vs. RandomForest feature importance What is the difference between Permutation feature importance vs. RandomForest feature importance? What are the disadvantages vs. advantages of the two techniques?

Forest:

Forest paper
"We show that random forest variable importance measures are a sensible means for variable selection in many applications, but are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories."

This is saying that if a feature varies on its ability to detect based on class it will be fault. This seems to imply the methodology is sensitive to data outliers like maybe a feature if 90% responsible for classification of 5% of the variables it may be the most important feature even though that might not really be accurate.

Permutation:

 Scikit-learn
"Permutation feature importance is a model inspection technique that can be used for any fitted estimator when the data is rectangular.
This is especially useful for non-linear or opaque estimators. The permutation feature importance is defined to be the decrease in a model score when a single feature value is randomly shuffled 1. This procedure breaks the relationship between the feature and the target, thus the drop in the model score is indicative of how much the model depends on the feature. This technique benefits from being model agnostic and can be calculated many times with different permutations of the feature."

The data being rectangular means it is a multivariate feature array table. The model working in non linear scenarios means it is able to be beneficial in prediction even when outputs follow non linear functions like XOR.


Why in this case are gradient steps not perpendicular to contour lines? There is a theorem that gradient at point is perpendicular to tangent line to contour line at given point.
Why in this picture it seems that this rule is not respected?

source: http://www.deeplearningbook.org/contents/numerical.html page 89

And also in this case:

source: http://www.deeplearningbook.org/contents/optimization.html page 310

In second case it is even weirder because the steps don't seem to be perpendicular to contour lines but to previous gradient vector? Am I missing something or treat images to serious? Thank in advance and regards.

That would be the case if the steps taken by gradient descent are "infinitesimal" in the mathematical sense. But in fact, it takes steps with some finite length defined by the learning rate. But the problem is, the gradient is calculated at each point, and does not really calculate how the slope changes at some finite length later. If you choose a large learning rate, you may "offshoot" from the optimal direction as shown in the figure. If you choose a small enough learning rate those oscillations will be minimal and you may move "almost" perpendicular to the contour lines and look like what you describe in the question, but it will take a very long time to complete training.

The second part of the question refers to the section titled "Conjugate Gradients" and it refers to a specific optimization method. The reason for the perpendicular lines in the second part is because of the vanishing gradient at the turning points. Quoting the text:

  The method of steepest descent involves jumping to the point of lowest
  cost along the line deﬁned by the gradient at the initial point on each
  step

Such a point is where the "directional derivative" with respect to theta, theta being the parameter in a given direction vanishes. When the derivative vanishes with respect to the parameter of the direction, The only possible way to go will be in a perpendicular direction. That is why they are called "conjugate" to each other, hence the name of the optimization method. This is detailed in the text above the figure.

To summarize the situation for that figure; An initial direction is chosen at the starting point by calculating the gradient there, and you keep moving in that direction until the gradient vanishes. That point is the minimum in that direction, so if you go any further on that straight line, the cost function will increase. The only direction you can go in which the cost function does not increase can be perpendicular to the initial direction.

To visualize this, look at image 4.5 in your first upload - page 88. Starting from that top of the saddle and following the gradient, you go down to the middle of the saddle. At that point going any further in the same direction will only take you up, while you want to go down. The only way to achieve this is to take a 90 degree turn at the middle of the saddle.


Dropout in a CNN vs Dropout in a FCNN In the PyTorch nn module there are 2 types of dropouts:

A normal Dropout - During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution. Each channel will be zeroed out independently on every forward call.
A Dropout2d - Randomly zero out entire channels. Each channel will be zeroed out independently on every forward call with probability p using samples from a Bernoulli distribution.

From Documentation

My question is when we use CNN, which of the dropout we use in the Convolutional layers. In this paper it is being said we use the first kind. But I am not so sure what is the general practice.

Check this paper:

https://arxiv.org/pdf/1411.4280.pdf

In section 3.2 they discuss in detail why standard dropout fails in convolutional layers and the idea of spatial dropout. Fully connected networks learn via connections between single neurons while convolutions learn via features. In a sense, dropping out entire features instead of single neurons can be thought as the convolutional version of regular dropout.


Is a correlation matrix meaningful for a binary classification task? When examining my dataset with a binary target (y) variable I wonder if a correlation matrix is useful to determine predictive power of each variable.

My predictors (X) contain some numeric and some factor variables.

Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.

There are quite a few answers on stats exchange covering this topic - this or this for example.


Need help with prolog program I am new to prolog and need help on how to write a prolog program. Here is what i am trying to do. I have downloaded the dataset from this link - https://archive.ics.uci.edu/ml/datasets/Blood+Transfusion+Service+Center
My goal is

Use Any of the decision tree algorithms to build a decision tree for the given data
Create rules from the decision tree.
Code the rules into a Prolog Knowledge base
Get the data about the blood donor from the user and predict if the donor has donated in March 2007.

I need some guidance on how to start the work. I have developed algorithms in python and understand #1 and #2 but have no idea on work with #3 and #4.

I can give you some basics but the proper way to do this is to learn the prolog language, and it's very different from any other programming language.
In Prolog you define predicates (similar to functions) which contain conditions for the predicate to be satisfied, for example:
dt(Feat1, Feat2, Feat3, labelA) :-
    Feat1 =< 3,
    Feat2 == 'green',
    Feat3 >= 0.54.

dt(Feat1, Feat2, Feat3, labelB) :-
    Feat1 =< 3,
    Feat2 == 'green',
    Feat3 < 0.54.

dt(Feat1, Feat2, _Feat3, labelC) :-
    Feat1 > 3,
    Feat2 == 'green'.

dt(_Feat1, Feat2, _Feat3, labelB) :-
    Feat2 == 'blue'.

Here the predictate is dt and it has 4 clauses. In order for dt to be satisfied at least one of the clauses needs to to be true (i.e. all the conditions it contains must be true).
Variables are represented with a capital, for instance Feat1, Feat2, Feat3. Prolog will try to find an instanciation of the variables which satisfies one of the clauses (it will actually try all the possibilities until one of them works, it's a solver).
In each clause above the label is the last parameter of the predicate. Basically each clause means "if all the conditions after :- are satisfied, then the label is labelX".

Assuming the 4 clauses above have been saved in a file dt.pl, one can use for instance the SWI Prolog interpreter like this:
?- consult('dt.pl').
true.

Now the way to use this toy decision tree would be:
?- dt(4,'green',0.3, Label).
Label = labelC .

?- dt(4,'blue',0.3, Label).
Label = labelB.

Note the capital L for Label: it means that now we let prolog find a value of Label which satisfies the predicate dt given the values provided for the features.

This is the basic idea. A more advanced and more generic representation of the DT would be possible using recursion and unification, but this would require an advanced understanding of Prolog.


Is there a way to get y_pred values from saved Keras model? I have a Keras model saved in a .h5 file. As you know there are a y_pred and a y_act that confusion matrix creates from, at run time, it's easy to get y_pred values but my model is saved and now I need the y_pred values from that saved model. Is there a way to do that?

If I've figured it out correctly, the answer is no. The point is that your saved model solely contains the network architecture and the parameters it has. What you want relates to the recall phase where you have to provide input to get output. This means that you need input data to be fed to your network in order to get output. What you want can be done using another approach. First, load your network and feed your data to your model. After that, get the outputs and store the inputs and outputs alongside each other using Numpy save method or maybe .h5 format.

The flow for achieving the y_pred can be like the following sequence of actions:

Load your model.
Feed your data to your model and get y_pred.
define a Numpy array of inputs and a Numpy array of outputs.
Store inputs, real outputs and y_preds using the methods which are available.
Later, when you want to make your confusion matrix, you can load your inputs and outputs, and the real outputs to make your matrix.


Unsupervised sequence identification I am looking for the best method to go from a sequence of events such as

time event
1 a
2 b
3 a
4 b
5 c
6 d
7 c
8 d
9 e

Where each letter corresponds to a certain event that occurs at a time. I want to reduce the number of events by aggregating frequently occurring events into a new event. A possible solution data set would look like,

1 a'
2 a'
3 a'
4 a'
5 b'
6 b'
7 b'
8 b'
9 e

where the clusters are created because they occur in a sequence following each other.

I was looking at the text mining algorithms in R with tm or RNA sequenceing
with edgeR. But I have no experience in this so I was hoping that someone can
shed me some light on a common approach for this type of problems.

You may use ngrams to get the frequent sequences of you events.

Here a little example

 library(tau)
 seq <- "ababcdcde"
 textcnt(seq, method="ngram", n=3L, decreasing=TRUE)

   _   a  ab   b   c  cd   d  _a _ab aba abc  ba bab  bc bcd cdc cde  dc dcd  de de_   e  e_
   2   2   2   2   2   2   2   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1   1

Select the longest and most frequent ngram, here ab and cd.
You may also trade off the length and frequency to get maximal compression. The n parameter limits the length of the ngram.


Optimizing regression weights for NN outputs with PyTorch So I'm basically trying to fit a regression on the relation of the input and output of a neural network model. Then the idea is, that these estimated regression weights should be optimized to some specific target value (let's say matrix of ones). So the weights are optimized, but have a direct relation to the neural network weights. For a simple problem, I can find a solution on paper, but I fail to implement this. The loss is not decreasing, but oscillating.
# Some arbitrary PyTorch model
my_nn = Net()

# Our target regression weights
target = torch.ones(2, 2)

# Optimizer for NN weights
optimizer = optim.Adam(my_nn.parameters(), lr=0.0001)

# Arbitrary loss function
criterion = nn.MSELoss()

# Training loop
my_nn.train()
for epoch in range(500):
    print('Running Epoch: ', epoch)
    # zero the parameter gradients
    optimizer.zero_grad()

    # output of model is 2-dim
    y_pred = my_nn(X)

    # weights for regression model
    w1 = Variable(torch.randn(2, 2).type(torch.float), requires_grad=True)
    # optimizer for regression model
    optimizer1 = optim.SGD([w1], lr=0.001, momentum=0.9)
    for nest_epoch in range(1000):
        optimizer1.zero_grad()

        y_hat = torch.mm(X, w1)
        loss = criterion(y_hat, y_pred)

        # Need to add retain_graph, as it doesn't work otherwise
        # but it does work, this loss will get close to 0
        loss.backward(retain_graph=True)
        optimizer1.step()

    # this loss does not decrease
    loss = criterion(target, w1)
    print('Loss:', loss.item())
    loss.backward()
    optimizer.step()
```

My solution is to update the variable manually and creating a new graph for autograd. Works perfectly well now!
my_nn.train()
for epoch in range(300):
    # zero the parameter gradients
    optimizer.zero_grad()
    my_nn.zero_grad()

    Y = my_nn(X)
    loss_output = criterion(Y, y)
    w = torch.zeros_like(target)
    w.requires_grad = True
    for nest_epoch in range(50):
        y_w = torch.matmul(X, w)
        loss_w = criterion(y_w, Y)
        grad = torch.autograd.grad(loss_w, w, create_graph=True)
        w = w - 0.1 * grad[0]
    loss_regression = criterion(w, target)
    loss = loss_output + loss_regression
    loss.backward()
    optimizer.step()


If I do not do any data normailization, is datagen.fit required in Keras? I use keras for training an image classification problem as follows:

datagen = ImageDataGenerator(
    featurewise_center=False,
    featurewise_std_normalization=False,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True)

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)

# alternative to model.fit_generator
for e in range(epochs):
    print('Epoch', e)
    batches = 0
    for x_batch, y_batch in datagen.flow(x_train, y_train, batch_size=32):
        model.fit(x_batch, y_batch)
        batches += 1
        if batches >= len(x_train) / 32:
            # we need to break the loop by hand because
            # the generator loops indefinitely
            break

I would like to know if datagen.fit is required if I do not apply any data normalization?

According to the comment of the documentation in the code:

# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)

It does the normalization, reducing mean and dividing by standard deviation, and more things like PCA. So it seems that you don't need to do normalization. That method does that, and normalizing features is required for accelerating training process and caring about all features with different scales the same.


plot multiple strings from a dataframe I am trying to read and plot several files which looks like as below when I open with python using pandas read.table. The columns to plot are "dev" vs timestamp.

The timestamp should be compose from the columns; mm dd,time and yyyy.

    mm dd time yyyy dev st fault typ
0   Jul 5 2:48:29 2018 aaa STANDBY HRW_FAULT neg
1   Jul 5 2:48:29 2018 aaa  SOFT SWF_FAULT  ack
2   Jul 5 2:48:29 2018 aaa HARDWARE  disable
3   Jul 5 2:50:47 2018 bbb STANDBY HRW_FAULT pos

...............

df['DateTime'] = df[['Year', 'Month', 'Day', 'Hour']].apply(lambda s : datetime.datetime(*s),axis = 1)

This might pop some error depending on the fact that that whether you have hour properly defined like it should be in between 0-23 , seconds in-between 0-59 etc...

Or try this

You can pass only the columns that you need to assemble.

In [33]: pd.to_datetime(df[['year', 'month', 'day']])
Out[33]:
0   2015-02-04
1   2016-03-05
dtype: datetime64[ns

]

From the docs itself,

pd.to_datetime looks for standard designations of the datetime component in the column names, including:

required: year, month, day
optional: hour, minute, second, millisecond, microsecond, nanosecond

Also Another Approach is that Since it's probably a .csv file, you can do this at the time of reading/parsing the file itself, pandas is very intelligent and interesting!

This is done by passing in the column names as a list([year, month, day, time ]) to the parameter  parse_dates, or via  infer_datetime_format to True or by using a custom date_parser in the function pd.read_csv()>...

Hope this helps!


How do stacked CNN layers work? The internet is full of pictures like this:

But how are the second/third/etc CNN layers able to extract features when the features are already extracted by the previous layers?
For example, the mid-level feature in the picture has a nose. When we apply this "nose" filter, the output feature map will be an image without the nose, right?
Then, this feature map is passed to the next CNN layer, but how is it able to extract the "high-level feature" if the feature map given to it doesn't contain the nose? And the more layers we stack in CNN, the less meaningful data will be extracted in the latter layers.

I think you might be misunderstanding the phrase "extract" here. Think of it as "gets activated by" instead.
For example, the "nose filter" gets activated by inputs which look like a human nose (more precisely: it gets activated by activation maps of previous layers which correspond to a nose in the input image). And simply put, the following higher level feature maps in your example, which might encode a human face, then get activated if the previous layers contain activated feature maps for a human nose, mouth, eye etc. This article explains it well.
As NNs in general, this process is loosely inspired by how visual perception in humans and other animals works.


Understanding backprop for softmax I'm looking on a given solution of the first assignment of cs231n course.

Down below a snippet from the loss function.
I don't really understand lines 140-143. Can you explain why dscores (the derivative of scores) is calculated like that?

Be aware that posting code in images very annoying to copy/paste and it's bad for web reference ment.

This is due to the derivative of the softmax, but to me it's seems fishy.

If $S$ is the softmax vector, then the Jacobian $DS$ consists of $S_j(\delta_{ij}-S_i)$. This could explain the -=1 part, but not the /=N, and not the shape either.


Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights? I have classification data with far more negative instances than positive instances. I have used class weights in my models and have achieved the discrimination I want but the predicted probabilities from the models do not match the actual probabilities in the modeling data.

Is there a way to adjust the predicted probabilities from the class weighted models to match the actual probabilities in the data? I have seen equations for under-sampling (https://www3.nd.edu/~rjohns15/content/papers/ssci2015_calibrating.pdf) but they don't seem to work for class weights. I have searched online for an answer but maybe I am not using the right language?

Thank you!

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists:

Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where $p_1$ is the proportion of the positive class in the original dataset, and $r_1$ is the proportion of the positive class in the resampled dataset.

Equivalently, multiply the odds by the quantity inside the logarithm.  (Unfortunately, this doesn't lead to a clean adjustment directly to the probabilities.)

Let's do a little rewriting to see the connection to your linked paper.  $1-r_1$ is the proportion of negative classes call it $r_0$, and similarly with $p_1$.  Use capitals $R_1, \dotsc$ to denote the number (or total weight) of samples rather than the proportions, and without subscripts $P,R$ to denote total numbers (or weights) of samples before and after resampling.  So the multiplier becomes
$$\frac{p_1(1-r_1)}{(1-p_1)r_1} = \frac{p_1 r_0}{p_0 r_1} = \frac{(P_1/P) (R_0/R)}{(P_0/P) (R_1/R)} = \frac{P_1 R_0}{P_0 R_1}.$$
In the context of the linked paper, positive class samples are not resampled, so $P_1=R_1$ and the adjustment simplifies to $R_0/P_0$, which is the parameter $\beta$ used in the paper.

Finally, using their equation (4), we check the change in odds:
$$\text{new odds} = \frac{p}{1-p} = \frac{1}{\frac1p - 1} =
\frac{1}{\frac{\beta p_s−p_s+ 1}{\beta p_s} - 1} =
\frac{\beta p_s}{1-p_s} = \beta\cdot\text{old odds}. $$

So, what about weightings instead of resampling?  Well, class_weights might have different effects in different algorithms, but generally the idea is that (positive) integer values of class_weights should correspond to duplicating the samples that many times, and fractional values interpolate that.  So, it should be about the same to use the multiplicative factor above.  Using the size version rather than the proportion version, we should interpret $R_0$ and $R_1$ as the total weights of the relevant classes.

I've not been able to find a reference for this version, so I put together a short experiment; it seems to verify that this shift works.
GitHub/Colab notebook

Finally, this shift in log-odds will fail to produce properly calibrated probabilities if the classifier is poorly calibrated on the weighted data.  You could look into calibration techniques, from Platt to Beta to Isotonic.  In this case, the shift above is probably superfluous.


Is Overfitting always bad? I have a data set of total 8000 sound samples. These are the results of my multi layer neural network, binary classifier:

Precision: [0.95 0.96]
Recall: [0.96 0.95]
F-Score: [0.95 0.95]
S: [1217. 1254.]
Accuracy training 1.0
Accuracy Test 0.95

I am happy with the test accuracy, precision and recall. But the 1.0 accuracy bugs me since its overfitting to the training-set. Is this a bad thing even the test-set accuracy is satisfactory?

Here below I load the car sound features(1) and other types of sounds features(0) from disk and assign the labels. I double checked and I dont add labels as a feature while I process and extract features from audio. sets are cut to be equal to 4100 samples each

car_features =np.load('car_features_final.npy')
car_labels =np.ones(len(car_features),dtype=int)
shuffle(car_features, random_state=12)

other_features = np.load('other_features.npy')
other_features=list(islice(shuffle(other_features, random_state=12),4100))
other_labels = np.zeros(len(other_features),dtype=int)
shuffle(other_features, random_state=12)

all_features = np.append(car_features, other_features, axis=0)
all_labels = np.append(car_labels,other_labels, axis=0)

X_train, X_test, y_train, y_test = train_test_split(all_features, all_labels, test_size=0.30, random_state=42)

clf = MLPClassifier(activation='relu', solver='adam', alpha= 0.1, hidden_layer_sizes=(300, 300, 300, 100), random_state=1, max_iter=500)
clf.fit(X_train, y_train)

y_predicted_test = clf.predict(X_test)
y_predicted_train = clf.predict(X_train)
p,r,f,s = precision_recall_fscore_support(y_test, y_predicted_test)

It is overfitting if you have an accuracy on training of 100%, but the test accuracy would be 5%. That's overfitting.

In your case, there is a good match between training and testing accuracy. To check for any sampling bias, do a K-fold cross validation with the same hyper-parameters.


BERT Self-Attention layer I am trying to use the first individual BertSelfAttention layer for the BERT-base model, but the model I am loading from torch.hub seems to be different then the one used in hugginface transformers.models.bert.modeling_bert:
import torch, transformers

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
torch_model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')

inputs = tokenizer.encode_plus("Hello", "World", return_tensors='pt')

output_embedding = torch_model.embeddings(inputs['input_ids'], inputs['token_type_ids'])

output_self_attention = torch_model.encoder.layer[0].attention.self(output_embedding)[0]

# compare output with using the huggingface model directly
bert_self_attn = transformers.models.bert.modeling_bert.BertSelfAttention(torch_model.config)

# transfer all parameters
bert_self_attn.load_state_dict(torch_model.encoder.layer[0].attention.self.state_dict())
# <All keys matched successfully>

output_self_attention2 = bert_self_attn(output_embedding)[0]
output_self_attention != output_self_attention2 # tensors are not equal?

Why is output_self_attention2 different from output_self_attention? I thought they would give the same output given the same input.

The problem is that you are using the models in training mode (the default mode), and the stochastic elements like dropout are active in training mode, therefore you obtain different results, not only between different models, but also on different runs of the same model.
You should invoke model.eval(), which makes some elements like dropout or batch normalization behave in inference mode, i.e. dropout is disabled altogether, batch-normalization uses the remembered statistics.
Also, it is better to avoid gradients being computed with torch.no_grad(), as you do not need them. The same effect can be achieved with torch.set_grad_enabled(False).

import torch, transformers

torch.set_grad_enabled(False) # avoid wasting computation with gradients

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
torch_model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')

torch_model.eval() # <------ set model in inference mode

inputs = tokenizer.encode_plus("Hello", "World", return_tensors='pt')
output_embedding = torch_model.embeddings(inputs['input_ids'], inputs['token_type_ids'])
output_self_attention = torch_model.encoder.layer[0].attention.self(output_embedding)[0]

# compare output with using the huggingface model directly
bert_self_attn = transformers.models.bert.modeling_bert.BertSelfAttention(torch_model.config)
bert_self_attn.load_state_dict(torch_model.encoder.layer[0].attention.self.state_dict())

bert_self_attn.eval()  # <------ set model in inference mode

output_self_attention2 = bert_self_attn(output_embedding)[0]
output_self_attention != output_self_attention2 # tensors are equal


ElasticNet Convergence odd behavior I am optimizing a model using ElasticNet, but am getting some odd behavior. When I set the tolerance hyperparameter with a small value, I get
"ConvergenceWarning: Objective did not converge"

errors. So I tried a larger tolerance value, and the convergence error goes away, but now the test data consistently gives a higher root mean squared error value. This seems backwards to me, if the model does not converge, what can cause it to give a better RMSE score, or even give consistent scores? I am using this inside a gridsearch function:
No error, bad score:
GridSearchCV(cv=KFold(n_splits=4, random_state=None, shuffle=True),
             estimator=ElasticNet(),
             param_grid={'alpha': [0.01], 'fit_intercept': [True],
                         'l1_ratio': [0.75, 0.8, 0.85, 0.9],
                         'max_iter': range(350, 450, 50), 'normalize': [False],
                         'random_state': [3], 'selection': ['random'],
                         'tol': [0.1]},
             refit='nmse', return_train_score=True,
             scoring={'mae': 'neg_mean_absolute_error',
                      'nmse': 'neg_mean_squared_error',
                      'nmsle': 'neg_mean_squared_log_error', 'r2': 'r2'})

Running GridSearchCV:
C:\miniconda3\envs\tf\lib\site-packages\sklearn\linear_model\_coordinate_descent.py:529: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 1253.5072980364057, tolerance: 848.4484859258542
  positive)

Error, good score:
GridSearchCV(cv=KFold(n_splits=4, random_state=None, shuffle=True),
             estimator=ElasticNet(),
             param_grid={'alpha': [0.01], 'fit_intercept': [True],
                         'l1_ratio': [0.75, 0.8, 0.85, 0.9],
                         'max_iter': range(350, 450, 50), 'normalize': [False],
                         'random_state': [3], 'selection': ['random'],
                         'tol': [0.01]},
             refit='nmse', return_train_score=True,
             scoring={'mae': 'neg_mean_absolute_error',
                      'nmse': 'neg_mean_squared_error',
                      'nmsle': 'neg_mean_squared_log_error', 'r2': 'r2'})

Increasing tolerance will result in "higher root mean squared error value" most of the time. Increasing tolerance is telling the model it is okay to stop earlier with higher error and not continue the search for a possible more optimal solution using smaller updates.


Decision Tree Optimize Deviation From Objective I have the following problem: I have three classes/modes, let's call them car, bike, and walking. For any given test data instance with some environmental variables such as distance, road quality etc, I would like to predict the cheapest mode. In all instances in the training and test data set, each of the modes is associated with a cost. I would like to use a decision tree due to it being easy-to-understand. Is it possible to change the "performance metric" during training such that the final tree minimizes the cost deviation due to wrongful classification rather than maximizing the accuracy?

I know that the goal of minimizing the cost deviation can be achieved by employing three (linear) regression models, and then choosing the cheapest mode, but I would like to keep the "easy-to-understand" property of decision trees. Also, some of the explaining variables are non-linear (for example, walking is preferred, if it is neither too hot nor too cold).

Is this possible in general?

It is possible, is not an easy task but it's possible.

Custom Loss Function

  Gradient boosting is widely used in industry and has won many Kaggle competitions. The internet already has many good explanations of gradient boosting (we’ve even shared some selected links in the references), but we’ve noticed a lack of information about custom loss functions: the why, when, and how. This post is our attempt to summarize the importance of custom loss functions in many real-world problems — and how to implement them with the LightGBM gradient boosting package.


Time of trainig vs time of prediction, which one is used during classification algorithms comparison? I need to use many algorithms for making a binary classification, such as Logistic regression, SVM, XGBoost, CatBoost, ... I get an interesting improvement but All of those algorithms (except LR) take a very long time for training data.

So I need to know if I must indicate training time in my comparison or it is doesn't need to be indicated and predicting time is enough ??

And how many time tuning parameters and fitting training should be done in a real application? does it done each time we insert a new row on data?

I get this answer, if it may helping you or if you have any other information to add :

If we are talking only about computational time then prediction time is more important if the training time is finite and accuracy is comparable. Training time can be handled easily by using a good machine.

This is important because any machine learning system is expected to generalize or at least have higher accuracy on the testing set. To achieve this better performance training time is often not considered. If one algorithm has significantly higher training time but good accuracy on testing test then it is considered better because once the training is done, the algorithm doesn’t need much tweaking. Obviously in this case we assume that the training time is finite which can be reduced by using a better machine.


Needed: Java library to calculate text readability/complexity In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall readability score, linsear write formula, gunning fog etc).

I guess there must be plenty of libs but I just cant find them ...

I looked for this as well and found only the unmaintained RAT library.
But basically all of these algorithms (most are described as formulas in this paper) can be directly translated from the single Python file with no extra dependencies.
One exception was the Linsear Write formula because of its needing a syllable counting method. But if you use the CMU pronunciation dictionary or equivalent, you can easily build the countSyllables method and the rest of the algorithm is trivial.


In text classification, how can I use a neural network on word embeddings? I have read some papers on text classification but they are pretty abstract. I fail at understanding how to train a multi layer perceptron with data made of sentences -> label.

A perceptron takes a vector input of size m and outputs a vector.

As a first step I reduced the dimensions and I map each word to a vector of n features, where n is a constant.

Now how do I mix the words in a sentence to make an input of size m ? Each word maps to a vector of size n but then what ? Knowing that every sentence has a different number of words.

Thanks

Okay. These are the steps to follow.

Pad your sentences to a fixed length. Use the maximum length. If you are comfortable with creating buckets, you can pad the sentences to maximum length for individual bucket. To pad, add  PAD tokens to each sentence till they become the size of the fixed sequence length you want to use.
Extract the unique words out of the padded corpus and assign a numeric id to every word. You can use there index in the unique list as IDs. Filtering out rare words and replacing them with unknown token will prove useful.
Now if you had a sentence like "I am John", and your maximum sequence length was 5 the sentence will become, "I am John PAD PAD." Once you extract the unique words, you can represent the sentence as IDs like [1,2,3,4,4] Now if you had 100 sentences you will have a matrix of size 100 X 5.

Now you can go in two ways, represent each word as a one hot vector i.e.a vectors of zeros with a one at the index it represents or a simpler way would be to train a word embedding.

For details look into this RNN tutorial by Denny Britz, this is from scratch.

If you are of the deep learning persuasaion then check out Denny Britz's CNN text classification tutorial where he uses tensorflow and trains his own word embedding. The first blog will give you all the information you need on how to prepare your text dataset.


Why does adding random pixels stop my model learning in cnn? I am using a very simple model to classify a 224x224 RGB image.

For a test, I have labelled my images (2 labels "Green" or "Red", 2,000 images of each) based on colour of a single fixed pixel from upper part of image.

My simple model achieves very high accuracy and very low loss, until I add more "random" pixels to lower part of image.

Why is my model getting confused?

The model easily copes with:

But, when I add more pixels in lower part of image - it achieves only 50% accuracy and high loss:

These project is a bit facile but I am researching a simple image structure I can control.

def VerySimpleNet():
    model = keras.Sequential([
        keras.layers.Conv2D(16, kernel_size=3, activation=tf.nn.relu, input_shape=(224, 224, 3)),
        keras.layers.Dropout(0.4),
        keras.layers.Flatten(),
        keras.layers.Dense(3, activation=tf.nn.softmax)
    ])
return model

  EDIT 1

Is it possible to add a bonus to Data Science questions to encourage more answers?

  EDIT 2

I would really be interested if anybody was keen to have a more interactive chat(email,txt?) with me on this question.

I think the problem here is that you are trying to relate what you know about the input to what the model is trying to predict. I suggest trying to interpret why your model is giving certain outputs for images.

I came across SHAP a while back when reading about model interpretation. The image plot of SHAP should be able to narrow in on why your model is interpreting a label for a specific Image.

This should lead you to figure out why your model is making the prediction.


Why feature crosses may work better than having them as individual features? On Google ML Crash Course it is said the following:

  If we build a feature cross from both these features:

  [behavior type X time of day]

  then we'll end up with vastly more
  predictive ability than either feature on its own. For example, if a
  dog cries (happily) at 5:00 pm when the owner returns from work will
  likely be a great positive predictor of owner satisfaction. Crying
  (miserably, perhaps) at 3:00 am when the owner was sleeping soundly
  will likely be a strong negative predictor of owner satisfaction.

Why cross features is better than having them as individual features?

It is not necessarily cross features. Actually, it is a subset of a general concept, namely, kernel trick. Using kernel trick you will be able to find decision boundaries which are more complex than usual hyperplanes which can be found using the linear combination of current feature space. Suppose you have a single feature $x$. By employing that you will be able to make a line as a separator of two classes. If you add $x^2$ alongside $x$ you will be able to have a parabola decision boundary rather than a simple line, which enables you to have a complex decision boundary in cases which is needed. Moreover, because you have both $x$ and $x^2$ you can set the coefficient of each to zero which means you have a kind of flexibility to decide to have a simple line or a parabola.


How to handle continuous values and a binary target? This is going to be a very beginner's question.

I have a datset of continues features like LoanAmount, LoanDuration(multiclass?), ... ClientIncome, ClientFreeSources, etc. and a binary target whether a contract was sued or not.

I'm not sure how to approach the problem as I'm fairly new to DS.
I can reformulate the target into limited number of values like 5, 11, 18, ... which I think would be probably precieved as multiclass by a model or into continues value expressed in DebtAmount after some work at the SQL Server end.

However, the simple flag Sued 1/0 would be prefered option at the begining if possible.
I also wonder how to treat high-cardninality categorical features like post code because the number of new dummy variables which seems to me too high.

Thanks for your answers in advance.

I am expecting this question is for a supervised learning problem where you have a mixture of continuous and categorical features as independent features and your dependent or target feature is a binary class feature. If that is so please follow the below advice.

There might be a much better solutions but this worked for me. I had more than 200 features and a categorical feature like pin code.

Target Feature as 1/0

I am assuming this is a supervised learning problem and the best way to go about will be by starting with a basic logisitic regression model.
If you are not impressed by the performance then you can even try some advanced classification algorithms like Naive Bayes, KNN, SVM etc.

Categorical Feature

You can use pandas one hot encoding for this

I kept a threshold, and used one hot encoding only on those unique categories which occurred more than this threshold
The ones which were less than the threshold can be dropped, but I used a special value to encode them just in case to not to loose them.
You can follow here to know how to perform one-hot encoding

I hope I answered at least to a certain extent. Again, I used this solution for my issue and it worked pretty well and hope it works for you too.


Levene test for equal variance I would like to run one-way ANOVA test on my data. I saw that one of several assumptions for one-way ANOVA  is that there needs to be homogeneity of variances. I have run the test for different data-sets. I find sometimes my p-values are larger than 0.05 and for some datasets it is smaller.
As I understand, if the p-value is smaller than 0.05, then I can reject the null hypothesis and tell that the variances are not equal (and then it makes sense to use ANOVA?)
The problem is that I see in different sources diferent conclusions: in some they run ANOVA if  they reject the null hypothesis, for some they run ANOVA if they cannot reject the null hypotheis. For example,
here it says that if is smaller we can run the ANOVA:

But here it recommended that p-value>0.05
To me, intuitively, it sounds like p-value should be greater than 0.05 but I would like to get clear answer regarding  that.

"As I understand, if the p-value is smaller than 0.05, then I can reject the null hypothesis and tell that the variances are not equal (and then it makes sense to use ANOVA."
Your understanding is incorrect. As a measure of goodness of fitness of data, null hypothesis has to be accepted if alpha is less than  or = .05. If you want to perform statistical test of variability across groups, you need  an implementation of Chi-square test of variability across groups.
Your intuition (question in last part of body) -p-value should be greater than 0.05  is not logical. This value indicates sampling fluctuations or error arising on account of chance in sample selection. An (sampling ) error greater than 5% is not acceptable by any standards.


How to run a model stored as a .Rda file? I am using an already written R code which has a line of code as shown below

model_predictors <- buildModel(flag, data, outcomeName, folder)
model<-model_predictors$model
predictorsNames<-model_predictors$predictorsNames
auc <- model_predictors$auc
save(model, file=paste(folder,studyName,'_model_', flag$model[1], '_', outcomeName,".Rda",sep=''))

As you can see in the last line of code, the (training) model is saved in folder in a .Rda file format with a naming convention _model_. Now, I have to use this model to test/evaluate on unseen dataset.

So now my question, I see a .Rda file in my folder path and I can load the .Rda file in Rstudio but how do I make inference from this model? I am new to R and can anyone help me with this please?

Can someone help me understand how to run this model please?

I suppose you come from Python. R is quite different as it does not require to save to external files your model, you just need to have it in your workspace. Once you have a model object, you just have to use the predict function and specify the unseen dataset. Here's an example of a tree model:

model<-tree(Y~X1+X2, data=dataset) #estimate of the tree model
predictions<-predict(model, newdata=unseen_dataset) #prediction on new data

Be sure variable names between datasets correspond.


how to predict content based demand this is my first post at ds StackExchange, so please be gentle and let me know if something is not clear :)

I have many products (>1M), and I save all the products purchases in a DB with a time stamp. ("purchases data")
each product has 'content features' (e.g product size, product safety rank etc.)

The "purchases data" looks like this:

  | time stamp | product id | content features 1 | ... | content features N |

Where each row is a purchase of a product with id at time stamp.

My main target is to get tomorrow most wanted products,
I translate the problem into predicting demand for the next day, or classify each product id and day as high demanded or low demanded),

I struggle with two main problems with these settings:

Generating demand data: I want to convert the "purchases data" into demand for a day ("demand data")- meaning that I group the data by product id and day,
Then I count the number of rows and save it as 'freq' (and also remove row duplication).
The problem is that the minimum product frequency per day would be 1 and not 0

  For example: if product #1 was purchase at Sunday 3 times and at Wednesday 2 times, the purchases and demand data would be:

"purchases data" fi(product id) is mapping to content feature i:

   | time stamp      | product id | content f 1 | ... | content f N |
   | Sunday  05:20   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  08:11   | 1          | f1(1)       | ... | fn(1)       |
   | Sunday  10:25   | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 08:10 | 1          | f1(1)       | ... | fn(1)       |
   | Wednesday 16:20 | 1          | f1(1)       | ... | fn(1)       |

"demand data":

   | day             | product id | content f 1 | ... | content f N | freq |
   | Sunday          | 1          | f1(1)       | ... | fn(1)       | 3    |
   | Wednesday       | 1          | f1(1)       | ... | fn(1)       | 2    |

    But if product #1 was not purchase at Monday there wouldn't be any row
    since there is no purchase
    data for this item at that timestamp.

Since there are over 1M products, I want to avoid creating rows with 0 frequency,

Is there a way to create (or to design) demand data from purchases data for a huge amount of items (products) without using 0 demand rows?

Content-based time series: After creating "demand data", I want to use it as a time series.
My problem is that I would need to split the data into over 1M series, one for each product/item id, and these series would also be very sparse...
I want to find a way to use the "content features" as input with the time series and have the model learn to use some kind of averaging of time series based on related content features.

What is the best way to model content based series time prediction?

Welcome to the site. I would encourage you to think about your problem in a different way. You are focused on "what sold today" whereas you should be focused on "who bought what over a historical timeline".

What you're looking for is known as a recommender system and there are (generally speaking) two types:

Content-based - what should you recommend based on attributes of products. The algorithm is basically saying, "You bought breakfast cereal, here are other products that might go with your cereal . . ."
Community-based - what should you recommend based on attributes of people who bought products. The algorithm is basically saying, "You are a female, under 30, with no kids. Other females, under 30 with no kids also liked these products . . ."

I will assume that you don't have info on your customers so let's focus on the content-based recommenders. You are on the right track by thinking about the attributes of products, but you should be thinking about them both (1) over a longer timeline than just yesterday and (2) how the products and their attributes relate to each other. The people who need attribute X might also need attribute Y and that is (most likely) across multiple products and will generate higher demand for those products.

Start researching content-based recommender systems in your language/tool of choice and you will end up with the desired algorithm. From there you can also think about user data collection and then move into a community-based recommender over the long term.


Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy) Which is better for accuracy or are they the same?
Of course, if you use categorical_crossentropy you use one hot encoding, and if you use sparse_categorical_crossentropy you encode as normal integers.
Additionally, when is one better than the other?

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]).

Formula for categorical crossentropy (S - samples, C - classess, $s \in c $ - sample belongs to class c) is:

$$ -\frac{1}{N} \sum_{s\in S} \sum_{c \in C} 1_{s\in c} log {p(s \in c)} $$

For case when classes are exclusive, you don't need to sum over them - for each sample only non-zero value is just $-log p(s \in c)$ for true class c.

This allows to conserve time and memory. Consider case of 10000 classes when they are mutually exclusive - just 1 log instead of summing up 10000 for each sample, just one integer instead of 10000 floats.

Formula is the same in both cases, so no impact on accuracy should be there.


GradientChecking, can I blame float precision? I am trying to GradientCheck my c++ LSTM. The structure is as follows:

  output vector (5D)

  Dense Layer with softmax (5 neurons)

  LSTM layer (5 neurons)

  input vector (5D)

My gradient check uses this formula

$$d = \frac{\vert \vert (g-n) \vert \vert _2 }{
\vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$

The Dense Layer returns descrepancy = 2.843e-05  with an epsilon of 5e-3

With 5e-5 or lower the network doesn't see any changes in Cost at all, anything greater than 5e-2 results in low precision.

This is only for top layer, and I am using 'float' 32 bit values for everything.

For the LSTM layer, I am using 5e-3 as well, but descrepancy is 0.95

The LSTM seems to converge fairly quickly on any task

0.95 bothered me, so I manually compared NumericalGradient array against the BackpropGradient array, side by side. All of the gradients match sign, and only differ in size of each entry.

For example:

numerical:    backpropped:

-0.015223     -0.000385
 0.000000      0.000000
-0.058794     -0.001509
-0.000381     -9.238e-06
 9.537e-05     2.473e-06
 0.000215      6.266e-0.6
-0.015223     -0.000385
 ...

As you can see, the signs do indeed match, and the numerical gradient is always larger than the back propped gradient

Would you say it is acceptable and I can blame float precision?

Edit:

Somewhat solved, - I simply forgot to turn-off "average the initial gradients by the number of timesteps" during backprop of my LSTM.

That's why my gradients were always smaller in the "backpropped" column.

I am now getting descrepancy of 0.00025 for LSTM

Edit: setting epsilon to 0.02 (lol) seems like a sweet-spot, as it results in descrepancy of 6.5e-05. Anything larger or smaller makes it deviate from 6.5e-05, so it seems like a numerical issue ...Only 2 layers deep though, weird af

Someone had this precision before?

The fact that there is a sweet spot is a common issue in numerical differentiation. The issue is that using a high epsilon value will give you high error due to the fact that the numerical derivatives have error $O(\epsilon)$ (or $O(\epsilon^2)$ if you use centered difference), and using a very low epsilon will result in cancelling errors (see catastrophic cancellation question) due to the fact that you are substracting two numbers that are very close. It seems that $\epsilon = 0.01$ is a fair trade-off between these two extreme situations in your case.


ReLU vs sigmoid in mnist example PLEASE NOTE: I am not trying to improve on the following example. I know you can get over 99% accuracy. The whole code is in the question. When I tried this simple code I get around 95% accuracy, if I simply change the activation function from sigmoid to relu, it drops to less than 50%. Is there a theoretical reason why this happens?

I have found the following example online:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)

Y_train = np_utils.to_categorical(Y_train, classes)
Y_test = np_utils.to_categorical(Y_test, classes)

batch_size = 100
epochs = 15

model = Sequential()
model.add(Dense(100, input_dim=784))
model.add(Activation('sigmoid'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')

model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=1)
print('Test accuracy:', score[1])

This gives about 95% accuracy, but if I change the sigmoid with the ReLU, I get less than 50% accuracy. Why is that?

I took your exact code, replaced
model.add(Activation('sigmoid'))

by
model.add(Activation('relu'))

and indeed I experienced the same problem than you: only 55% accuracy, which is bad...
Solution: I rescaled the input image values from [0, 255] to [0,1] and it worked: 93% accuracy with ReLU! (inspired from here):
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.utils import np_utils

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255

Y_train = np_utils.to_categorical(Y_train, 10)
Y_test = np_utils.to_categorical(Y_test, 10)

batch_size = 100
epochs = 15

model = Sequential()
model.add(Dense(100, input_dim=784))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', metrics=['accuracy'], optimizer='sgd')

model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1)

score = model.evaluate(X_test, Y_test, verbose=1)
print('Test accuracy:', score[1])

Output:

Test accuracy: 0.934

Potential explanation: when using an input in [0, 255], then when doing the weighted sum for the layer $L$: $z = a^{(L-1)} w^{(L)} + b^{(L)}$, the value $z$ will often be big too. If $z$ is often big (or even if it's often > 0), let's say around 100, than $ReLU(z) = z$, and we totally lose the "non-linear" aspect of this activation function! Said in another way: if the input is in [0, 255], then $z$ is often far from 0, and we totally avoid the place where "interesting non-linear things" are going on (around 0 the ReLU function is non linear and looks like __/)... Now when the input is in [0,1], then the weighted sum $z$ can often be close to 0: maybe it sometimes goes below 0 (since the weights are randomly-initialized on [-1, 1], it's possible!), sometimes higher than 0, etc. Then more neuron activation/deactivation is happening... This could be a potential explanation of why it works better with input in [0, 1].


use the same gradient to maximize one part of the model and minimize another part of the same model I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two optimizers responsible for two part of the network/model and one of the optimizers has a negative learning rate. But it seems that PyTorch does not allow negative learning rate.
In this case what I am doing is:
loss.backward()
optimzer_for_one_part of the model.step()

and then
(-loss).backward()

Problem is, This time the again calculated gradient will not be the same(values are different but flopped of course) because some weights of the same network (same computation graph) have already been changed. But, Ideally, I want to use the flipped version of the previous gradient.
How can I achieve this?

The trick you are looking for is called the Gradient Reversal Layer. It is a layer that does nothing (i.e., identity) in the forward pass, but it reverts the sign of the gradient, so everything behind the layer optimizes the opposite of the loss function.
There are several PyTorch implementations:

https://github.com/janfreyberg/pytorch-revgrad
https://github.com/jvanvugt/pytorch-domain-adaptation/blob/master/utils.py

Initially, it was introduced for unsupervised domain dataptaion. Now it has quite a lot of applications, such as removing sensitive information from CV representation or removing language identity from multilingual contextual embeddings.


How can I download data from online table quickly? I am trying to do download all the marathon results from 1996 to 2016 from the following link. There are more than 400000 entries, but I can only show 1000 results per page.

Is there any way that can let me download all of them quickly?

Thanks in advance!

http://chicago-history.r.mikatiming.de/2015/

There doesn't appear to be an easy way - there is no mention of an API to get the data programmatically.

You could try contacting them via Twitter or something. Perhaps they would be happy to give you a dump of the data.

Otherwise, you might have to go down the route of writing a scraper e.g. using Selenium and BeautifulSoup. That would allow you to:

execute the search
download the file
navigate to the next search results page
repeat from step 2. until finished

They might be fairly easy, but do respect the websites wishes - perhaps they have a file called robots.txt which will tell you if they mind you doing that or not. In any case, build some time.sleep()s into your code, as not to bombard their server!