Which is better for Data Science, a double major in Math & CS or Physics & CS? My dream occupation is to work as a Data Scientist.

I'm soon going to be completing my first semester as a physics student, as part of an incredibly prestigious physics program. We get to run an experiment at CERN in our third year! I'm currently ranked highly in this program (top 25%), and I could push myself to top 10% if needed.

Thing is, I'm not really interested in working as a physicist.

Should I switch to Math & CS? (I'd be in a mediocre program.)

Note - I'm currently in both physics and CS, not just physics. I'm asking if I should switch the physics major with a math one.

My background is Physics undergrad, Physics grad, now in industry.  Some advice off the top of my head:

If you can push top 10%, do it (always do your best).
Don't be surprised when jobs in industry don't care about you ranking/gpa (some do, some don't.). Rank/gpa is  important for grad school however, and grad school is important for careers in data science
Major in what makes you happy.
If you stay in physics, learn to code and take some stats.
Physics will always impress people and there are many machine learning/ data science opportunities in physics.
Physics will teach you how and where to apply machine learning rather than being given perfect datasets to learn with.


How can a neural network get unstuck from a local minimum? Since, there are so many local minimums in so complex neural network function, it is common for a neural network to get stuck on a local minimum. How will the neural network get unstuck from that local minimum.

In various ways suchs as

momentum: think of momentum as a means of dampening oscillations and speeding up the iterations, leading to faster convergence. This means replacing gradients with a leaky average over past gradients.
sparse features and preconditioning (Adagrad): decrease the learning rate dynamically on a per-coordinate basis. This means, use the magnitude of the gradient as a means of adjusting how quickly progress is achieved - coordinates with large gradients are compensated with a smaller learning rate.
RMSProp: a combination of momentul and adagrad, combining leaky averages and coefficient-wise preconditioner
Adadelta: The learning rate cannot be parameterised but rather adapts itself given rate of change in the model parameters
Adam: a great algorithm that summarises features from all of the above
Scheduling: decreasing the learning rate during training

sources: https://distill.pub/2017/momentum/, https://d2l.ai/chapter_optimization


In the context of Deep Learning, what is training warmup steps I found this term "training warmup steps" in some of the papers, what exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect?

As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence.

The idea that this helps your network to slowly adapt to the data intuitively makes sense. However, theoretically, the main reason for warmup steps is to allow adaptive optimisers (e.g. Adam, RMSProp, ...) to compute correct statistics of the gradients. Therefore, a warmup period makes little sense when training with plain SGD.

E.g. RMSProp computes a moving average of the squared gradients to get an estimate of the variance in the gradients for each parameter. For the first update, the estimated variance is just the square root of the sum of the squared gradients for the first batch. Since, in general, this will not be a good estimate, your first update could push your network in a wrong direction. To avoid this problem, you give the optimiser a few steps to estimate the variance while making as little changes as possible (low learning rate) and only when the estimate is reasonable, you use the actual (high) learning rate.


Macro and micro average for imbalanced binary classes Micro and macro averaging are metrics for multi-class classification. However, for binary classification when data are imbalanced, it seems that micro and macro precision have different results. My question is that: does it make sense to use micro and macro precision in binary classification problems when classes are imbalanced?

does it make sense to use micro and macro precision in binary classification problems when classes are imbalanced?

In general micro- and macro-average performance are not relevant in binary classification, whether the classes are balanced or not. Their value can be especially misleading if there is a strong imbalance, because it takes into account both the minority class (harder for the classifier) and the majority class (easier):

By definition micro-average gives more weight to the majority class, so the micro-average performance can be high even if the classifier does a terrible job at distinguishing the two classes.
The macro-average is not biased towards any of the two classes, still it's uselessly complex, it makes it harder to understand what's going on than simple performance on the positive class, which is normally the minority one (because that's the challenging one).

Of course there can be cases where it makes sense not to follow this standard evaluation setting, it's always a matter of choosing the appropriate way to evaluate a particular task.
The below example illustrates why micro- and macro-average are confusing in a standard case of imbalance:
              true A   true B
predicted A     90        9
predicted B      0        1

For A: precision = 0.91, recall = 1, f1-score = 0.95
For B: precision = 1, recall = 0.1, f1-score = 0.18
micro-average: precision = 0.91, recall = 0.91, f1-score = 0.91
macro-average: precision = 0.95, recall = 0.55, f1-score = 0.70

Assuming we don't know anything else than the selected performance measure, this classifier:

performs almost perfectly according to the performance of the majority class A,
performs very well according to micro-average,
performs decently according to macro-average,
performs terribly according to the performance of the minority class B.

Looking at the confusion table, it's clear that the classifier doesn't do a good job at distinguishing the two classes. So the most "honest" performance measure is the last one, i.e. the non-averaged performance on the minority class.


How to measure deviance resulting from different random seeds in machine learning? I'm running an xgboost model to predict probabilities to a binary classification problem. Then I aggregate the results based on the Age variable (what is the aggregated risk of getting the sickness for Age x). I made a mistake and did not set the seed number to be constant so when I rerun the model I get slightly different aggregated results. Could you give me a reference why it is not substantial? So I can avoid running the model on several seed settings to get a confidence interval? I would not want to do it because the learning process takes a few hours. Thank you in advance!

Unfortunately it actually can be substantial. This nice article goes into depth about it and this question shows some clear impact as well.
So depending on the model performance, the model / algorithm used and especially the distribution of your data set you can expect the random seed to influence your results almost as much as optimizing any other parameter (~2-4% points in both examples).
You can minimize this by reducing imbalance in your data sets (e.g. resampling train and test data would have similar effects) or simply fixing your random seed and never touching it.
Now that you are where you are there is not a lot you can do. Maybe pointing at the two sources I have will let you get away with a fixed "interval" or maybe you simply use your last result.


How to calculate the mini-batch memory impact when training deep learning models? I'm trying to calculate the amount of memory needed by a GPU to train my model based on this notes from Andrej Karphaty.
My network has 532,752 activations and 19,072,984 parameters (weights and biases). These are all 32 bit floats values, so each takes 4 bytes in memory.
My input image is 180x50x1 (width x height x depth) = 9,000 float 32 values. I don't use image augmentation, so I think the miscellaneous memory would be only related to the mini-batch size. I'm using a mini-batch size of 128 images.
Based on Andrej's recommendation, I get the following memory sizes:
Activations: 532,752*4/(1024^2) = 2.03 MB
Parameters: 19,072,984*4/(1024^2) * 3 = 218.27 MB
Miscellaneous: 1289,0004/(1024^2) = 4.39 MB
So the total memory to train this network would be 224,69 MB.
I'm using TensorFlow and I think I'm missing something. I haven't run the training yet, but I'm pretty sure (based on past experiences) that the memory in use will be much higher than what I've calculated.
If for each image in the mini-batch, TensorFlow keeps their gradients so it can normalize them later for a single weights/biases updates step, then I think the memory should take into account another 532,752 * 128 values (gradients for each image in the mini-batch). If that is the case, then I'd need more 260.13 MB to train this model with 128 images/mini-batch.
Can you help me understand the memory considerations for training my deep learning model? Are the above considerations right?

@StatsSorceress TL;DR:

I'm going through this activity to see if I can calculate the memory required myself:

Activations: 532,752 * 2 * 4 / (1024^2) = 4.06 MB

Parameters: 19,072,984 * 4 / (1024^2) * 3 = 218.27 MB

Miscellaneous: 128 * 9,000 * 4 / (1024^2) = 4.39 MB

Total Memory: (4.06 * 128) + 218.27 + 4.39 = 742.34 MB

(Someone please correct me on this if I'm wrong. FYI, you already multiplied miscellaneous by 128, so that's why I didn't multiply it by 128 above)

I would point you to this article and the corresponding video. They helped me to understand what is going on a lot better.

NOTE: The memory required to use a network for predictions is far less than that required for training for two reasons:

When predicting, we only send an image forward through the network and not backward (so we don't multiply memory X 3; see below)
There's one prediction per image (so we don't need to multiply the memory required for one image by a batch size because we don't use batches in prediction).

Process (Memory to Train)

Calculate the memory required to train on one image
Multiply this number by the number of images in your batch

(REMEMBER: Mini-batching says we take a subset of our data, compute the gradients and errors for each image in the subset, then average these and step forward in the direction of the average. For convnets, weights and biases are shared, but the number of activations is mutliplied by the number of images in the batch.).

STEP 1: Memory for 1 Image

To train one image, you must reserve memory for:

Model Parameters:

The weights and biases at each layer, their gradients, and their momentum variables (if Adam, Adagrad, RMSProp, etc., optimizers are used)

To approximate the memory for this, calculate the memory required to store the weights and biases and multiply that by 3 (i.e. "by 3" because we're saying the amount of memory needed to store the weights and biases is (roughly) equal to that needed for the gradients and for the momentum variables)

EQUATIONS:

Convolutions:

weights(n) = depth(n) * (kernel_width * kernel_height) * depth(n-1)

biases(n) = depth(n)

Fully Connected (Dense) Layers:

weights(n) = outputs(n) * inputs(n)

biases(n) = outputs(n)

where n is the current layer and n-1 is the previous layer, and outputs are the number of outputs from the FC layer and inputs are the number of inputs to the FC layer (if the previous layer is not a fully-connected layer, the number of inputs is equal to the size of that layer flattened).

NOTE: The memory for the weights and biases alone, plus the memory for the activations for one image (see below), is the total amount of memory you need for predictions (excluding some overhead for memory for convolutions and some other things).

Activations (these are "Blobs" in Caffe):

(I'm using terms loosely here, bear with me)

Each convolution in a convolution layer produces "number of pixels in image" activations (i.e. you pass an image through a single convolution, you get a single feature map consisting of "m" activations, where "m" is the number of pixels from your image/input).

For fully-connected layers, the number of activations you produce is equal to the size of your output.

Convolutions:

activations(n) = image_width * image_height * image_num_channels

Fully Connected (Dense) Layers:

activations(n) = outputs(n)

Note that your input is really only an image at the beginning of the network. After convolutions, it turns into something else (feature maps). So really replace "image_width", "image_height", and "image_num_channels" with "input_width", "input_height", and "layer_depth" to be more precise. (It's just easier for me to think of this concept in terms of images.)

Since we also need to store the error for the activations at each layer (used in the backward pass), we multiply the number of activations by 2 to get the total number of entities we need to make room for in our storage space. The number of activations increases with the number of images in the batch, so you multiply this number by the batch size.

STEP 2: Memory to Train Batch

Sum the number of weights and biases (times 3) and the number of activations (times 2 times the batch size). Multiply this by 4, and you get the number of bytes required to train the batch. You can divide by 1024^2 to get the answer in GB.


Integer programming formulation: which algorithms I have a complex problem that I have simplified coming to a simple integer linear programming formulation.

Given the scalar $K > 0$, the vectors $v(t) \in R^n$ and $b_i \in R^n, \forall i=1,\ldots,K$ are known. In particular $v(t)$  is the measured value (ex: every 10 minutes) while the $b_i$ are fixed (computed ones, previously).
Consider that I have: $$K >> n,$$
i.e., I have to identify the presence of $K$ (ex: $K=20$) elements, each one of dimension $n$ (ex: $n=2$).

So, for every fixed $t$, the problem that I have to solve has the following formulation:

$$ \newcommand\norm[1]{\left\lVert#1\right\rVert}
\begin{equation*}\begin{aligned}
& \underset{x_i}{\text{minimize}}
& & \norm{v - \sum_{i=1}^{K} b_i x_i}_{2} \\
& \text{subject to}
& & x_i \in \{0,1\},\ \forall i = 1, \ldots, K.
\end{aligned} \end{equation*}$$

Do you know which algorithms solve that formulation? In particular I am looking for an algorithm implemented in Python.

You may reparameterize $x_i$ as $x_i = sigmoid(\alpha \cdot z_i), z_i \in \mathbb{R}$, with $\alpha$ being a constant such that $\alpha \gg 1$, and then proceed with your favorite optimization method for finding $z_i$, and from there $x_i$.

This, of course, may present problems, as $x_i$ are no longer bounded in $\{0, 1\}$, but may suffice your use case.


Churn Prediction Model based on Customer Activities I am new to data science, so forgive me if i have not done my research well.

I want to build a system that calculates the churn scores for each customer and hence try to prevent it. I just want to test the concept, so result is not really important.

The business was about sending newsletter to customer and receive ad revenue for it.

The data
We have customer activity like open emails, read emails, click emails, and that is about it.

The idea is to look into the customer activity and see if we can identify if the customer is about to cancel the subscription.

I have look into several classification models and wonder if there is a model that can look into the activities history, not just the current events.

Any help will be appreciated!

First of all, you need to see your churn rate to see how rare it is. If your churn rate is really low ( below < 5% of total labels ), then you could treat this as an anomaly detection system, meaning you train your model on non-churn users ( if churn is the label you're predicting then the non-churns would have 0 as label ), and flag any activity different from what you trained your model on as an anomaly ( in case you don't have enough examples of activities that led to churn ), or if you have enough 'churners' to train your model on, treat this as a classification problem. I'll suggest some indicators for you even though i haven't worked on a churn model before.

Assuming you only have email activity ( following your statement ) you could :

create a feature count on the email activity to see if there's a pattern to churn after certain numbers of non-reads,non-clicks,non-opens.
try to collect more features if there are more because 3 is a low feature count for a predictive model ( maybe not only the activity but the user's personal information like age / type of profession etc.. )
you can also see this article discussing user churn.

Hope this helps you.


users' percentile similarity measure Having n vectors of percentile ranks for a list of common users between group #1 and groups #2:n

e.g. vec1 = {0.25, 0.1, 0.8, 0.75, 0.5, 0.6}
     vec2 = {0.35, 0.2, 0.6, 0.45, 0.2, 0.9}

The percentile ranks represent activity frequency within the group for instance, opening times.
The goal is to find similarity between group #1 and groups #2..n in terms of these common users's rank.

The direction taken so far is to use dot product in order to take the magnitude into account (due to rank). The problem is that the scalar answer can take any value and so a threshold cannot be drawn.

Do I need to draw a threshold with respect to the dot product of vec #1 (group 1) with itself? or is there another way to set a threshold (maybe dynamic) ?

As I understand it, you don't want to use the dot product, because it is unbounded.  Yet you also don't want to use the cosine similarity metric:

, which is just the dot product normalized by the length of the two vectors being dotted because it does not take magnitude into account e.g. $(1,1)\cdot(1,1)=1=(1,1)\cdot(2,2)$

Perhaps you could therefor use the sorensen dice coefficient or possibly 1-soresen:

This varies from 0 to 1 and accounts for magnitudes of the constituent vectors e.g.:

$$s_{\nu}[(1,1),(1,1)]=1$$

, where as

$$s_{\nu}[(1,1),(2,2)]=\frac{4}{5}$$

Hope this helps!


Error while trying glmnet() in R: "Error in storage.mode(xd) <- "double" : 'list' object cannot be coerced to type 'double'" I'm trying to create a logistic regression model using Ridge, this is the code:
glmnet(X_Train, Y_Train, family='binomial', alpha=0, type.measure='auc')

And this is the error message I'm getting:
Error in storage.mode(xd) <- "double" : 'list' object cannot be coerced to type 'double'

I tried converting all the variables into "numeric" but still doesn't work.
I'm going to post the code for those two datasets so you can reproduce it:
libraries:
library(dplyr)
library(fastDummies)
library(missForest)
library(glmnet)

Data:
url <- 'https://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.data'
crx <- read.csv(url, sep = ",", header = F)

Getting rid of null-values:
crx[crx == "?"] <- NA
crx <- type.convert(crx, as.is=FALSE)
crx.i <- missForest(as.data.frame(crx))
crx <- crx.i$ximp

Data transformations:
crx <- crx %>%
rename(Gender = V1,
         Age = V2,
         Debt =  V3,
         Married = V4,
         BankCustomer = V5,
         EducationLevel = V6,
         Ethnicity = V7,
         YearsEmployed = V8,
         PriorDefault = V9,
         Employed = V10,
         CreditScore = V11,
         DriversLicense = V12,
         Citizen = V13,
         ZipCode = V14,
         Income = V15,
         ApprovalStatus = V16)

crx = subset(crx, select = -ZipCode)

crx <- crx %>%
mutate(ApprovalStatus = recode(ApprovalStatus,
                  "+" = "1",
                  "-" = "0"))

# Normalizing numeric variables:
crx$Age <- scale(crx$Age)
crx$Debt <- scale(crx$Debt)
crx$YearsEmployed <- scale(crx$YearsEmployed)
crx$CreditScore <- scale(crx$CreditScore)
crx$Income <- scale(crx$Income)

crx$Gender <- NULL
crx$DriversLicense <- NULL

Creation of dummy variables:
df <- dummy_cols(crx, remove_selected_columns = T)

df$ApprovalStatus_0 <- NULL
df$ApprovalStatus_1 <- NULL
df$Married_l <- NULL
df$BankCustomer_gg <- NULL

df$ApprovalStatus <- crx$ApprovalStatus

Creation of Training datasets and Test datasets:
X <- df %>% dplyr::select(-ApprovalStatus)
Y <- df$ApprovalStatus

X_Train <- X[0:590, ]
Y_Train <- Y[0:590]

X_Test <- X[591:nrow(X), ]
Y_Test <- Y[591:length(Y)]

And trying to use the glmnet:
glmnet(X_Train, Y_Train, family='binomial', alpha=0, type.measure='auc')

I did some research and I found an article saying that you have to convert everything into numeric class, so I tried converting everything into numeric variables like this:
Y_Train <- as.numeric(Y_Train)
X_Train <- as.data.frame(apply(X_Train, 2, as.numeric))

And still doesn't work. What am I doing wrong exactly?

Glmnet requires a matrix as input for both, $X$ and $y$. So you need to define as.matrix() on all model inputs.
For further examples also see the Glmnet Vignette by Trevor Hastie and Junyang Qian.


How to specify output_shape parameter in Lambda layer in Keras I don't understand how to specify the output_shape parameter in the Lambda layer in Keras/Tensorflow. The documentation says:

output_shape: Expected output shape from function. This argument can
be
inferred if not explicitly provided. Can be a tuple or function. If a
tuple, it only specifies the first dimension onward;
sample dimension is assumed either the same as the input: output_shape = (input_shape[0], ) + output_shape or, the input is None and
the sample dimension is also None: output_shape = (None, ) + output_shape

If we use a tuple how should I interpret these two expressions?
output_shape = (input_shape[0], ) + output_shape
and
output_shape = (None, ) + output_shape

Let's say you pass in output_shape as a tuple (50, 50, 10) where we can call the values (height, width, channels)` to the lambda layer:
your_layer = tf.keras.layers.Lambda(lambda x: x, output_shape=(50, 50, 3))

The part of the documentation:

If a tuple, it only specifies the first dimension onward;

means that the batch dimensions itself is simple carried forward, unchanged.
If you have e.g. batch_size=3 during training, the incoming tensor to your_layer might be (3, n, p, q), where n p and q could be anything, but the layer is expected to produce a shape (3, 50, 50, 10). So the 0 dimension remains unchanged, and we have concatenated it with your output_shape:
(3,) + (50, 50, 10) -> (3, 50, 50, 10)

This corresponds to the expression: output_shape = (input_shape[0], ) + output_shape, so we see that input_shape is the true shape of the incoming batch tensor during training, as we only took the batch dimension to produce the layer's outgoing batch tensor.
For the second expression it is really just the same thing, but if you haven't provided a batch shape, Tensorflow & Keras represent that as something that could be anything, and store it as None. So in that case you get:
(None,) + (50, 50, 10) -> (None, 50, 50, 10)


Experience Replay, must return minibatch back to Memory Bank? During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.

When done, should we return the minibatch back to the memory bank, or should we throw it out?

Doing the former seems to grow the memory indefinitely, and might result in some state being sampled a few extra times in the future, if the state is lucky and ends up in future mini batches.

If you throw it out, training might suffer in the beginning as you would have even less data at that time.

If you do throw it out, then you have to be aware of your fit interval, your expected episode length, and batch size so you aren't throwing out data faster than you generate it.

As for the memory growing indefinitely, I've used a deque to limit its size.

from collections import deque
memory = deque(maxlen=10000000)


How can I override the values of 1 column in a big dataframe using a small second dataframe? I have a Pandas dataframe (donations_df) that contains thousands of donations. Each donation row has many columns, but the two key ones for my question are:

A recipient_id column indicating who received the donation
An office column indicating what legislative office the recipient holds.

Some of the values of the office column in donations_df are outdated, and I want to replace/override them.

So I have a second, much smaller dataframe (overrides_df). It contains the preferred office names I want to use. It has only a few rows (one for each recipient). For example:

| recipient_id | office             |
| ------------ | ------------------ |
| 3839314      | Governor           |
| 24811446     | Secretary of State |
| 12733609     | Auditor            |
| 6676482      | Attorney General   |

What is the proper way to join/merge/update this?

What I want to happen is that for each donation in donations_df, if that donation's recipient_id exists in overrides_df, then then the donation would pick up the new office value from overrides_df

What is the proper way to join/merge/update these two to achieve that result?

Try this:

for recipient_id, office in override_df.values:
    donations_df.loc[donations_df["recipient_id"] == recipient_id, "office"] = office

However, you should use a reference table to avoid this problem!


Selection of Features and Data in Random Forest First, I am confused whether at each node in all the trees, do we randomly pick features from the lot to be pitted for best split or does each tree get a random subset of feature and then all the nodes in that tree have to work with those features?

Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees.

First, I am confused whether at each node in all the trees, do we
  randomly pick features from the lot to be pitted for best split or
  does each tree get a random subset of feature and then all the nodes
  in that tree have to work with those features?

m-features are chosen randomly at each split of the tree, not at the creation of the tree itself.

  Now, regardless of the case whether we pick random features at tree or node level, why do choose random subset of data for each tree? Because if the features are different at tree/node level, the trees will be different. Why do we need to add another level and make the distribution of data also random to the trees and not just feed the complete original dataset to the trees?

This is exactly what makes random forests unique compared to other algorithms. Indeed, we sample with replacement (bootstrap) from the training dataset and each bootstrap sample is unique to each tree (which you appear to understand).

However, we need to consider how the final predictions come out of a random forest in the first place. Recall that the final prediction for an observation $y_i$ will be the average predicted probability among all of the trees in the forest. This is supposed to work because, under the (crucial) assumption of zero/negative correlation between the trees, we know that the sample mean has variance that goes to zero as n goes to infinity. This is one key element of random forests; if I have a bunch of overfit trees (low bias, high variance) I can simply get rid of this high variance for each tree by combining the predictions of each individual tree and averaging. The result is hopefully low bias, and low enough variance for my final, combined model.

However, if there is positive correlation among the trees, the variance of the sample mean actually increases (as  $ n \rightarrow \infty$) and if n is large enough, the covariance between the trees actually starts to dominate the variance of each individual tree itself. What this results in is a decrease in predictive performance compared to just using a single (or relatively few) tree(s). After all, the whole point is to fit low bias, high variance base learners and to then reduce the variance by averaging. If we now cannot reduce the variance due to correlation between base learners (and start increasing it instead!), our model is likely to be severely overfit, even moreso than using a comparatively less overfit base learner.

So the question becomes...how do we reduce the correlation between the trees? One solution is to introduce randomness not only through the data sample itself (i.e. bootstrap/sample with replacement) but also through randomly selecting predictors at each split of a node. By doing this, we hopefully force each tree to learn something different about the data. Indeed, if only a few predictors are useful than chances are, all of the trees will pick up on this and stick to using these few predictors over and over again. The result is that all of the trees have learned exactly the same thing, substantial correlation among the trees, and not much purpose to using the trees together.

Bagging (which is what you refer to as the first level of randomness) and other ensembling based methods work best when we have what people call "diverse base learners". This means that in general, if we can find many models that are similar in performance but uncorrelated (that is, they have learned different, but equally important parts of the data) then combining the models in some fashion will almost certainly be better than using either one alone. Creating model diversity is difficult but some common ways are;

1) Do something similar to random forests; give each base learner a different set of features to use.
2) Use different algorithms that hopefully learn different parts of the data due to the differences in how they are fit; example: random forest + neural network + gradient boosting, etc.


Negative range for binary cross entropy loss? So based on the website:

https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a

The binary cross entropy loss is defined as:

Which is applicable for output range {0,1}. However, what if i scale the output to be now {-1,1} instead? How would the new cross entropy loss be derived?

One thing you can do, is forcing your labels (${-1,1}$) to be ${0,1}$ using this simple linear transformation:

\begin{equation*}\hat{y} = (y + 1) / 2\end{equation*}

This way, -1 maps to 0, and 1 maps to 1.

For practical purposes, you can either change the outputs and labels of your model directly (before applying the original BCE), or slightly change to BCE loss function according to the linear transformation:

\begin{equation}H_p(q) = -\frac{1}{N} \sum_{i=1}^N \frac{(y_i+1)}{2} \log\left(p\left(y_i\right)\right) + (1-\frac{(y_i+1)}{2})\log\left(1-p\left(y_i\right)\right)\end{equation}

Notice that I changed only the coefficients, assuming that the probability function $p$ only returns meaningful values for ${-1,1}$

NOTE: obviously, all of this is relevant only for the binary cross entropy case.


Clustering Observations by String Sequences (Python/Pandas df) I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both columns to index form to simplify the data. As such, the set has 2 columns; 1. A unique ID and 2. A bag of key strings (they are identifying numbers but in string format, separated by spaces). Example format:

ID |  KeyWords
0 |   1 2
1 |   2
2 |   3
3 |   2 3

I initially vectorised KeyWords by TD-IDF and used a simple and dirty KMeans (via sklearn) to cluster them into 13 clusters (selected by the elbow method), which seemed to work and output intuitive and interesting results.

However, the issue I have come across is that there is always one dominant cluster that holds ~80% of the observations. I assume that KMeans was the limiting factor with assumed constant variance, and looked into Latent Dirichlet Allocation (LDA), as well as Gaussian Mixture Modelling (GMM) and BayesianGMM as alternatives. Using the same data format as above, LDA seemed to produce unintuitive results, for which all the topics were very similar and did not make much sense (vastly different to KMeans clusters, expected to see some key similar topics?).

I'm quite new and out of my depth with clustering and have been reading quite a lot about it, but if anyone could provide any insights into immediate logic flaws or a push in the right direction I would very much appreciate it.

Edit:
There's about 3300 unique URLs (or KeyWords in this case). The number of tokens/URLs per document is quite skewed actually. The median is 1, mean is ~1.5 (there is a small number of ID's that have a very large number of tokens, some I can deem as outliers but most I cannot). Another key fact is that in the dominant cluster, almost all of these "big players" who have looked at a large number of KeyWords are all within the dominant cluster. I removed them to see how it played out, and the dominant cluster remains.

As URLs are usually unique strings, I am not sure if NLP methods are proper things here. That is interesting that TFIDF+Kmeans gave you an intuitive results however as the problem is unsupervised, it´s pretty difficult to say what is intuitive that way.

In my opinion your problem is a typical Community Detection in Bipartite Graphs e.g. this paper. This approach is certainly worth to try. Of course the result can be combined with other algorithms e.g. in a voting schema.

The last but not the least is to take a clustering method which does not limit to resolution e.g. in Community Detection, any Modularity-based method is only able to capture clusters larger than a resolution limit. One approach to this is a hierarchical approach to partitioning the data in which you again apply clustering on the significantly large cluster you find.

Hope it helps!


RNN package and problems with "Predictr" I have two questions about how to use R's RNN package, specifically the trainr and predictr functions.

Let's suppose I have a time series of 4000 steps for 5 different variables.

How should this be passed on to the RNN, since the input has to be a 3D array? Should it be an array of dim(1, 4000, 5) or dim(4000,1,5) or something completely different?

Moreover, let's suppose I successfully train my RNN and would like to make some predictions using the predictr function.

Which dimension will the output have and how can it be interpreted?

Until now I have tried to input the data as 3D array (4000, 1, 5) with the following code:

RecNN <- trainr(TrainR, TrainI,
  learningrate = 0.003, momentum = 0.003, hidden_dim = c(100),
  network_type = "rnn", batch_size = 50,
  numepochs = 100, sigmoid = "logistic", use_bias = TRUE)

print("--- DONE LEARNING ---")

And then used the predictr function with the input (250, 1, 5):

results1 <- predictr(RecNN, TestI, hidden = TRUE, real_output = TRUE)

List of 2
 $ : num [1:250, 1, 1:100] 0 0 0 0 0 0 0 0 0 0 ...
 $ : num [1:250, 1, 1:2] 1 1 1 1 1 1 1 1 1 1 ...

Thanks a lot!

The manual gives needed dimensions (see help on trainr function):

Dimensions of input X need to be as:

dim 1: samples
dim 2: time
dim 3: variables

and dimensions of output Y should be:

dim 1: samples (must be equal to dim 1 of X)
dim 2: time (must be equal to dim 2 of X)
dim 3: variables (could be 1 or more, if a matrix, will be coerce to array)


Do I need to buy a NVIDIA graphic card to run deep learning algorithm? I am new in deep learning. I am running a MacBook Pro yosemite (upgraded from Snowleopard). I don't have a CUDA-enabled card GPU, and running the code on the CPU is extremely slow. I heard that I can buy some instances on AWS, but it seems that they don't support macOS.

My question is, to continue with the deep learning, do I need to purchase a graphic card? Or is there other solution? I don't want to spend too much on this...

I would recommend familiarizing yourself with AWS spot instances. It's the most practical solution I can think of for your problem, and it works your computer too. So, no you don't have to buy an Nvidia card, but as of today you will want to use one since almost all the solutions rely on them.


Pattern Recognition, Bishop - MLE and Least squares section 3.1 Please refer to an equation in Pattern recognition and Machine Learning by Bishop. My query is related to manipulation of an error term, equation 3.18:
$$E_{D}(\boldsymbol{w}) = \frac{1}{2}\sum_{n=1}^N\{t_{n} - w_{0} - \sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})\}^{2} \tag{3.18}$$
the author goes on to get:
$$w_{0} = \overline{t} - \sum_{n=1}^{M-1}w_{j}\overline\phi_{j} \tag{3.19} $$
$$where \quad \overline{t} = \frac{1}{N}\sum_{n=1}^Nt_{n} \qquad \overline{\phi_j} = \frac{1}{N}\sum_{n=1}^{N}\phi_{j}(x_{n}) \tag{3.20}$$
If I take the derivate of 3.18, w.r.t $w_{0}$, set it to zero, and divide both sides by $N$, I get:
$$w_{0} = \frac{1}{N}\sum_{n=1}^{N}t_{n} - \frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$$

Am I creating any error above? If so, please help. If not, my problem is related to the second term (R.H.S., above equation):

  The right term,
  $\frac{1}{N}\sum_{n=1}^{N}\sum_{j=1}^{M-1}w_{j}\phi_{j}(x_{n})$, is the
  average of double sum. I can't understand how we can simply 'push'
  $\frac{1}{N}$ inside to get
  $\sum_{j=1}^{M-1}\frac{1}{N}\sum_{n=1}^{N}w_{j}\phi_{j}(x_{n})$ and
  eventually equation for $\overline\phi_{j}$ (ref. 3.20), since:

$$\frac{1}{2}[(1+2) + (3+5)] \quad != 1 + 3 + \frac{1}{2}(2  + 5)$$

LHS is average of sum of sums and RHS is average 'pushed' inside.

We have

$$\frac12 [(1+2)+(3+5)]=\frac12(1+3)+\frac12(2+5)$$

We can swap the order of summation of the LHS since addition is commutative and associative.

After we decided how to pair them up, we just distribute $\frac12$ into them using distributive law.

\begin{align}
\frac1N \sum_{n=1}^N \sum_{j=1}^{M-1}w_j \phi_j (x_n) &=\frac1N \sum_{j=1}^{M-1}\sum_{n=1}^N w_j \phi_j (x_n) \\
&= \sum_{j=1}^{M-1}\frac1N\sum_{n=1}^N w_j \phi_j (x_n) \\
\end{align}

where the first equality is due to we are dealing with finite sum, and hence we can swap the order of additions, that is summing up the row first or column first doesn't matter.

After which, we use distributive law to bring $\frac1N$ inside the first summation.


Changing the batch size during training The choice of batch size is in some sense the measure of stochasticity :

On one hand, smaller batch sizes make the gradient descent more stochastic, the SGD can deviate significantly from the exact GD on the whole data, but allows for more exploration and performs in some sense a Bayesian inference.
Larger batch sizes approximate the exact gradient better, but in this way one is more likely to overfit the data or get stuck in the local optimum. Processing larger batch sizes also speed-ups calculations on paraller architectures, but increases the demand of RAM or GPU RAM.

Seems like a sensible strategy would be starting from smaller batch sizes to have a lot of exploration in the initial stages, and then increase gradually the batch size to fine-tune the model.
However, I have not seen implementing this strategy in practice? Did it turn out to be inefficient? Or the appropriate choice of learning rate  scheduler with the Dropout does this well enough?

Efficient use of resources
It is a balancing game with the learning rate, and one reason you don't normally see people do this is that you want to utilise as much of the GPU as possible.
It is commonly preferred to start with the maximum batch size you can fit in memory, then increase the learning rate accordingly. This applies to "effective batch sizes" e.g. when you have 4 GPUs, each running with batch_size=10, then you might have a global learning rate of nb_gpu * initial_lr (used with sum or average of all 4 GPUs).
The final "best approach" is usually problem specific - small batch sizes might not work for GAN type models, and large batch sizes might be slow and sub-optimal for certain vision based tasks.
Friends don't let friends use large batch sizes
There is literature to support usage of small batch sizes at almost all times. Even though this idea was supported by Yann Lecun, there are differences of opinion.
Super convergence
There are also other tricks that you might consider, if you are interested in faster convergence, playing with learning rate cycling.


ROC curve and optimal threshold I am doing a practice problem predicting a binary outcome. I have plotted an ROC curve and found the optimal threshold percentage to call future predicted observations a 1. I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?

I see that this threshold always matches the percentage of observations equal to 1 in my original data. Is there any conceptual explanation for this?

Yes, although the fact it always matches exactly is probably a coincidence or maybe due to a small sample.

The training data contains a proportion $p$ of instances labelled 1. From the ROC plot you can see all the possible values for setting the threshold at a certain level and the resulting performance; for every possible level you can calculate the corresponding proportion $q$ of instances predicted as 1:

if $q$ is much lower than $p$, then the system predicts many 0s, so there are many false negative errors and that makes the recall lower. Precision is high in this case.
if $q$ is much higher than $p$, then the system predicts many 1s, so there are many false positive errors and that makes the precision lower. Recall is high in this case.

I assume that you optimize on the F1-score right? The fact that the F1-score is based on the product of the precision and recall means that both values need to be reasonably high, otherwise the F1-score drops. As seen above, having very different values for $p$ and $q$ will cause either the precision or recall to be low. Therefore the optimal F1-score is achieved when $q$ is close to $p$.


what's the motivation behind BERT masking 2 words in a sentence? bert and the more recent t5 ablation study, agree that

using a denoising objective always results in better downstream task performance compared to a language model

where denoising == masked-lm == cloze.
I understand why learning to represent a word according to its bidirectional surroundings makes sense. However, I fail to understand why is it beneficial to learn to mask 2 words in the same sentence, e.g. The animal crossed the road => The [mask] crossed the [mask]. Why does it make sense to learn to represent animal without the context of road?
Note: I understand that the masking probability is 15% which corresponds to 1/7 words, which makes it pretty rare for 2 words in the same sentence to be masked, but why would it ever be beneficial, even with low probability?
Note2: please ignore the masking procedure sometimes replacing mask with a random/same word instead of [mask], T5 investigates this choice in considerable length and I suspect that it's just an empirical finding :)

Because BERT accepts the artificial assumption of independence between masked tokens, presumably because it makes the problem simpler and yet gave excellent results. This is not discussed by authors in the article or anywhere else to my knowledge.
Later works like XLNet have worked towards eliminating such an independence assumption, as well as other potential problems identified in BERT. However, despite improving on BERT's results on downstream tasks, XLNet has not gained the same level of attention and amount of derived works. In my opinion, this is because the improvement did not justify the complexity introduced by the permutation language modeling objective.
The same assumption is made by other pre-training approaches, like Electra's adversarial training. The authors argue that this assumption isn’t too bad because few tokens are actually masked, and it simplifies the approach.


How can I save my learning rate on each finished epoch using Callbacks? I used LearningRateScheduler for my model training. I want to save learning rates on each epoch in CSV file (or other document files).
Is there any way to save those learning  rates using callbacks?

You may write a Custom Callback and save the LR in a file.
You will get it by - self.model.optimizer.learning_rate
 Custom Callback - Keras docs
class CustomCallback(keras.callbacks.Callback):

    def on_epoch_end(self, epoch, logs=None):

        print("LR - {}".format(self.model.optimizer.learning_rate))

my_callbacks = [ CustomCallback() ]

LR - <tf.Variable 'Adam/learning_rate:0' shape=() dtype=float32, numpy=0.001>


Precision or Recall when dealing with critical cases? I have to create an AI that classifies mutliple objects in order to accept them or not inside a machine.
The problem is that some objects could be really harmful to the machine if they get accepted.
Should I focus on an high Precision or an high Recall, in order to not accept this kind of objects?

Let us say your target variable "1" represents "Harmful Object".
Precision answers the question - Of all objects predicted as Harmful - How many are really harmful ?
Recall answers the question - Of all the Harmful objects out there - How many were correctly identified ?
You want to increase on Recall in your case - since that reduces the overall risk to the machine.


Building a deep learning model I want to create a deep learning model for a binary classification problem. I have 16 features and 50k samples. How would I select number of layers and number of neurons of each layer  for my problem?

Please share the links which explains the same.

There are no rules to infer neural network hyper-parameters from a problem description. With only number of features, number of examples and the fact you have a binary classification problem, it is far too little to even make an educated guess.

First, are you sure you are ready to build a deep learning model for your data? Have you looked at the data, or scatter plotted it reduced to two dimensions using PCA or t-SNE, to get a feel for how easy your data is to separate (easy to separate raw data implies using simpler/shallower model)? You could try a basic model such as logistic regression or SVM in order to establish a benchmark so you can tell whether the deep model was doing anything useful.

Assuming you are ready to go ahead, the time-honoured approach is to try out variations, and measure the results using cross-validation. You can do this methodically, by for instance starting with one hidden layer with 64 neurons, and either adding/removing layers or adding/removing neurons in each layer. Generally these searches don't cover all possible variations, but stop once you have tried those that seem interesting and have reached a reasonable result for your problem.

Do note that other hyper-parameters will affect results and can interact with your choices for numbers of neurons and layers. You cannot isolate your choice of network depth from other choices, such as optimisation method, activation functions, regularisation. When starting to build a model, it is just as reasonable to spend a long time exploring these other factors as it is to look at network size/shape.

It is very easy to get a deep NN to over-fit your data. Cross-validation is therefore a necessity as you explore hyper-parameters for your problem. One reasonable method of searching against number of neurons is to increase them until you start to notice over-fitting, then adjust regularisation to stop the over-fit. At that point, with all other factors remaining the same, there is probably no need to explore networks with many more neurons (although deeper/shallower networks may still be worth exploring, and if you do so, you will once again want to explore number of neurons per layer).


Python: Unable to install pandas_profiling I have tried installing profiling pandas profiling library with the following command from the terminal:

conda install pandas-profiling

Output:

ModuleNotFoundError: No module named 'pandas_profiling'

What am I doing wrong?

Try:

pip install pandas-profiling

or:

conda install -c anaconda pandas-profiling


Does hierarchical agglomerative clustering with centroid-linkage suffer from chain-effect? It is known that the results of hierarchical agglomerative clustering using single-link method in order to determine the inter-cluster distance suffer form the chain-effect (natural clusters tend to extend trough a line of few points, like in the image below).

Does centroid-linkage have the same disadvantage?

The potential worst cases of centroid linkage are probably too crazy to explain as easy as the single-link effect... To see how it reacts to the classic Single-Link problem, why don't you just try yourself?

Roughly, because of the way centroid linkage works, it may end up using a virtual cluster center that is outside of the actual cluster. You then may see some very weird links to happen. That is also why it can have non-monotone linkage levels (so later merges may be cheaper than earlier merges).


what is nn.index mean in KNN output I am getting attr(, "nn.index") as part of my KNN output in R. What is meant by that and how is this value getting calculated?

knn.pred <- knn(tdm.stack.nl_train, tdm.stack.nl_Test, tdm.cand_train)
print(knn.pred)

> knn.pred

[1] Silent      Silent      Silent      Silent      Silent      Silent      Silent
 [8] Silent      Silent      Silent

attr(,"nn.index")

      [,1]
[1,]  292
[2,]  292
[3,]  343
[4,]  444
[5,]  250
[6,]  445
[7,]  270
[8,]  228
[9,]  302
[10,]  355

I guess you are using the fnn package.

attr is a list of attributes which can be used for both nn.index and nn.dist. In this case, you are using index.

So, index returns an n x k matrix for the nearest neighbor indice.

And the definition of the nearest neighbor index is:

  The nearest neighbor index is expressed as the ratio of the observed
  distance divided by the expected distance.

Definition reference


How to improve Naive Bayes? I am solving a problem that address this question "What are the Actions that lead to high or low score?"

I have the following Data that consist of text and score , I want to derive the words or Actions from text that lead to high/low score

I have huge data , and text length can be a paragraph, sample data like below

+----------------------+-------+
|         Text         | Score |
+----------------------+-------+
| Support Team Goal    |    90 |
| Generate Lead        |    80 |
| Contact 30 customers |    30 |
| Support Team Goal    |    30 |
+----------------------+-------+

Approach followed:
I followed Naive Bayes way.

First I classified my Data into High (score(75 and above)) and Low (score below 75)
I converted the High to Term Document Matrix and the Low as well
I found words appearing only in low and judge they lead to low score (include word if frequency greater than 7 lets say)
I found words appearing only in high and judge they lead to high score (include word if frequency greater than 7 lets say in high)
I found probability of common words, For words appearing in both high or low, I calculated the probability of the word occurrence to happen in low/high (include word if probability greater than 75%)

Note: I worked with bigrams. Not sure if this is the right approach to analyze my problem, or there are better techniques. Please advise

Without looking at the actual data, all we can really do is second guess and suggest best practices. Here are some pointers you can pursue -

Gather more data - If it can be done, nothing like it.
Improve data quality - The algorithm will always be as good as the data. Try extensive cleaning methods like -
Lowercase (basic) so all your data is standardized
Stemming / Lemmatization - These techniques reduces a word to its root form
Try parts of speech tagging (POS) and retain important parts of speech like nouns and look at their overall importance
Stopword removal - Just like you tried doing word frequencies, there are popular word sets like the word the that don't help much. They are filtered out
Correcting Spelling mistakes - If you are expecting there will be spelling mistakes, might be a good idea to correct them
Converting the unstructured data - I see that you have used bigrams. Try using unigrams and trigrams as well, or in combinations, run your algorithm and see which one works better. Try CountVectorizer, TfidfVectorizer and other techniques like word embeddings as well
Algorithms - Finally focus on the algorithm itself. For Naive Bayes, focus on MultinomialNB. Try RandomForestClassifierand other ensemble family algorithms. Try Deep Learning techniques with keras. Fine tune hyperparameters based on the validation results

Of course, there are other best practices like splitting your data into train, test and cross validation sets. Evaluating the right metrics such as accuracy, precision, recall and the confusion matrix.

Hope this helps!


Linear Regression + KFold cross validation I have a prepossessed data set ready and the corresponding labels (8 classes).
I've already done KFold cross validation with  K=10 with some classifiers such as DT,KNN,NB and SVM and now I want to do a linear regression model, but not sure how it goes with the KFold , is it even possible or for the regression I should just divide the set on my own to a training and testing sets ?

The 8 classes are ages (for NLP problem) , so I want to check both options of classification and regression.

Cross-validation

An extremely important concept to understand is that:

Cross-validation works independently of the model you use.

Cross-validation is just the process of splitting your data into multiple pairs of training and test sets. Once this is done, you can train that data using whatever model you like. In sklearn, which I assume you are using given your tags, you can use cross-validation with any classifier/regressor you'd like.

Cross-validation is way to evaluate how well the model will perform on data that it hasn't seen before. When you train your model on all your data, you do not have any data that the model hasn't seen before to test on. This is why you cross-validate so that you can get multiple estimates of what the performance might be. This is more robust than just splitting your data once between a training and test set because in this case, it might happen that your test set is easier very difficult or very hard. By cross-validating, you smooth this out.

Linear vs Logistic regression

Apparently, your output is ages, and therefore, you could use linear or logistic regression. If you use linear regression, you will try to guess the age directly. If you use logistic regression, you could round the output to the closest integer to obtain the proper target classes.

I would advise you to use logistic regression because it will allow your model to learn better. If you use linear regression, your classes are all as different as each other (from a mathematical point of view). However, my guess is that the 13 class is closer to 14 than it is to 20+.


Is my model over fitting or not? I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I don't understand why. I replaced all outliers with 5th and 95th quantile. Standardized the data yet it is showing this unreal accuracy.
A bit of online search and people suggested to check for the difference between training and test data accuracy, which i did and for Random Forest it came

Training Accuracy: 0.997975
Test Accuracy: 0.9715

For logistic regression it shows

Training Accuracy: 0.967225
Test Accuracy: 0.9647

This is the code I used for running the model:

clf = LogisticRegression()
trained_model = clf.fit(X_train, y_train)
trained_model.fit(X_train, y_train)
predictions = trained_model.predict(X_test)

accuracy_score(y_train, trained_model.predict(X_train))
accuracy_score(y_test, predictions)

I also tried kfold cross validation which gave similar results

skfold = StratifiedKFold(n_splits=10, random_state=100)
model_skfold = LogisticRegression()
results_skfold = model_selection.cross_val_score(model_skfold, X, Y, cv=skfold)
print("Accuracy: %.2f%%" % (results_skfold.mean()*100.0))

Lastly I applied regularization technique to check for results and this is the result  I got

for c in C:
    clf = LogisticRegression(penalty='l1', C=c, solver='liblinear')
    clf.fit(X_train, y_train)
    y_pred_log_reg = clf.predict(X_test)
    acc_log_reg = round( clf.score(X_train, y_train) * 100, 2)
    print (str(acc_log_reg) + ' percent')
    print('C:', c)
    print('Coefficient of each feature:', clf.coef_)
    print('Training accuracy:', clf.score(X_train_std, y_train))
    print('Test accuracy:', clf.score(X_test_std, y_test))
    print('')

The results

96.72 percent
C: 10
Coefficient of each feature: [[-2.50e+00 -1.40e-03  2.65e+00  4.09e-02 -2.03e-03  2.75e-04  1.79e-02
  -2.13e-03 -2.18e-03  2.90e-03  2.69e-03 -4.93e+00 -4.89e+00 -4.88e+00
  -3.27e+00 -3.30e+00]]
Training accuracy: 0.5062
Test accuracy: 0.5027

96.72 percent
C: 1
Coefficient of each feature: [[-2.50e+00 -1.41e-03  2.66e+00  4.10e-02 -2.04e-03  2.39e-04  1.68e-02
  -3.29e-03 -3.80e-03  2.52e-03  2.62e-03 -4.22e-02 -9.55e-03  0.00e+00
  -1.73e+00 -1.77e+00]]
Training accuracy: 0.482525
Test accuracy: 0.4738

96.74 percent
C: 0.1
Coefficient of each feature: [[-2.46e+00 -1.38e-03  2.58e+00  4.03e-02 -1.99e-03  2.22e-04  1.44e-02
  -4.49e-03 -5.13e-03  2.03e-03  2.20e-03  0.00e+00  0.00e+00  0.00e+00
   0.00e+00 -6.54e-03]]
Training accuracy: 0.616675
Test accuracy: 0.6171

95.92 percent
C: 0.001
Coefficient of each feature: [[-1.43e+00 -6.82e-04  1.19e+00  2.73e-02 -1.10e-03  1.22e-04  0.00e+00
  -2.74e-03 -2.55e-03  0.00e+00  0.00e+00  0.00e+00  0.00e+00  0.00e+00
   0.00e+00  0.00e+00]]
Training accuracy: 0.655075
Test accuracy: 0.6565

The codes I used for Standardization and replace outliers

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
X_test_std  = std_scale.transform(X_test)

X.clip(lower=X.quantile(0.05), upper=X.quantile(0.95), axis = 1, inplace = True)

Do let me know if any other information is required and any guidance will be appreciated

I have less than enough karma to leave a comment, but I'd like to support Derek O's assessment but also add 1 more point: if your 50K observations are repeated measures (multiple rows coming from the same individual or unit) - then you'll want to make sure that in your cross-fold setup that you are ensuring that 100% of each individual's observations are falling into the same fold.
An analogy would be that if you were modeling teeth for a probability of developing a cavity - a human head has 32 teeth, and if a particular human's teeth end up in both your testing and training folds, this would be considered a form of leakage. This is because the 32 teeth are not totally independent, but correlated with other teeth within the same head. This particular form of leakage often slips people's minds.


Forcing class imbalance to mirror the target data I'm trying to do binary classification on some data, my source data has a class split of 40% A / 60% B while my target data has a split of 70% A / 30% B.

Is it a worthwhile strategy to use SMOTE to over-sample A such that I'm training on a class split that mirrors the data I'm trying to classify? The only metric that concerns me is accuracy.

Disclaimer: I am assuming that by "source data" you mean the data you are using for training and by "target data" you mean the real world data the model will aim to classify.

SMOTE Is usefull when your data is highly imbalance i.e. 1% A/ 99% B or something around this proportions. In your case I would expect the model to work just fine when classifying a different distribution of data as long as you have enough training examples in each class. If this approach doesn't work you can use cost sensitive learning to fit the imbalance but I would consider this a secondary path


How to compute AUC in gridsearchSV (multiclass problem) I'm working on a multiclass classification problem, comparing results from SVM and Random Forest classificators. I would like to use gridsearchCV for hyperparameters tuning and find that AUC is the most used metrics for this kind of problem.

I know how to use other metrics of scoring like accuracy etc. but the default ROC_AUC only works for binary class. Is there a method to use AUC in gridsearchCV for multiclass problems?

Metrics are independent from ML algorithms, so it doesn't matter which algorithms did you use.

To calculate multiclass AUC you could use lib pRoc in R or use code this link(in Python).

Sources:

https://web.expasy.org/pROC/
https://medium.com/@plog397/auc-roc-curve-scoring-function-for-multi-class-classification-9822871a6659


In Conditional Random Fields, is mandatory to use features related to following and preceeding tokens? I am training a CRF classifier to classify document rows as a heading (1st level),  heading (2nd level) or simple text.
I am using Conditional Random Fields for their ability to account sequential aspects.
Reading some tutorials, I noticed that usually, among the features, there are some features related to the preceeding or following token.
[...]
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:word.isupper()': word1.isupper(),
            '-1:postag': postag1,
            '-1:postag[:2]': postag1[:2],
        })
    else:
        features['BOS'] = True
[...]

I wonder if the sequential aspect is learned from these features or is connate in CRF. In other words, do we need these features related to other tokens?

In other words, do we need these features related to other tokens?

No, these features are not needed. But they are often useful: CRFs handle sequential dependencies between the labels, however it's up to you to provide the relevant features, in particular some to represent the dependencies between (certain) features if needed.

With text, this relation between consecutive tokens is very often a relevant indicator. I'd suggest that you try both version, without and with this feature, and you're likely to observe a higher performance in the latter case. In my experience it's often worth trying different combinations, including trying features which go two or three steps back.


Why is T test reweighting on a word X word co-occurrence matrix so effective? I am going through Stanford NLP class: http://web.stanford.edu/class/cs224u/

A task in the homework is to implement T-test reweighting on a word X word co-occurrence matrix:
https://nbviewer.jupyter.org/github/cgpotts/cs224u/blob/2019-spring/hw1_wordsim.ipynb#t-test-reweighting-[2-points]

$$\textbf{ttest}(X, i, j) =
\frac{
    P(X, i, j) - \big(P(X, i, *)P(X, *, j)\big)
}{
\sqrt{(P(X, i, *)P(X, *, j))}
}$$

I have 2 questions:

What is the intuition behind this formula? It looks a little like PMI but I can't understand what it's doing. The T-test explanation out there seems to be unrelated to this task.
It works amazingly well (when evaluated by this test): raw matrix yield a correlation score of 0.014, PMIed matrix 0.123 and t scored matrix: 0.408979. This number seems almost too good to be true for such a simple model. Can anyone bring some intuition/experience about why that is?

IT is very similiar to PMI, here you just expand it to the whole dictionary matrix (matrix representation of the whole vocabulary), normalize it by subtracting quantitive representation of the sum of words found in row i column j and than standardize. (Like when using sklearn Standardize(), similiar atleast)
Intuition? Well why is tf-idf working (generally as text quantification technique), you are focusing on essential n-grams and minimising away the rest, with this re-weighting you are getting close to tf-idf representation in a sence.


Need an advice on research topic I am about to choose ML research topic for my master thesis, but i am at a dead end. The problem is, that while reading research papers, i find solutions, but not an open problems.

For now, a came up with such ideas:

Research neural network quantization algorithms - i started to research this topic 3 months ago and it seems to me, that current papers reveals high accucary methods. I have not idea what can i to do in this field
Implement online video-processing system using quantized neural networks. It sounds also good, but it could be done with few lines of Tensorflow Lite code.
Intelligent Image Database - image database, that has image classifiers and detectors for automatic annotation. It seems to be a good idea, but i need to find here a research part. For me this tasks sounds like pure engineering task
Make some signal processing research. It could be connected with first idea as optimal hardware implementation. But i don't know where to search for suitfull specific task for ML, because a lot of dsp problems are solved with classical analytical mathematical methods.

As you can see, i always see some disadvatages in ideas. It stopping me from starting to work. Can you give me an advice how to develop one of this ideas to make it suitable both for engineering and research? Generally, how to search for research topics?

You're probably aiming too high: a research topic doesn't have to lead to a major breakthrough, and very often it's impossible to know what it leads to before doing it. A Master thesis is not very long so you need to find a research topic which is feasible within the time frame. A common mistake by Master students is to spend too much time on implementation, then botching the analysis and/or writing a poor quality dissertation because there's not enough time left.

Discuss with your advisor, they have the experience and they know what is a good topic.
Read a few random papers in your domain. Why? Because if you read only the top quality papers (yes, of course these are important to read), you can get the impression that you need to achieve the same level of quality. But that's not the case: these top quality papers are 1 in a million and are often done by people who have years of experience behind them. A "regular" research contribution is often just a small improvement or a decent analysis on a specific problem.
Typical process: just pick a general topic/problem you're interested in, then study the state of the art thoroughly (write your state of the art section at this stage), reproduce a few experiments and analyze the results: what can be observed? what are the limitations? what would be a reasonable idea to overcome one of these limitations? Then you implement and evaluate this idea.

Keep in mind that in research what matters is not the complexity of the topic or even the performance, it's the rationale and the method (and of course originality of the work, but even that doesn't matter that much at Master level).


Classification model for recommender system? I have some data for various customers choosing one of 'n' products or no product. I have some useful features for each customer. I can build a multi-class classification problem out of this data and use a classification model (say, random forest) to learn the data. This model will then output one of the following categories:

[0, 1, 2, ..., n]

where 1, 2, ..., n is the 1st, 2nd etc. products, and 0 is when the customer chose not to buy anything. I want to make this model as a recommender system, i.e., when a new customer (along with all other features) comes along, I know which one of the n products he will more likely buy.

The question is this: in the above setting, I don't know what I should do when the models outputs '0' for the new customer. What should be my action when such a situation, i.e., the new customer is not likely buy any of the products? Or should I formulate the classification problem in some other way? Is there any way we shouldn't formulate the problem as a classification problem at all?

One option could be to model the output slightly different. Instead of returning the type of product 1, 2, ..., n or 0 in case of no product maybe you could return a list of pairs (product, probability), if the highest probability is the one corresponding to 0 i.e. "no product" you could just recommend the 2nd best.

An advantage of this approach is that you could recommend more than one product, maybe the top 3, or a list of products whose probability is higher than a specific threshold, etc.

If the customer is new and there is no information you could base your recommendation on, maybe you could recommend the product that you would offer to a similar customer (if you can cluster the customers in any way) or simply the top selling products.


Learning rate Scheduler A very important aspect in deep learning is the learning rate. Can someone tell me, how to initialize the lr and how to choose the decaying rate. I'm sure there are valuable pointers that some experienced people in the community can share with others. I've noticed that many choose to do a custom scheduler rather than use available ones.

Can someone tell me why and what influences the change in the lr? And when to describe a lr as being small, medium or large? I want to understand it enough to actually make sound choices. Thank you kind souls. I appreciate this community very much.

Finding an optimal learning rate is an important step in optimizing a neural network. As discussed at length here it is not a trivial question, but there are some ways to get a good starting value. Main idea here is to plot loss vs learning rate for different values and choose the learning rate where the slope is the highest:

This image is from the link above. This figure tells me that anything from a little right of 10-5 to around 10-3 can be a good learning rate. One can also set a relatively high learning rate, and reduce it when the loss function reaches a plateau, so in the above example, you would be better off by starting with the highest end; and lower the rate later as needed. This can be achieved by a learning rate scheduler (such as the one in Keras callbacks). This way, you won't spend a lot of time at the initial epochs where there is a lot to learn and the loss drops quickly. Three key parameters of a scheduler are "factor","patience" and "min delta" . If the loss function does not change by "min delta" in "patience" number of epochs, the learning rate is reduced by the ratio defined in "factor".

You also mention decay in your post, which can be considered as the default learning rate for some optimizers. And if I remember correctly, it actually associates a different learning rates for each parameter, and reduces them at a different rate. You can find a more detailed comparison here. But the main reason to use a LR scheduler even when a decay is present is to have more control on the learning process. When the training reaches a plateau, you might have to wait for a long time before the default decay in Adam (for example) gets to a low enough value to get over that region and start learning again. But with a scheduler, you can define when to lower the learning rate by a significant amount (compared to default decay), so that further learning is possible. Keep in mind that there is not much to learn in the later epochs compared to the ones in the beginning.


How does multicollinearity affect neural networks? Multicollinearity is a problem for linear regression because the results become unstable / depend too much on single elements (source).

(Also, the inverse of $X^TX$ doesn't exist so the standard OLS estimator does not exist ... I have no idea how, but sklearn deals with it just fine)

Is (perfect) multicollinearity also a problem for neural networks?

Multi colinearity affects the learning of Artificial Neural network.  Since the information in the dependent variable is very less compared to the other variables, the neural network will take more time to converge.

In packages like sklearn, the dependent variables are identified and omitted from the calculation.  I have used the lm function in R and it marks the coefficient of the dependent variable with NA.  one can remove the variable from the calculation and still the coefficients are going to be same.  In these cases, the rank of the x matrix will be less than the number of columns.

Even though there are no inverse exists for xTx, most of the packages will not calculate the inverse directly, but they will calculate the pseudo inverse.


Knn and euclidean distance I'm studying the knn classification algorithm.
Why can the euclidean distance be considered a nice measure of affinity between examples ?

In one dimension (1 attribute) this seems correct, but if I add dimensions, can the euclidean distance still be considerd a good measure of affinity? Why?

You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine distance or $L^p$ metrics. In fact, if $p$ is very high, then Euclidean distance is not a good measure, as it tends to make the distances too uniform (see this paper).

Edit: When $p$ is very high:

See this magnificient answer to the issues that very high $p$ may have.


Metric to evaluate words generated by Neural Network I have this task at hand and I would be grateful for some directions. Perfectly not the final solution as I would like to do it myself.

Let's say I need to create new fruit names based on existing ones.

I made it using textgenrnn library.

Now I have a list of potential new names.
I would like to build a metric which would calculate that e.g.:

(Fabic, Alis, Brooty, Morange etc) are potentially good fruit name
(Ssae, Sriew, Adeoie, Seeea) are potential bad fruit names.

Is there any list of metrics for word generating?

As of now, I found following resources:

Evaluating Generative Models for Text Generation https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1174/reports/2737434.pdf
Character-Level Language Modeling with Hierarchical Recurrent Neural
Networks http://150.162.46.34:8080/icassp2017/pdfs/0005720.pdf
Graves A. Generating Sequences With Recurrent Neural Networks
https://arxiv.org/pdf/1308.0850.pdf
Subword language modeling with neural networks.
http://www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf

But since I learnt about them today, I would need a more vanilla introduction to this topic of validation.

Maybe somebody will have a better idea but the default method would be to generate a set of names, then ask a few annotators to label them as good or bad (possibly scoring them from 1 very bad to 5 very good), and finally train a supervised model to recognize the good from the bad ones. This approach would also give you the opportunity to check the inter-annotator agreement, i.e. assess how subjective the choice good vs. bad is. In general a subjective task is difficult (or impossible) to do with an unsupervised metric.


How to decide optimal threshold for my classification model from FPR, TPR and threshold I am building my model in Python to classify customer in buyer/ non-buyer category. I used mutiple agorithms for this problem and then after evaluation selecting the best out of all.

sklearn package in python gives me an array of FPR,TPR and threshold for all my predicted records.

While analyzing the value of FPR,TPR and threshold I got confused how to interpret threshold value. I am getting threshold value for each record.

For example: My testing dataset consist of 100 records tehn I got array of size 100 for TPR,FPR and threshold.

How to decide the optimium threshold for my model using these values??

from sklearn import metrics
fpr, tpr,threshold = metrics.roc_curve( Y_test, status[:,1])

I’m willing to bet that you have a 100 value array for both the ROC and the data by coincidence. You aren’t getting a threshold for each data record but for each threshold evaluated.

At this point, you have to define what ‘optimal’ means in your context. By default (in many R packages at least), sensitivity and specificity are equally weighted, in which case the ‘optimal’ threshold corresponds roughly to the threshold where a 45 degree tangent line intersects the ROC curve. You can choose any threshold you want for a given ROC, depending on how you weight sense/spec.  R has a few packages that help compute that; I’d be surprised if Python did not have an equivalent.


generalized likelihood ratio test (GLRT) I am having trouble in understanding the generalized likelihood ratio test (GLRT). Can anyone explain what it is to me, or point me toward an easy-to-understand reference? Is it a supervised or unsupervised method? How GLRT relates to Bayesian approaches?

Likelihood-ratio tests are a mainstay of classical hypothesis testing. The idea is to form the likelihoods of the two hypotheses under consideration, and choose the one with the highest likelihood if their ratio is sufficiently large. Hypotheses come in two flavors: simple, and composite. Simple tests are those for which the hypothesis uniquely defines the distribution; e.g., the mean is that, or the variance is that. If it is not simple, it is composite; e.g., the mean is not equal to something, or the variance is less than something. Generalized likelihood ratio tests apply to composite hypotheses, and the goal is to find the distributions out of all possible options in the hypothesis space that maximize the likelihoods, and consider their ratio.

In Bayesian statistics one considers the posterior probabilities instead of the likehoods, and the corresponding ratio is termed the Bayes factor.

Supervised/unsupervised learning is something else.

I suggest you consult a textbook for worked examples; I think my explanation does not do the subject justice.


Intuitive explanation for representing gradient in higher dimensions I do not understand how complex networks with many parameters/dimensions can be represented in a 3D space, and form a standard cost surface just like a simple network with, say, 2 parameters.
For example, a network with 2 parameters that correspond to the X and Y axis, respectively, and cost function that corresponds to the Z axis makes sense...but how can we have a network with 1000 dimensions being represented in a 3D space, on a planar cost surface (not sure if word planar is being used correctly). Am I thinking of dimensions in the network the wrong way?

Yes, you are thinking of multiple dimensions in the wrong way, but I'm not sure anybody knows the right way to think about them.
You don't represent complex networks with many parameters in 3D space. We start in low dimensional 2D and 3D space to develop an intuition about what happens with the gradient, how to move along it, and so on. Then, we go into thousands dimensional spaces, were we don't know what happens, and we apply our intuition - those spaces work the same way, just that we cannot visualize or comprehend them.
Your cost function will have thousands of dimensions, and your gradient will be the same. But we rely on our intuition from low dimensional spaces to manipulate them.


How to add numbers to the axes of a graph? I am trying to make a graph of k (as in k-means) vs error, and I can't get it to show the actual number of k on the x-axis (1-15, not just the even numbers), nor any any numbers at all if I add a title to X.

I have looked at the data frame documentation and the matplotlib axes documentation, and I couldn't find anything telling me how to get it to show the numbers? Even if I have to manually specify them?

Please see images below:

rdf.plot.line(title='Number of clusters Vs mean accuracy')

rdf.plot.line(title='Number of clusters Vs mean accuracy',
                x='parameters',
                y='mean_validation_score')

I don't understand how to get the x axis to show all 15 numbers (1-15).

Use xticks. e.g.

x=np.arange(1,16)
y = -60000*(3+np.log(1/x))
plt.plot(x,y,'b')
plt.xticks(x)
plt.show()


Meaningful and Non-Meaningful Data? I can understand what meaningful data is like its important information that can be used to evaluate something but I don't get what non-meaningful data is? Is it less important data?

"meaningful" is a vague word anyway, but yes you got the idea: in the context of a particular task meaningful data is the information which contributes to solving the task. Non-meaningful is the opposite, so information which doesn't help for the task. Sometimes non-meaningful data makes it harder to the ML algorithm to capture the relevant information, since it has to correctly identify what is meaningful and what is not (as opposed to the case where it's provided only with relevant information).

Example: if you want to predict how fast a car can go, engine type and manufacturer are meaningful data. Colour is not.


What Naive Bayes method is being used in this example? I'm reading an old book called "Data Mining Practical Machine Learning and Techniques";  It uses a tool called Weka to automate examples as well, but I'd prefer to use Python to do so.

I'm quite new to Machine Learning, and I'm just trying to line up examples with a computer API; I'm not certain of the type of Naive Bayes being used in the example (which I implemented in a spreadsheet from the book).  From the look of it, based on other examples in the book, it doesn't look to be multinomial-bayes, but I don't seem to be able to find an API that lines up with it either.

The example tries to determine if a sports team will go outside and play or not (yes, no) based on the Outlook (sunny, overcast, rainy), Temperature Classification (hot, mild, cold), Humidity Classification (high, normal) and if it is Windy or not (TRUE, FALSE).

The values are INPUT from data rows in the Weather Data Table 1.2 on page 11.

The processing of the INPUT begins and then in Table 4.2 The weather data with counts and probabilities they tally up probabilities and totals based on possible attribute values and totals.

They then add a new day row to the input and process which does not have a Will the team play value, but which does have all the other attributes with a single value for each.  The likelyhood is calculated for yes and no, and then in the Output a probability value for yes and a value for no are calculated.

I've included the spreadsheet here.

I realize this is really basic Machine Learning, but I just wondered if anyone could point me towards a python library or python algorithm that would handle such a problem...I've actually also tried to get it to work in Weka, and run into a similar issue in which I appear to be able to find a Naive Bayes algorithm without churning out the correct result presented in the book and in my spreadsheet.

So does this particular Naive Bayes method have a name and equivalent API?

A basic Naive Bayes is being used in this example. Each feature can have a number of different values within the ranges of 2 or 3.

Bernoulli Naive Bayes requires that each feature be either true or false or 0 or 1.

Multinomial Naive Bayes allows features to be of values 0+ as it is counting occurrences of features.

Hope this helps. The Naive Bayes wikipedia pages summarizes this pretty well as well.


Clustering based on distance between points I am trying to cluster geographical locations in such a way that all the locations inside each cluster are at max within 25 miles of each other. For this, I am using Agglomerative clustering. I am using a custom distance function to calculate the distances between each location. I do not want to specify the number of clusters. Instead, I want the model to cluster until all the locations within each cluster are within 25 miles of each other. I have tried doing this in both Scipy and Sklearn but haven't made any progress. Below is the approach that I have tried. It only gives me one cluster. Please help. Thanks in advance.

from scipy.cluster.hierarchy import fclusterdata
max_dist = 25
# dist is a custom function that calculates the distance (in miles) between two locations using the geographical coordinates

fclusterdata(locations_in_RI[['Latitude', 'Longitude']].values, t=max_dist, metric=dist, criterion='distance')

I think for HAC (Hierachical Aglomeritive Clustering) it's always helpful to obtain the linkage matrix first which can give you some insight on how the clusters are formed iteratively. Besides that scipy also provides a dendrogram method for you to visualize the cluster formation, which can help you avoid treating the clustering process as a "black box".

import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage

# generate the linkage matrix
X = locations_in_RI[['Latitude', 'Longitude']].values
Z = linkage(X,
            method='complete',  # dissimilarity metric: max distance across all pairs of
                                # records between two clusters
            metric='euclidean'
    )                           # you can peek into the Z matrix to see how clusters are
                                # merged at each iteration of the algorithm

# calculate full dendrogram and visualize it
plt.figure(figsize=(30, 10))
dendrogram(Z)
plt.show()

# retrive clusters with `max_d`
from scipy.cluster.hierarchy import fcluster
max_d = 25       # I assume that your `Latitude` and `Longitude` columns are both in
                 # units of miles
clusters = fcluster(Z, max_d, criterion='distance')

The clusters is an array of cluster ids, which is what you want.

There is a very helpful (yet kinda long) post on HAC worth reading.


Bad results for training set and good results for test set When training an algorithm, I have ROC = 0.5896, Sensitivity = 0.3333, and Specificity = 0.8375. When considering the test set, Sensitivity = 1 and Specificity = 1. This could happen or is a problem?
Another question:
When comparing several learning algorithms, I choose the best based on the results of the test set?
Any help will be appreciated.

Seems quite strange. I bet you've got a bug or test set size is small enough. Like if there're only 2 objects: 1 positive and 1 negative, you could have classified them correctly even with a random classifier. I assume your test set is not that small, which means there's a bug somewhere. Also, in theory, if both of your sets are large enough and are drawn from the same data distribution, you can usually have one of 2 types of behavior:

about the same result both on training and testing sets (when your model underfits or fits well)
better results on the training set then on the testing (when your model if overfit)


Can finite state machines be encoded as input/output for a neural network? I want to encode finite state machines (specifically DFAs) as output (or input) of a neural network for a supervised learning task.

Are there any ways in the literature for doing this?

I've already found some algorithms being able to extract a DFA from a recurrent neural network, but nothing about DFAs either as input or output of ANN.

The better question might be - Why you do want to mix deep neural networks (DNN) and a finite state machine (FSM)?

There is research showing that a DNN can simulate any FSM. Since there are more frameworks for DNN and DNN can perform more tasks than FSM, it appears to be more useful just to forgo FSM altogether.

More specifically, "Neural network for synthesizing deterministic finite automata" shows how a relatively simple neural network (NN) can quickly and automatically learn the correct deterministic finite automaton (DFA).

There is a strong trend towards end-to-end DNN.


What does a deep learning model look like in file? So after you train your model with millions of rows of data, what is the model being saved in disk look like? Is it file with millions of weights and biases that would do the best predictions?

Yes you are right. It depends on which frameworks you are using for training the model.

For example, if you use tensorflow framework, the trained model will be saved into several files. You can save and load model into two ways:

Checkpoint
Savedmodel (freeze)

Please check this link. Here

In keras, Model weights are saved to HDF5 format. This is a grid format that is ideal for storing multi-dimensional arrays of numbers. you can save and load your model to file:

Save Model to JSON.
Save Model to YAML.

The JSON format of the model looks like the following:

    {
   "keras_version":"2.0.2",
   "backend":"theano",
   "config":[
      {
     "config":{
        "dtype":"float32",
        "bias_regularizer":null,
        "activation":"relu",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "kernel_regularizer":null,
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_1",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "batch_input_shape":[
           null,
           8
        ],
        "units":12
     },
     "class_name":"Dense"
  },
  {
     "config":{
        "kernel_regularizer":null,
        "bias_regularizer":null,
        "activation":"relu",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_2",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "units":8
     },
     "class_name":"Dense"
  },
  {
     "config":{
        "kernel_regularizer":null,
        "bias_regularizer":null,
        "activation":"sigmoid",
        "bias_constraint":null,
        "use_bias":true,
        "bias_initializer":{
           "config":{

           },
           "class_name":"Zeros"
        },
        "activity_regularizer":null,
        "kernel_constraint":null,
        "trainable":true,
        "name":"dense_3",
        "kernel_initializer":{
           "config":{
              "maxval":0.05,
              "minval":-0.05,
              "seed":null
           },
           "class_name":"RandomUniform"
        },
        "units":1
     },
     "class_name":"Dense"
  }

   ],
   "class_name":"Sequential"
}


How to score the health of a company? i'm currently doing dual apprenticeships. My main mission is to represent the health of a company based on accounting records for multiple companies over multiple years. The part of an accounting record i'm interested in is simply the date and the amount spend or receive. We're in a context of time series.
The objective is to build scores, each one representing a part of the company. I have a few indicators for each one of this company's part.
I didn't find any scientific articles about it, so i proposed an algorithm to do so. Here is it :
For each indicator

Build the mean for this indicator for each company
Use a regressive model to get the trend
Compute the difference to this trend (the objective is to increase the amount of data)
Make a forecast
Build the mean for this forecast
Use a regressive model on this forecast to build the trend
Compute the difference to this trend for each company
Based on this difference, build the score

I know this is not perfect, but after a lot of thinking, that's what i ended up with.
For the forecast part, i'm planning to use an LSTM.
I would like to hear your thought about my algorithm, even if you think it is completely crap. My objective is to improve my skills and to build the best system i can. I'm still a bit lost : should i try to transform the problem and go for more classical, like a simple classifier, or should i do that just for the last part ? Well, i don't know.
Thank you.
EDIT
As Julio made me realize, i forgot to mention one important thing. In France, each one of the record has an identifier, which represent what type of of income/outcome it is. For example : each accounting record with the id 40 are accounting charges. In this example i showed you a simple 2 digits identifier, but it can go up to 8.

To me it looks like your question is more about expert knowledge than ML approach: the main problem is how to build a sensible score based on the available indicators. I have zero knowledge about this domain, but in general this kind of problem requires a lot of back and forth with domain experts in order to correctly specify the task in detail.
However I have one piece of advice which might help (or not, maybe you already know this) for the evaluation of any system you build for this task: assuming that you obtain a predicted score which is meant to represent the future of the company, you could evaluate how well this score matches "the future" as follows:

Split the data at a particular point in time in the past, say 2010
Use only the data before 2010 to train and test a model, i.e. obtain predictions based only on data before 2010
Evaluate how well the predicted scores match the "future" of the companies, i.e. the actual data after 2010. For example a low score should lead to poor indicators, a high score should lead to good indicators (you might need to devise a specific method to correlate the scores with the indicators).


Detect named entities inside words Some languages have word endings with their nouns (like Finnish, e.g. "in Berlin" -> "Berliinissä"). I have tried to annotate the characters in the training data as entities, but then I run the model, it doesn't detect the characters inside the word. When those characters are a separate word, only then they're detected. I am unable to think of an implementation to effectively detect named entities within a word. Any suggestions would be helpful.

I would recommend to look into character level named entity recognition. For example: Kuru et al, CharNER: Character-Level Named Entity Recognition, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016)

The authors evaluate on many highly inflected languages including Turkish, so this should be adequate for your Finnish use case

The code is here: https://github.com/ozanarkancan/char-ner

You should hopefully be able to download and get it running out of the box for training. Of course I am assuming you have a tagged NER corpus in Finnish, which you would need to preprocess to get into the same format as the CSV file that they use for Czech in the repo.


(Crowdsourced) Dataset with label/annotation metadata like duration/quality I'm looking for a research project for datasets (the more the better), potentially crowdsourced, which go beyond basic feature vectors + labels, and include additionally some metadata about the labels. Specifically I'd like the annotation time, or some other cost metric, and if the dataset was crowdsourced, the individual labels per annotator. On top, some measure of the quality of the labels by different annotators would be helpful as well. The Dataset domain is not relevant.
Also I'm not sure if https://datascience.stackexchange.com is the right place to ask for that, but maybe you can help me.
So far I could only find this dataset.

I did a project with the CheXpert dataset which includes metadata and labels for uncertainty. I don't know if it's the right kind of metadata you are looking for. In general, medical datasets usually contain metadata. A similar one would be from the NIH.
Otherwise, this is a decent index of high quality datasets.


Cross validation for unbalanced dataset using Orange data mining tool I am using the Orange data mining tool to build and analyze models (decision tree, ANN, ...) predicting customer churn. As this is an imbalanced class problem (10% churn, 90% not churn), I need to oversample within the cross validation. However, I am not totally able to implement this by myself. Is there anyone with some Orange knowledge that could help me?
Thank you!

Orange does not have over/undersampling. Our reasoning is that if you model a problem with 10% positive class, than you should not train the model with 50:50 class distribution - it will not reflect the real life. However, there's an option in Orange in LogReg and Random Forest to balance class distribution, which considers class distribution when building a model.


Logistic regression for classification? I have a dataset with most columns having Boolean values and categorical values.
A sample of it is:

Name    Country approved political
bbc.com  US       true   True
stackoverflow.com US true False
Number.com US      False False

...

Based of values above, I would like to determine if other websites have been approved or not.

My questions are:

does any heat/correlation map/matrix sense with categorical variables?

Would it be possible to predict if a website is approved or not (target variable), using caterogical values?

Is there any other model that should be preferable?

Thanks

Logistic regression is a standard method of performing binary classification, which matches your task here. Categorical variables can be dealt with, depending on the model you choose.

You can see from the Scikit-Learn documentation on logistic regression, that your data only really needs to be of a certain shape: (num_samples, num_features). It might ignore the columns that are non-numerical, so you should convert e.g. strings to class IDs (e.g. integers) - see below.

Computing the correlation can make sense for categorical values, but to compute these, you need to provide numerical values; strings like "bbc.com" or "US" won't work.

You can map each of the values to a numerical value and make a new column with that data using pd.factorize like this:

df["Country_id"] = pd.factorize(df.Country)[0]   # taking the first return element: the ID values
df["Name_id"] = pd.factorize(df.Name)[0]

You don't need to do it really for the approved and political columns, because they hold boolean values, which are seen by Python as 0 and 1 for False and True, respectively.

Now you can do something like this to see a correlation plot:

import matplotlib.pyplot as plt    # plotting library: pip install matplotlib

# compute the correlation matrix
corr_mat = df[["Name_id", "Country_id", "approved", "political"]].corr()

# plot it
plt.matshow(corr_mat)
plt.show()