Crop background from Image I try to write a program to crop background from an image.

This is a sample of my training data.
I have images with and without a background. (manually cropped)

The background is always similar (changes of light and so on. Sometimes people on random places.)

The object is always on the same place but not always the same (there are different models).

Can someone give me an advice how I can solve this Problem?
What is the right way to do something like this???

This is indeed a simple problem if tried to be tackled using semantic segmentation. Semantic segmentation itself is a computer vision problem that could be understood as an extension of object detection and could be understood as follows:

Using semantic segmentation done using a network called as UNET, the model could be trained for the required image and then it can be extended to find the boundary of the required object and finally extract it. UNET architecture could be understood using the following diagram:

UNET's are generally used to create mask that could be XOR with the actual image and the background of the image could be subtracted easily.

Completely explaining image segmentation using UNET or any other technique falls beyond the limit of the answer and thus a better explanation could be found in this article

If you want the practical implementation/codes of the same, they could be found here on kaggle kernels of the following contests:

TGS Salt Identification Challenge
Carvana Image Masking Challenge


Retail Store Testing I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores).

Any suggestions for a good book/blog/any source where I could learn about this. The challenges I see are - the sample size is usually one or 2 stores for the test so how do you do statistically significant testing with such a small sample; test stores or control stores are matched 1-1 but often not selected randomly. Any guidance in learning about the data science in this area is greatly appreciated.

Thanks!!

"The New Science of Retailing: How Analytics are Transforming the Supply", Book by Ananth Raman and Marshall Fisher has some very good information of data science applied to Retail. Although, they don't address your case directly, they do have a chapter on store level execution that might help, and another chapter on assortment planning which might have relevant information.


Magnification factor in image classification If a CNN is trained on images focusing on an object, will it also recognize when multiple such objects are present in the image?

For example can a network trained on single flower images also identify image having many such flowers in a bunch?

Is the answer same if situation is reversed? That is, can a CNN trained on bunch of flower images also identify single flower image?

This question greatly depends on the pixel-space scale of the objects, but if the scale of the objects is similar and enough data are provided to curb overfitting, then yes, it should work both ways.

The Caveats:
A Neural Network is merely a glorified (hierarchical) Template Matcher. If the visual appearance of multiple objects in an image is strongly correlated with the learned template, those objects will each get a high response from that convolutional kernel. Convolutional templates can achieve some degree of scale-invariance through stacking multiple layers and using a downsampling strategy such as Max Pooling, but virtually all convolutional neural networks are practically quite limited as to the scale of features that they will respond to. As a simple example, a neural network consisting of two convolutional layers of 3x3 spatial kernels each with padding of 1 separated by a 2x2 Max Pooling downsampling layer would have a receptive field of at most 8x8 pixels per output location (3x2+2), where the last 3 is the last layer's kernel width, the middle 2 is the downsampling factor, and the last 2 is the input padding reachable from the first kernel's output. Trying to classify any objects larger than this would result in a failure to match the learned template, so in fact CNNs are sensitive to the scale of their inputs, but due to weight sharing across spatial locations, a strong response trained from a single item should activate for multiple items of similar scale and (to some extent) vice-versa.


Confusion in applying k-fold Cross Validation to dataset I have a dataset which is already divided into 10 folds with each fold having training, validation and test sets. I'm not able to understand how to apply 10-fold cross validation on this dataset.
In general, if we want to apply k-fold cross validation on a dataset, the procedure is as follows:

In my case, the dataset is already divided into 10 folds and each fold contains validation and test sets in addition to training set. It would be helpful if someone can guide me, how to do 10-fold cross validation for this kind of dataset.

In 10 fold cross-validation, you split your dataset into 10 sections, 9 of them are for train and one for test set (there is no validation set), for example, if your dataset is 100 samples, inside a loop, in the first fold (first loop iter), the model train on 90 samples and the rest 10 are for testing the model, and loop is continued until all the dataset is used for training and testing.

for more, see here

and in python, you can implement 10 fold cross-validation using sklearn library here

Now, because your dataset is already split into 10 fold, you have two choices:

1- The easiest way is to combine your dataset into one set then using a specific library to do the 10 fold cross validation for you.

2- write code by yourself to loop over your 10 fold data, in the first iter use the first section for testing and the rest 9 for the training, in the second iter, use the second section for testing, and the first and other 8 sections for training, the loop should continue 10 times until all the data is used for training and testing.

this is the idea behind 10 fold cross validation if this not applicable for your dataset, I think 10 fold is not good in your case.


Importance Sampling in Off-policy n-step Sarsa In Chapter 7.3 of Reinforcement Learning: An Introduction by Sutton and Barto, the off-policy pseudocode has the following update equation for $Q$:

Compute importance sampling ratio:
$$ \rho \leftarrow \prod^{\min(\tau+n-1, T-1)}_{i=\tau+1} \frac{\pi(A_i \mid S_i)}{b(A_i \mid S_i)} $$
Compute truncated return
$$ G \leftarrow \sum^{\min(\tau+n, T)}_{i = \tau+1} \gamma^{i-\tau-1} R_i $$
Compute discounted estimate:

$$ \text{If } \tau+n < T, G \leftarrow G + \gamma^n Q(S_{\tau+n}, A_{\tau+n}) $$

Update $Q$

$$ Q(S_\tau, A_\tau) \leftarrow Q(S_\tau, A_\tau) + \alpha \rho [G - Q(S_\tau, A_\tau)] $$

($\tau$ is the time whose estimate is being updated, $t$ is current timestep, $n$ is $n$-step return, and $T$ is termination timestep)

I believe that the importance sampling ratio $\rho$ should only be multiplied to $G$ and not to $Q$, since $Q$ is for target policy $\pi$ and $G$ is from behavior policy $b$. In other words, this is how I see this equation:

$$ Q_\pi \leftarrow Q_\pi + \alpha \rho_{b \to \pi} [G_b - Q_\pi]$$

Am I correct in thinking that this is a typo, or did I miss something?

Thank you for your help!

It seems like someone else asked themselves the same question on Cross Validated. It makes sense for the ratio to be multiplied to G only, since it's only G that uses the behavior policy to sample the next n rewards.


How to create custom Activation functions in Keras / TensorFlow? I'm using keras and I wanted to add my own activation function myf to tensorflow backend. how to define the new function and make it operational. so instead of the line of code:

model.add(layers.Conv2D(64, (3, 3), activation='relu'))

I'll write

model.add(layers.Conv2D(64, (3, 3), activation='myf')).

First you need to define a function using backend functions. As an example, here is how I implemented the swish activation function:

from keras import backend as K

def swish(x, beta=1.0):
    return x * K.sigmoid(beta * x)

This allows you to add the activation function to your model like this:

model.add(Conv2D(64, (3, 3)))
model.add(Activation(swish))

If you want to use a string as an alias for your custom function you will have to register the custom object with Keras. It can be done like this:

from keras.utils.generic_utils import get_custom_objects

get_custom_objects().update({'swish': Activation(swish)})

This allows you to add the activation directly to layer by name:

model.add(Conv2D(64, (3, 3), activation='swish'))

For more advanced activation functions, with trainable parameters and such, it is best to implement them as a Keras Layer. Here the swish function is used in a layer, allowing beta to be learned while training:

from keras.layers import Layer

class Swish(Layer):

    def __init__(self, beta=1.0, trainable=False, **kwargs):
        super(Swish, self).__init__(**kwargs)
        self.supports_masking = True
        self.beta = beta
        self.trainable = trainable

    def build(self, input_shape):
        self.beta_factor = K.variable(self.beta,
                                      dtype=K.floatx(),
                                      name='beta_factor')
        if self.trainable:
            self._trainable_weights.append(self.beta_factor)

        super(Swish, self).build(input_shape)

    def call(self, inputs, mask=None):
        return swish(inputs, self.beta_factor)

    def get_config(self):
        config = {'beta': self.get_weights()[0] if self.trainable else self.beta,
                  'trainable': self.trainable}
        base_config = super(Swish, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

    def compute_output_shape(self, input_shape):
        return input_shape

You then would add the activation function the same as any other layer:

model.add(Conv2D(64, (3, 3)))
model.add(Swish(beta=1.0, trainable=True))


When would C become nescessary to do an analysis or manage data? I use python in my day to day work as a research scientist and I am interested in learning C.  When would a situation arise where python would prove insufficient to manipulate data?

Most of the common libraries you would use for data manipulation do actually use C (or C++ or Fortran, etc.) under the hood.
There are even libraries such as CuPy, which offers the entire NumPy API, but can run your code on a GPU. Using GPUs for speed is a much more common use case these days (in my experience), compared to writing the C/C++ version.
EDIT: here is a related answer, about which programming languages are most competitive for AI, ML, DataScience, etc.

In my opinion, you might need to "do it yourself" in one of 3 cases:
1. Speed
You need it to run faster than current libraries offer - e.g. if the clustering algorithms in Scikit-Learn are too slow
2. Memory
You need to use less memory that existing algorithms - perhaps a specific method on your Pandas DataFrame uses more memory that you have available
3. New Algorithms
You need something that is fairly fast or very low level, and no existing library offers it. I would normally suggest trying your idea first using NumPy though, before trying to roll your own binaries.


How do I assign a title to each of the histograms I have posted at the same time? fig, axes = plt.subplots(1,7, sharey=True)

df2.loc[df2.country == 'BR','date_created'].hist(xrot=90,bins=30,ax=axes[0],figsize=(20,4));
df2.loc[df2.country == 'CH','date_created'].hist(xrot=90,bins=30,ax=axes[1],);
df2.loc[df2.country == 'CL','date_created'].hist(xrot=90,bins=30,ax=axes[2])
df2.loc[df2.country == 'DE','date_created'].hist(xrot=90,bins=30,ax=axes[3])
df2.loc[df2.country == 'CO','date_created'].hist(xrot=90,bins=30,ax=axes[4])
df2.loc[df2.country == 'GR','date_created'].hist(xrot=90,bins=30,ax=axes[5])
df2.loc[df2.country == 'JP','date_created'].hist(xrot=90,bins=30,ax=axes[6])
plt.xticks(size = 12);

Using the axes variables you have, which is a numpy.ndarray of size 7

axes[0].set_title('Title for Fig. 1') #change title for first histogram
axes[1].set_title('Title for Fig. 2') #change title for second histogram
axes[2].set_title('Title for Fig. 3') #change title for third histogram


Append and replacing data in r I am writing a few lines of codes to create a data table full of NAs but don't know how to append the data of results into the data table I created while replacing the position of NAs.
Here is my data table

games<-10
game_results<-as.data.frame(matrix(NA,games,3))
install.packages("data.table")
library(data.table)
setnames(game_results,c("V1","V2","V3"),c("winner","round","winnerscore"))
game_results

You could change in this case with command

game_results$winner[1] <- "Dad"

But this copies the whole table, so i will recomend you to create an empty data in another way

game_results <- setNames(data.table(matrix(nrow = 10, ncol = 3)), c("winner","round","winnerscore"))

and assignment by reference in the way you prefer (hole column or specific cell), here instructions for both cases.


Is decision tree regression comparable to locally weighted regression I am new to decision tree method. For decision tree regression model, does it just fit a piece wise step function over data? When and why would people prefer it over some traditional regression like locally weighted (LOESS) regression?

There are two important differences between decision trees and regression:

Decision tree fit a straight a line (mean of the dependent variable for the feature space). Regression fits a sloped line (rise over run).

Decision trees typically do not predict values outside the observed range. Linear regression can predict values outside the observed range.


Identifying which known groups are the most similar or most dissimilar I have a data set of 5 groups and their associations to different interests. The data is structured as follows with 2000+ interests and 5 known groups:

      Interest     Group1     Group2     Group3     Group4     Group5
01.   Sports         10%        40%        30%       80%         65%
02.   Music          35%        60%        55%       10%         15%
03.   Fashion        80%        10%        75%        5%         25%
 .
 .
 .
1998.  Movies        70%        20%        45%       50%         65%
1999.  Books         50%        70%        35%       65%         95%
2000.  Art           90%        50%        25%       15%         30%

I would like to gain an understanding of what would be the best way to determine which groups are closest to each other and which ones are the most different from each other in terms of interests.

I am leaning towards cluster analysis, however the structure of my data is throwing me off.

Any suggestions would be greatly appreciated.

You only have five groups, so full blown clustering is probably not a good idea here, but looking at similarity scores between the group vectors may be insightful. An easy one to try at first would be cosine similarity, which essentially measures the angle between each of your group vectors:

similarity = $cos(\theta)$ = $\frac{\mathbf{A} \cdot \mathbf{B}}{||\mathbf{A}||_2 ||\mathbf{B}||_2}$

where $\mathbf{A} \cdot \mathbf{B}$ gives the dot product of two vectors, and $||\mathbf{A}||_2$ is the magnitude of a vector $\mathbf{A}$.

All of your vector values are positive, so the result will be in the range $(0, 1)$. The closer this value is to $1$, the more similar the vectors. $0$ means they are completely decorrelated (vectors are orthogonal). You can compare the cosine similarities of each pair of groups to work out which ones are most similar/dissimilar.


Reading values from a column into a variable and then correlating using Python I want to be able to correlate values from various IDs, where the date is the same with one another.

The data looks something like this;

ID      Time(secs)   Date
AAAA    1            01/01/1990
AAAA    6            02/01/1990
AAAA    5            03/01/1990
AAAA    2            04/01/1990
AAAA    4            05/01/1990
BBBB    2            01/01/1990
BBBB    4            02/01/1990
BBBB    6            03/01/1990
BBBB    3            04/01/1990
CCCC    3            01/01/1990
CCCC    4            02/01/1990
CCCC    1            03/01/1990
CCCC    6            04/01/1990
DDDD    7            01/01/1990
DDDD    4            02/01/1990
DDDD    5            03/01/1990
DDDD    3            04/01/1990

I want to find the correlation coefficient between each combination of these IDs, where there are matching dates. N.B. Not all of the IDs have the same date;

ID             CorrCoef
AAAA>BBBB      ????
AAAA>CCCC
AAAA>DDDD
BBBB>CCCC
BBBB>DDDD
CCCC>DDDD

I think I need to feed the data from each ID into a variable and then run the following;

data1.corr(data2)


How to evaluate distance in k-means clusters? I try to use k-means clusters (using SQLserver + R), and I was wondering how we could estimate a distance correctly.

For instance, if we consider Euclidean distance form the center of the clusters, what happen if we have, for the same dataset, clusters of different sizes ? a "normal" point in a big cluster will have a distance higher than an "outlier" point in a little one.

So:

Is it relevant to center / scale euclidean distance on each cluster ? (and then consider outliers as the ones with the highest scaled distance)
Are there other kind of distance to consider ?

There are several important points to keep in mind in considering your questions:

You should always normalize or standardize your data before applying k-means clustering. This is true of most other clustering algorithms also. If you are clustering in more than one dimension, the distance metric is meaningless unless each dimension has the same weight, so normalization is essential.  Imagine clustering people by body weight and income. Without normalization the results will depend on whether you think in pounds and dollars, kilograms and pesos, or moles and euros. The lack of normalization introduces non-determinism.
Strictly speaking, the stability of the k-means algorithm has been shown for Euclidean distance metrics and there is no assurance of convergence with other distance metrics.  More practically, most sensible metrics attain convergence and its not much of an issue, but its worth putting that warning out there.
k-means isn't a clustering algorithm that readily lends itself to statistical analysis within the cluster. Every point in the space is a member of one of the k clusters regardless of how much of an outlier the point is. There are other clustering methods that are more adept at finding and ignoring outliers.  DBSCAN is one such algorithm that is very good and finding clusters and ignoring noise.

Now, answering your questions:

Is it relevant to center / scale euclidean distance on each cluster ? (and then consider outliers as the ones with the highest scaled distance)

Yes, you can certainly do this. Combining k-means with outlier detection is certainly possible but is probably not the most elegant or efficient algorithm.  It kind of sounds like poor-mans's DBSCAN.  Euclidean distance works fine, but just do a second set of normalizations using the centroids and the standard deviation of the cluster.

Are there other kind of distance to consider?

There are lots of other metrics that are useful for many different reasons.  As stated, the k-means convergence proofs hold only for Euclidean distance. For outlier detection Euclidean seems best, but there may be cases where Cosine Similarity metrics could be useful. People may suggest L1 (Manhattan) distance metrics, but I find this is only useful when there is significant linear dependence in your data, which can be remedied with dimensionality reduction.

Short answer:

Give it a try as Euclidean should work fine, but also take a look at clustering via DBSCAN, which has outlier detection built into it.

Hope this helps!


Time-series prediction: Model & data assumptions in AI/ML models vs conventional models I was wondering if there was a good paper out there that informs about model and data assumptions in AI/ML approaches.

For example, if you look at Time Series Modelling (Estimation or Prediction) with Linear models or (G)ARCH/ARMA processes, there are a lot of data assumptions that have to be satisfied to meet the underlying model assumptions:

Linear Regression

No Autocorrelation in your observations, often when dealt with level data (--> ACF)
Stationarity (Unit-Roots --> Spurious Regressions)
Homoscedasticity
Assumptions about error term distribution "Normaldist" (mean = 0, and some finite variance)
etc.

Autoregressive Models

stationarity
squared error autocorrelation
...

When dealing with ML/AI approaches, it feels like you can throw whatever you like as an input (my subjective perception). You are satisfied with the result as long as some prediction/estimation error measurement is good enough (similar to a high, but often misleading R²).

What assumptions have to be satisfied for an RNN, CNN or LSTM model that find application in time-series prediction?

Any thoughts?

ADDED

Good Article describing my question/thoughts.
Medium Article discussing model assumptions + tests, but not in the context of more advanced models
I read the 100-page ML Book- Unfortunately almost no content about model assumptions or how to test for them.

Great question @Maeaex1 !

First of all why do we even need assumptions in models (generally speaking
)?

Well we can express a task as an optimization one. And in order to converge to optimal solution, under certain constraints, we need to satisfy certain assumptions.

Regarding DNN (deep neural networks) and mathematical theory behind it , convergence assurance is given with famous Universal Approximation theorem that states that every smooth function can be estimated given enough parameters.

Caveat just because we can do it in theory does not mean its possible. For example approximating a function that generates random numbers would require infinite recources

But what about non-smooth functions (such as Time-series) one?

Well the TL;DR of the DNNS FOR NON-SMOOTH FUNCTIONS is that for a special set of piecewise smooth functions "convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions"

What is piecewise smooth function? function whose domain can be partitioned locally into finitely many "pieces" relative on which smoothness holds, and continuity holds across the joins of the pieces.

Ok but WHY can a DNN approximate these types of functions?

" The most notable fact is that DNNs can approximate non-smooth functions with a small number of parameters, due to activation functions and multi-layer structures. A combination of two ReLU functions can approximate step functions, and a composition of the step functions in a combination of other parts of the network can easily express smooth functions restricted to pieces. In contrast, even though the other methods have the universal approximation property, they require a larger number of parameters to approximate non-smooth structures"

So to conclude: there is a mathematical theory that insures approximations of a set of certain non-smooth functions using DNN. So if we have non-smooth function that satisfies these constraints, we can find an optimal architecture and get optimal convergance rates.


Preventing fitting Regression CNN to the mean when dataset has only few outliers I am trying to train a CNN for regression on a dataset where most of the points lie around a similar output value. There are however a few outliers that are very important but they are less represented and the trained network thus tends to predict all output values close to the mean of the whole dataset (underfitting). This leads to a somwhat small error (and good precision) because the vast majority of points lie in that range, but the error is way higher for points even slightly outside the "normal" case.

But since this regressor would be most useful to predict the output for outliers (quality-control use case), it's currently pretty much useless.

Is there a way to prevent this kind of behavior and have a CNN that is giving a greater weight to outliers and extrema, in order to avoid underfitting?

To some extent, the Random Forest method, although much better at predicting the output for outliers, still exhibits a higher error for points in the extrema, while the error around the mean is very small. The "low" points are predicted with too high a value, and the "high" points are predicted with too low a value (each time closer to the mean). So any idea for that case would be great too!

Thanks a lot

I'm not sure what you are trying to do. CNNs are good for image-related tasks, as they attempt to extract spatially local features from the input data. They can be used for regression problems but only as long as the input resembles an image.

Random Forests on the other hand are bad at image-related tasks, unless some sort of feature extraction has been performed beforehand.

Does your dataset consist of images? If not don't use CNNs!


Why yolo4 pytorch re-training loss seems high as like first time training? I had a setup a yolo4 pytorch framework in google colab by cloning git clone https://github.com/roboflow-ai/pytorch-YOLOv4.git.
I generated checkpoints by giving training. As we need more robust training model, I given training again with assigning pretrained checkpoints but it seems loss started with high value as like first time training.
Code is for training !python train.py -b 2 -s 1 -l 0.001 -g 0 -pretrained ./Yolov4_epoch100_latest.pth -classes 1 -dir ./train -epochs 100.
Not sure if my pretrained checkpoint is used in 2end training? If it is used then why second time in starting training loss value seems high like first time training? Please share your thought if you have any

Apparently, only the lower layers weights of the pretrained model are being updated. to change it open models.py and scroll down to the class definition of Yolov4.
As one can see, only down1-down5 and neck layers are being updated by the pretrained model weights while the head is starting fresh. you should change this code section to something similar to:
if yolov4conv137weight:
        self.neck = Neck()
        **self.head = Yolov4Head(output_ch)**
        _model = nn.Sequential(self.down1, self.down2, self.down3, self.down4, self.down5, self.neck, **self.head**)
        pretrained_dict = torch.load(yolov4conv137weight)

        model_dict = _model.state_dict()
        # 1. filter out unnecessary keys
        pretrained_dict = {k1: v for (k, v), k1 in zip(pretrained_dict.items(), model_dict)}
        # 2. overwrite entries in the existing state dict
        #print("update weights")
        model_dict.update(pretrained_dict)
        _model.load_state_dict(model_dict)
    # head
    **else:
        self.head = Yolov4Head(output_ch)**

Run again and observe the expected loss (hopefully).
Good luck!


Training Encoder-Decoder using Decoder Outputs I am trying to build an encoder-decoder model for a text style transfer problem. The problem is I don't have parallel data between the two styles so I need to train the model in an unsupervised setting.

Some papers I have seen use an auto-encoder to train the encoder and decoder components separately. By setting the problem as an auto-encoder, they can train the decoder by passing the target sequence (equal to the input sequence) into the decoder. (Here are some examples, https://arxiv.org/pdf/1711.06861.pdf, https://arxiv.org/pdf/1804.04003.pdf)

Instead of an auto-encoder, I would like to know if it's possible to train a decoder by feeding its predictions at time, t-1, into the input at time-step t. I would pass the generated output into a classifier to check the style and to obtain a training signal. Is this sensible and what are the pros / cons of doing so? Thanks.

I would like to know if it's possible to train a decoder by feeding
  its predictions at time, t-1, into the input at time-step t.

Yes, it is possible to do it.

But I don't see why you would do it.

Υou will have accumulated error propagated and amplified in every new prediction, making your prediction to diverge from the ground truth sooner or later.


What is the range of values of the expected percentile ranking? I'm currently reading

  Hu, Koren, Volinsky: Collaborative Filtering for Implicit Feedback Datasets

One thing that confuses me is the "expected percentile ranking", an function the authors define to evaluate the goodness of their recommendations. They define it in the Evaluation methodology on page 6 as:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}$$

where $u$ is a user, $i$ is an item (e.g. a TV show), $r_{ui} \in [0, \infty)$ is the amount how much user $u$ did watch show $i$. $\text{rank}_{ui} \in [0, 1]$ is the percentage rank of item $i$ for user $u$. For example, it is 0 if for user $u$ the item $i$ has the highest $r$ value and 1 if the item $i$ for user $u$ has the lowest $r$ value.

I'm not super sure if I understood it correctly.

The authors write that lower values of $\overline{\text{rank}}$ are more desirable and for random predictions would lead to an expected value of $\overline{\text{rank}}$ of 0.5.

Examples

Assume there is only one item. In this case $\text{rank} = 0$. Makes sense, as there cannot be any predictions.
Assume there is only one user and two items with $r_{1,1} = 1$ and $r_{1,2} = 2$. Then:

$$\overline{\text{rank}} = \frac{1 \cdot \text{rank}_{1, 1} + 2 \cdot \text{rank}_{1, 2}}{1+2}$$

This means $\overline{\text{rank}} \in \{2/3, 1/3\}$.

If there is only a single user and all $|I|$ values of $r_{ui}$ are the same, then $\overline{\text{rank}} = \sum_{ui} \text{rank}_{ui} = \frac{|I|}{2}$

Questions

Is my understanding of the metric correct? Especially my last example and the statement by the authors that $\overline{\text{rank}} \geq 50\%$ indicated an algorithm is no better than random seem off.
What is $t$?

What is $t$?

It means observed $r_{ui}$ in the one-week test set (page 6-left).

  Is my understanding of the metric correct?

First two examples are correct. Assuming user-item relation $r_{ui}^t$ is constant $a$ for all items in the test set, and predicted ranks are uniform across $[0, 1]$, then, the third one would be:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}=\frac{\sum_{u,i} a \text{ rank}_{ui}}{\sum_{u,i} a}=\frac{1}{|I|}\sum_{u,i} \text{ rank}_{ui}=\frac{1}{|I|}\frac{|I|}{2}=\frac{1}{2}$$
This makes sense. Items are identical to the user, therefore no model can do better than random guessing, since there is no observed preference to help the model favor one item over the other. Of course, another assumption here is that training (4 weeks) and test (next week) sets are from the same distribution.


How to filter out a portion of the text data with re.match or any other regex method with Python? I have the following data :

Node : WATER-7609-WR1

IP Address : 172.20.150.81

Severity : 5 - Critical

ServerSerial : 50505223

SNSupportGroup : UC-WAN

Node Location : CO 160 WATER 8 FL RCV

Event Details : Unknown Not Link Down or Up trap on Serial Channel:  Unknown Severity: 4
NOTE:Configuration Item is not found in CMDB. Please update Incident with related Configuration Item

I am interested in all text after "Event Details :",
 I have used the the following function :

def eventdetails(text):
   match =re.search(r'Event Details :[^.]*',text)
   return match

print(eventdetails(dfnetcool_inc["Description"][3]))

output: <_sre.SRE_Match object; span=(159, 292), match='Event Details : Unknown Not Link Down or Up trap >

--> Am not getting all of the text after Event Details. Perhaps there is an EOL charecter - how do I get remainder of the text, request your help, TIA!

nice that you solved it. This would also work:

import re

def eventdetails(text):
   match =re.search(r'Event Details :(.*)',text, flags=re.S)
   return match

result = eventdetails(text)
print(result.group(0))


after cross-validation, what measure of 'stopping' training should i use when training final model on whole dataset? I have performed 10 fold cross validation for performance estimation on a keras model. I now want to train a final neural network using ALL data for the FINAL model that will be deployed.
Given that this is the recommendation (to generate final model on all data), and I now no longer have any holdout data, conceptually, could I also average epochs taken to train during cross validation over the 10 folds to determine an epoch range that will not result in overfitting? or should i simply run 'x' epochs, overshoot and then simply determine the plateu?
is there a general rule here?
or... should I still have a validation split during training but no test set. This option makes no sense to me as this is exactly what cross validation was for.
thanks!

If you use the scikit-learn GridSearchCV class (from sklearn.model_selection) together with the scikit-learn wrapper in keras, you can get your final model refit on the whole training set directly via the best_estimator_ attribute (i.e. the model instanced with the best hyperparms found in the CV process) already refit with the whole training dataset.
I have seen some source which make a "manual" retrain on the whole training set, but it is not necessary as scikit-learn already let's you accessing the refit model on the whole set.
Interesting links:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
for using the keras wrapper for scikit: https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/


Setting class-weights in a batch in where a certain class is not present I'm handling a high imbalanced dataset, thus, I'm weighing the loss function in order to penalize the misclassification of the minority classes, I set the weights in each batch as follows:

w[i] = num_total_instances / num_instances_of_class[i]

Where 'i' goes from 0 to N-1 (for N classes), 'num_total_instances' is the total number of instances in the batch, and 'num_instances_of_class[i]' is the number of instances of class 'i' in the batch.

The problem is, as I'm doing it on each batch, It may occur that, in a certain batch there is no instance of class 'j', thus w[j] = inf. ¿What weigh should I set for class 'j'? At this moment, I'm setting this w[j] to 1E6

I know that there are other approaches to fight against imbalanced datasets and I also know that I can pre-compute the weights for the whole dataset and use them (fixed) cross all batches, but, I'd like the specific answer for this question, whereas it makes sense. For sure, it will be well received other suggestions for the calculation of the weights.

I don't think that the weight you use matters, whether you set it to $10^6$ or $10^8$. This is because, as examples of that class are not in the batch, the loss function for that batch will not have any of those examples contributing, so that weight won't appear. If something does not appear in the loss function, it is not used at all. For that reason, it doesn't really matter what weight to use.


Machine learning - Algorithm suggestion for my problem using NLP I am looking for a machine learning algorithm for my problem.

I have a set of sentences like,

["The cat in the hat disabled", "A cat is a fine pet ponies.", "Dogs and cats make good pets.","I haven't got a hat."]

and the search-words like,

["cat","hat"]

I want to convert my sentence list and search-words to a vector space and find matching score between my sentence list and search-word list.

the type of output I am expecting is,

[("The cat in the hat disabled",0.9), ("A cat is a fine pet ponies.",0.5), "(Dogs and cats make good pets.",0.6),("I haven't got a hat.",0.49)]

Please suggest a machine learning algorithm for my task, if possible please share a reference link.

let me know if you have any queries or need more information.
I am currently using semanticpy for this https://github.com/josephwilk/semanticpy

I am getting key-error for many search-words. its not performing stemming and lemmatization for the sentence list but only performing for the search-words list.

Please help on this.

There is a really good video about this topic from a PyCon in 2016. There is a pretty in-depth description on how to vectorize your sentences as well as make predictions based on those vectors.

I think that would greatly help you out. That is what I used when I was learning about how to perform sentiment analysis.


"other" class in Image classification In the MNIST dataset, you have 10 defined classes, one for each digit. But you don't have a "not a digit" class. It seems that most image classification datasets are the same. But in a business setting, for a production model, you could certainly get invalid images that don't correspond to any of the defined classes.

So let's say you were to create a handwritten digit image classifying model to be used in a real-world project. If you don't have a "not a digit" class, then if somebody submits a picture of the letter "M", then it will incorrectly be classified as one of the 10 digit classes.

So, in this example, should you define a "not a digit" class and train the model on a set of images that the model may be expected to receive that have nothing to do with handwritten digits such that invalid images are correctly classified as "not a digit"?

You can have that but I recommend something else for your case. Suppose that you have 10 classes of digits that they are mutually exclusive. Consequently, your output vector will always be on-hot encoded. In a simple case, your output vector should contain ten entries for each class. To make your model distinguish between those inputs which are digits and those which are not, do the following:

Increase the size of output to eleven. Provide data-set that contains eleven-entries labels. the first entry corresponds to the existence of a digit. If it is a digit the first entry should be one and next 10 entries should be activated for the according to the value of the digit. So, your labels will not be one-hot encoded. If the input is not a digit the first entry of the output should be zero and other entries are don't cares.


should I shift a dataset to use it for Time series regression with RNN/LSTM? I'm seeing this tutorial to know how to use LSTM to predict time series data and I noticed that he shifted the target/labels up so that the features are all in time t but the target is t+1

so my simple question is, should we always do this when working with time series data?

I have a time series data and I want to use it to build a regression model with RNN. I used feed forward NN first but it didn't work good enough so I decided to use RNN/LSTM. My question is should I let the dataset as it is now where every row have features and target at time t or should I shift the target column so that the features would always be at time t but the target is always shifted(t+1) ahead?

Yes, you are making prediction, which means using historical features(such as at time t) to predict the target in the future(say t+1). In many cases, you could use past few timestamps' features(say t-n,t-n+1,...t) to predict. Preprocess your data in the way (X_historical,y_future) in case of data leakage.


How can I handle a column with list data? I have a dataset which I processed and created six features:

['session_id', 'startTime', 'endTime', 'timeSpent', 'ProductList',
   'totalProducts']

And the target variable is a binary class (gender).

The feature 'productList' is a list:

    df['ProductList'].head()
Out[169]:
0    [13, 25, 113, 13793, 2, 25, 113, 1946, 2, 25, ...
1    [12, 31, 138, 14221, 1, 31, 138, 1979, 1, 31, ...
2                               [13, 23, 127, 8754, 0]
3    [13, 26, 125, 5726, 2, 26, 125, 5727, 2, 26, 1...
4           [12, 23, 119, 14805, 1, 23, 119, 14806, 0]
Name: ProductList, dtype: object

Now, it is obvious that I can't use this feature as it is. How do I handle this feature? I can explode the list and create a row for each list item, but will it serve my purpose?

Update: I applied OHE after exploding the list, and it results in 10k+ columns, which my GCP instance and my computer can't handle; when applying PCA.

PS: There are over 17,000 unique products.

You basically want to create a column for each product bought, as the presence or absence of each in the list is a feature in itself. See Hadley Wickham’s definition of tidy data.

That being said, you seem to have a large number of products. To avoid the course of dimensionality, what I would do is to take your binary bought/not features (or count values might be even more effective if you have that data) and do dimensionality reduction to get a reasonable set of features. Latent Dirichlet Allocation (which comes from topic modelling), PCA, t-SNE, our UMAP are all easy to implement and worth trying. PCA is the least sophisticated and the fastest to run and would be a good baseline.

When you have your smaller list of features, you might want to try using a classifier that’s further selects the most relevant features, like gradient boosted trees.


scale_pos_weight using XGBoost's Learning API I see it is possible to add a weight for unbalanced problems in XGBoost's Scikit-Learn API through scale_pos_weight. Does it have an equivalent in the Learning API? If not,

is there a reason behind this?
Could this corrective factor/weight also be somehow implemented using the learning API?

Yes, you can use scale_pos_weight in the native python API; it goes in the params dictionary.  E.g.,

params = {'objective': 'binary:logistic',
          'scale_pos_weight': 2.5}
model = xgboost.train(params, dmat)

https://xgboost.readthedocs.io/en/latest/parameter.html#parameters-for-tree-booster
https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/speedtest.py


Reinforcement Learning in NLP for chatbots Is anyone aware of any successful implementation of reinforcement learning for NLP. I am looking to for chatbots which can learn automatically.

Tried searching internet but found very few articles like Reinforcement Learning For Natural Language Processing - Medium or papers like A Survey of Reinforcement Learning Informed by Natural Language.

But none of them provides a robust code which shows that it is working. Kindly suggest.

That's because using NN for chatbots have proven super challenging. Basically once you type a query the NN (whether its NN or RL) has to tag it to a particular intent based on which u give a templatized response ( generating a human like response goes into even more complex territory of natural language generation)

The issue with NN like LSTM , GRU etc is that they don't understand "turns" in a conversation. So if u type in a query one after the other the bot will treat them as separate sentences and try and tag them individually. But a human would form a context based on the whole conversation and can handle turns. There's some labelled data for a few verticals that shows turns in the conversation but it's just not enough

Coming to RL , it seems like a good fit for handling conversation since the current state (sentence) totally depends on the prior state. But what kind of reward function do you have in mind ? How, based on the action of the agent, will the network learn a policy to identify a "conversation"? When will u stop the episode ? When a conversation ends ? Or a fixed number?  Me thinks these are still hard problems to solve since convergence is a challenge in such use cases

If someone has cracked it, they most likely will not publish anything since a true researcher will only publish stuff that is reproducible or they are just being secretive :)


Can't load custom Keras metrics using mlflow.pyfunc I have a DNN in Keras, which includes a custom metric function and which I want to pipeline with some SKlearn preprocessing. I further want to persist the model using MLFlow for easy deployment. The requirement to pipeline with sklearn means that I can't use the mlflow.keras versions of .log_model() and .load_model(), and have to instead use the mlflow.pyfunc versions, which is fine.

Saving the model seems to work fine, but when I try to use mlflow.pyfunc.load_model() to reimport the saved model I get this error message (full stack trace at link):

  ValueError: Unknown metric function:custom_mse

To try and make sure that the custom function makes its way through to MLFlow I'm persisting it in a helper_functions.py file and passing that file to the code_path parameter of .log_model(), and then attempting to import that function in .load_context() before using keras.models.load_model() to reimport the saved keras model.

helper_functions.py:

import keras.backend as K

def custom_mse(y_true, y_pred):
    return K.mean((y_pred - y_true) ** 2)

The PythonModel I'm trying to persist is this:

class ProductRecommender(PythonModel):

    def __init__(self, pipeline):
        self.pipeline = pipeline

    def load_context(self, context):

        from helper_functions import custom_mse

        self.keras_model = keras.models.load_model(context.artifacts["keras_model"], custom_objects={'custom_mse', custom_mse})
        self.sklearn_preprocessor = joblib.load(context.artifacts["sklearn_preprocessor"])

        self.sklearn_model = KerasModelRegressor(self.keras_model, epochs=5, validation_split=0.2)

        self.pipeline = Pipeline(steps=[
            ('preprocessor', self.sklearn_preprocessor),
            ('estimator', self.sklearn_model)
        ])

    def fit(self, X, y):
        self.pipeline.fit(X, y)
        self.pipeline.named_steps.estimator.model.save('artifacts/keras_model.h5')
        joblib.dump(pr.pipeline.named_steps.preprocessor, 'artifacts/sklearn_preprocessor.joblib')

    def predict(self, context, X):
        return self.pipeline.predict(X)

Note I import custom_mse function from the helper_functions module and pass it as part of custom_objects to keras.models.load_model()

Here's the mlflow.pyfunc.log_model() call:

with mlflow.start_run() as run:

    run_id = run.info.run_id

    conda_env = {
        'name': 'mlflow-env',
        'channels': [
            'defaults',
            'anaconda',
            'conda-forge'
        ],
        'dependencies': [
            'python=3.7.0',
            'cloudpickle',
            'keras==2.2.5',
            'joblib==0.13.2',
            'scikit-learn==0.20.3'
        ]
    }

    artifacts = {
        'keras_model':'artifacts/keras_model.h5',
        'sklearn_preprocessor':'artifacts/sklearn_preprocessor.joblib'
    }

    mlflow.pyfunc.log_model(
        artifact_path='Model',
        code_path=['artifacts/example_sklearn_wrapper.py', 'artifacts/helper_functions.py'],
        python_model=pr,
        conda_env=conda_env,
        artifacts=artifacts
    )

What's happening here? Why isn't keras seeing my custom_mse function?

Long story short is to use cloudpickle instead of joblib or pickle to dump thing to disk, and this all works much more cleanly.


Tidy data in panda dataframe There is a quite famous article by H. Wickham, Tidy data, where he defines a certain type of cleaned data and calls it (dataframe-)tidy, and illustrates in on several example using R. At the end, he compares his own definition of dataframe-tidy to other possible ways of achieving tidiness and mentions array-tidy in the following paragraph (but does not give any further explanations or examples):

  Fortunately, because there are many efficient tools for working with high-dimensional arrays, even sparse ones, such an array-tidy
  format is not only likely to be quite compact and efficient, it should
  also be able to easily connect with the mathematical basis of
  statistics. This, in fact, is the approach taken by the pandas Python
  data analysis library (McKinney 2010).

What does array-tidy data mean and why does he imply that this is somehow the default for the pandas library in Python?

TL;DR: Array-tidy is extending tidiness across multiple data frames that are indexed by an additional shared dimension, such as time.

From what I understand, array-tidy and dataframe-tidy refer to the underlying data models that are used to achieve your tidy data. It seems to be a reference to how plyr and pandas differ in their data structure logic

Pandas was developed around the idea of panel data, which is your typical two dimensional data frame of observations and variables, but with samples accumulated over a period of time. Pandas name even stems from this particular type of data (panel data structures, per this McKinney presentation and the pandas docs). The inclusion of a third dimension, thinking about your data over time, seems to have informed some thinking around how pandas would evolve. I recommend checking out the slides (specifically the one titled "The pandas killer feature: indexing").

Relating this back to the Tidy Data paper, creating multiple data frames across an additional dimension (like you would with a Panel in prior versions of pandas) seems to qualify as the "high-dimensional arrays" Wickham discusses in the section you quoted.

I hope this helps!


Which method of NLP is this? I have been searching for 2 weeks and I got no where so far.

There is  a list of diseases

Diabetes
Heart Transplant
Fingertip amputation
Injury by sharp tools
.
.
.

and My dataset is a list of medical text reports.

the training dataset has diseases that can be generated from each record

example that I made

  This patient has suffered a cut while using his Carving Chisel and led
  to losing the fingertip therefore we had to operate to sew
  the tip.....

from this report we get these diseases

1- Injury by sharp tools
2- Fingertip amputation
3- Sewing injury

another report results may have 3 or less or more diseases

I have searched a lot
I found many examples about NLP classification

where a text will be classified into Sports, Politics, Culture, Science, etc.

I found NER where person names, locations, dates, etc can be extracted from a text.

But did not find anything for a single text could have multiple values (similar to my dataset)

I dont know where to start.

Could anyone please help me finding what is the name of this method of extracting list of issues from a text?

Edit

What else do I need to exclude the negations, if the report says

  "This patient has stomach problem but not diabetes "

How can I make AI understand there is a negation (NOT) before diabetes so it should not be included?

So the result will be

stomach problem

as diabetes should be excluded because of the negation word

You might want  to look at Multi-Label-Classification . If you have sufficient number of samples as your training data, you can build a model that can predict more than one label for a test sample. You can find more about the implementation at the sklearn page for the same here .


How to rotate the plot and find minimum point? I want to rotate the below curve to 45 degree and then find the minimum point.

For this, I have tried with below code:

def rotate_vector(data, angle):
    theta = np.radians(angle)
    co = np.cos(theta)
    si = np.sin(theta)
    rotation_matrix = np.array(((co,-si), (si, co)))
    return np.matmul(rotation_matrix, data)

rotated_vector = rotate_vector(data, -45)
elbow = rotated_vector.min()

But what I get is this curve:

Please refer to the original post.

The problem was the np.radians(angle). I corrected that.

Edit: The problem was not the radian conversion but the different scales of the x- and y-axis.
45 degrees rotation was simply not enough.

Let's use a modified version of the code you posted:

def rotate_vector(data, angle):
    # make rotation matrix
    theta = np.radians(angle)
    co = np.cos(theta)
    si = np.sin(theta)
    rotation_matrix = np.array(((co, -si), (si, co)))
    # rotate data vector
    rotated_vector = data.dot(rotation_matrix)
    # return index of elbow
    return rotated_vector

Now, we define a square as test-data:

data = np.array([
    [1, 0], [2, 0], [3, 0], [4, 0], [5, 0],
    [0, 0], [0, 1], [0, 2], [0, 3], [0, 4],
    [5, 1], [5, 2], [5, 3], [5, 4], [5, 5],
    [0, 5], [1, 5], [2, 5], [3, 5], [4, 5],
])

plt.scatter(data[:, 0], data[:, 1])

Plotting the data gives us the following figure:

Now, we rotate that vector using the function from above:

rotated_data = rotate_vector(data, 45)

plt.scatter(rotated_data[:, 0], rotated_data[:, 1])

If we plot the rotated data, we get the following figure:

So it does work. However, if we want to rotate the graph that you posted to find the minimum, we need to remember that the scales of the two axes are different. That means the graph may look like a 45° rotation would be enough while actually, it is not.

I corrected the original post again. Also, I added a method to find an approximate rotation angle to account for the different axis scales.


Conda Verification Failed I was trying to install jupyter package for anaconda in my current environment but constantly getting the following error.

  Preparing transaction: done Verifying transaction: failed

  CondaVerificationError: The package for ipython located at
  D:\Anaconda\pkgs\ipython-7.1.1-py36h39e3cac_0 appears to be corrupted.
  The path 'Lib/site-packages/IPython/lib/tests/test.wav' specified in
  the package manifest cannot be found.

  CondaVerificationError: The package for notebook located at
  D:\Anaconda\pkgs\notebook-5.7.2-py36_0 appears to be corrupted. The
  path
  'Lib/site-packages/notebook/static/components/MathJax/extensions/a11y/invalid_keypress.mp3'
  specified in the package manifest cannot be found.

I have Googled many Q&A for this problem, mostly I found:

conda clean --packages --tarballs
conda clean --all

After trying both I tried the installation again, but none worked...

Any solution for this?

It seems that problem was anyhow with conda installation.
I tried pip installation for jupyter and it worked fine.

python -m pip install jupyter


XGboost and regularization Does the XGBClassifier method utilizes the two regularization terms reg_alpha and reg_lambda, or are they redundant and only utilized in the regression method (i.e., XGBRegressor)?

I think that some hyperparameters in XGBoost could have no effect on specific methods (e.g., scale_pos_weight in XGBRegressor).

XGB uses the two kinds of regularization in both classification and regression; each leaf is a continuous score, these scores added together for the final prediction (of log-odds in the classification case), so penalizing the weights makes sense in either setting.
See also L1 & L2 Regularization in Light GBM

But yes, some hyperparameters (scale_pos_weight) seem to be vestigial:
What does xgb's scale_pos_weight parameter do for regression?
https://discuss.xgboost.ai/t/scale-pos-weight-for-regression/218/10


How to output my uploaded text in Shiny in R at the position of the screenshot? I would like to position a long text on the right side of this Shiny app, as shown on the screenshot below.
    library(quanteda)
    library(shiny)
    library(tm)
    library(tidytext)
    library(tidyverse)
    library(shinydashboard)
    library(shinythemes)

    war <- readLines("war.txt")

    war_corpus <- corpus(war)

    sentences <- tokens(war_corpus,what="sentence")

    make_sentences <- function(word) {
      grep(word,sentences,value=TRUE)}

    make_sentences("prince")

    ui<- shinyUI(fluidPage(

  # Application title
  mainPanel(
    img(src='image.jpg', align = "right"),

    #titlePanel(title=div(img(src="image.jpg"))),

    fluidRow(HTML(" Search Bar")),
    #fluidRow(HTML(" Date: 06-29-2020") ),

    fluidRow(
      br(),
      p("User Name : ")),
    br(),
    br(),

    fluidRow(HTML("Enter a word.Click \"Next words\" after") ),
    fluidRow( p("\n") ),

    # Sidebar layout
    sidebarLayout(

      sidebarPanel(
        textInput("inputString","Enter a word here",value = " "),
        submitButton("Next words")
      ),

      mainPanel(
        h4("Are you looking for this ?"),
        tags$style("#mytext {white-space: pre-line;}"),
        verbatimTextOutput("mytext")

      )
    )
  )))

    server <- function(input, output, session) {
      output$mytext <- renderPrint({

        sentences <- make_sentences(input$inputString)
        length(sentences)
        cat(paste0(1:length(sentences),"  ",sentences,sep= '\n'))
      })
    }

here is the rcloud link of the project to experiment with : https://rstudio.cloud/project/1497912

In your code you use the Sidebar layout which allows only two columns, instead you should use the grid layout to position your elements where you want them, see https://shiny.rstudio.com/articles/layout-guide.html.
You probably need something like this (not tested):
ui<- shinyUI(fluidPage(
  fluidRow(
    column(3,
      # content left
    ),
    column(5,
      # content centre
    ),
    column(4,
      # content right
    )
  )
)


How can I split an image into rectangles? I have a labelled form to which people will add their name and a series of numbers. They will then take a picture of the form. Like so:

I can get decent results by simply sending this to AWS Rekognition

but I feel like it would do even better if I was able to split the image into parts before sending it to the OCR API. What techniques can I use to accomplish this?

I can modify the form for better results if that helps. Maybe make borders more or less dark? Whatever is needed.

AWS Rekognition works very well for detecting typewritten text but not so well for detecting hand written symbols. You can see this is true in your example. What I would do is detect all the boxes associated with the words Problem. Then you can extract a box relative to the position of this word. Then you will have extracted the hand written numbers in each box.


How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task? I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task.

I know that this parameter is correlated to the algorithm's ability in classifying unseen words during training. How changes text classification using low or high values of alpha?

Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a  tweet made by Trump or not. You are given a tweet, "I have huge respect for women" and your goal is to classify it to 'Trump' or 'Not Trump'. Moreover, you are given 5 other random tweets and you also know whether those tweets were by Trump or not (Basically, they are already classified) this is your training set.

in other words you are calculating,

$$P(I\,have\,huge\,respect\,for\,women) = P(I)\times P(have)\times P(huge)\times P(respect)\times P(for)\times P(women)\\$$
$$\\$$
$$P(I\,have\,huge\,respect\,for\,women|Trump) = P(I|Trump)\times P(have|Trump)\times P(huge|Trump)\times P(respect|Trump)\times P(for|Trump)\times P(women|Trump)\\$$
$$\\$$
$$P(I\,have\,huge\,respect\,for\,women|Not\,Trump) = P(I|Not\,Trump)\times P(have|Not\,Trump)\times P(huge|Not\,Trump)\times P(respect|Not\,Trump)\times P(for|Not\,Trump)\times P(women|Not\,Trump)$$
$$\\$$
Depending on the probablity of equations (2) and (3) the user would make a decision whether the statement was made by Trump or not. Now lets say that none of the 5 training set tweets have the word huge in it, in which case,
$$\\$$
$$P(huge|Trump)=P(huge|Not\,Trump)=0$$
$$\\$$
And hence, equations (2) and (3) are now zero and this is bad. A solution to this is to do smoothing or rather Laplace smoothing. The basic idea is to increase the probablities of all bigrams in your non-maximum likelihood equation (Since we are changing the counts from what they occurred in hopes to make it better) by 1 to make everything non-zero. That increase by 1 is called adding the pseudocount or as you know it as $\alpha$. Now, $\alpha=1$ may not always give the most accurate probablities and hence they could attain any finite non-negative integer. The way to know what $\alpha$ gives the most accurate responses is through iterating over all values of $\alpha$ on the training set unfortunately. The way to do that is whole different topic.

PS : Trump did say he has huge respect for women. Such irony.


In RNNs why do networks always use the last output vs the last input? All the descriptions of RNNs introduce some equation like:

$\ h_t = f_W(h_{t-1},x_t)$

and I'm wondering why we don't just go straight to the "source", ie the last input like :

$\ h_t = f_W(x_{t-1},x_t)$

The only thing I can maybe think of is maybe the RNN can store some sort of "state" in ht-1

But then, along the same vein, what would happen in an LSTM architecture if we used previous inputs vs previous outputs? It seems to me that the presence of the memory functionality should be able to avoid this potential problem (if it even is one)

That would work if the Markov Assumption was in place here (i.e. the last state was enough to determine the new state).

However, that is exactly what RNNs try to break. RNNs try to encode the whole history of inputs and predictions to determine the next output.

So in the formula, the $h_{t-1}$ is the last output, but it is a hidden state which encodes all the previous inputs, outputs, and calculations. So when you have new information coming in (i.e. $x_t$), your model can make use of all the history and the new input to determine the new input.

The same applies to LSTM, you are working with a sequence that the new output depends not only on the last input but on more things from the past, and this information is encoded in $h_{t-1}$.


Optimizing Weka for large data sets First of all, I hope I'm in the right StackExchange here. If not, apologies!

I'm currently working with huge amounts of feature-value vectors. There are millions of these vectors (up to 20 million possibly). They contain some linguistic/syntactic features and their values are all strings.

Because most classifiers do not handle string data as values, I convert them to binary frequency values, so an attribute looks like this:

@attribute 'feature#value' numeric

And per row, the value is either 1 or it is absent (so note it's a sparse ARFF file).

The thing is, with 250K rows, there are over 500K attributes and so, most algorithms have a hard time with this.

There are a lot of algorithms. I'm really curious as to what you would consider a suitable one (preferably unsupervised, but anything works), and if you even have some ideas how I could improve performance. I can train on small subsets of data, but the results only get better when using large amounts of data (at least 7 million events).

For now, I've been using NaiveBayes variations (Multinomial and also DMNBText) and those are really the only ones that are able to chew up data with acceptable speed.

Thanks a lot. If you need more information, please let me know.

Cheers.

I would go for dimensionality reduction. You can start with SVD (should be available in Weka). If SVD is too slow / too memory consuming, then there are still some options:

CUR-decomposition: a variant of singular-value decomposition that keeps the matrices of the decomposition sparse if the original matrix is sparse (see: this chapter of Mining Massive Datasets book)
Random projections: projecting the data onto a random lower-dimensional subspace (see: the Random projection in dimensionality reduction: Applications to image and text data paper)
Coresets: Given a matrix A, a coreset C is defined as a weighted subset of rows of A such that the sum of squared distances from any given k-dimensional subspace to the rows of A is approximately the same as the sum of squared weighted distances to the rows in C see the Dimensionality Reduction of Massive Sparse Datasets Using Coresets paper)

That's the tip of an iceberg. More approaches are there in the wild. The problem is that I doubt any of these solutions come with Weka (please, correct me if I am wrong on this). I would search for a usable Java implementation of any of these algorithms and try to port it to work with Weka's arff files.


How to compare number of occurrences in two data frames for a column and extract the similarities I'm trying to find the same number of occurrences in both data frames
This is a follow-up question for my previous question
I got 2 data frames
df1=pd.DataFrame([[1,None],[1,None,],[1,None],[1,'item_a'],[2,'item_a'],[2,'item_b'],[2,'item_f'],[3,'item_e'],[3,'item_e'],[3,'item_g'],[3,'item_h']],columns=['id','A'])
df2=pd.DataFrame([[1,'item_a'],[1,'item_b'],[1,'item_c'],[1,'item_d'],[2,'item_a'],[2,'item_b'],[2,'item_c'],[2,'item_d'],[3,'item_e'],[3,'item_f'],[3,'item_g'],[3,'item_h']],columns=['id','A'])

 df1
        id  A
    0   1   None
    1   1   None
    2   1   None
    3   1   item_a # id 1 has 1 occurrences in total in df1
    4   2   item_a
    5   2   item_b
    6   2   item_f #id 2 has 3 occurrences in total in df1(id 2 has 3 occurrences here)
    7   3   item_e
    8   3   item_e
    9   3   item_g
    10  3   item_h #id3 has 4 ccurrences in total in df1

df2
    id  A
0   1   item_a
1   1   item_b
2   1   item_c
3   1   item_d
4   2   item_a
5   2   item_b
6   2   item_c
7   2   item_d
8   3   item_e
9   3   item_f
10  3   item_g
11  3   item_h

I got an answer on how to find similarities by using
previous result:
d=pd.merge(df1,df2,how='inner')
        id  A
3   1   item_a # id 1 has 1 occurrences in total in d
4   2   item_a
5   2   item_b # id 2 has 2 occurrences in total in d(id 2 has 2 occurrences here which does not match all the occurrences(3) in df1)
7   3   item_e
8   3   item_e
9   3   item_g
10  3   item_h #id 3 has 4 occurrences in total in d

What I've tried to find same number of occurrences in both data frames:
d[d['id'].value_counts()==df1['id'].value_counts()]
Which gave me an error:Can only compare identically-labeled Series objects
I've also tried different things using rename to put a column name for value_counts and merge them but failed. Match: Count of occurrences in df1 for an id match count of occurrences in result data frame d
        cnt_in_df1|cntin_d
for id1:     1    | 1  count #match => id 1 should be in the desired output.
for id2:     3    | 2  count #mismatch=> id 2 should not be in the desired output
for id3:     4    | 4  count #match => id 3 should be in the desired output.

My desired output for this question:

        id  count
    0   1    1
    1   3    4

EDIT: Thanks for you clarifying the question. So now the problem is checking the counts of ids in two data frames are the same.
Here is how you could go about it:
d1 = pd.DataFrame(df1[~df1['A'].isnull()].groupby("id").size())
d2 = pd.DataFrame(d[~d['A'].isnull()].groupby("id").size())

d = pd.merge(d1,d2,on="id")

ids_ = d[d["0_x"] == d["0_y"]].index.values

RETURN:
array([1, 3])

This will now give an array of ids where the counts in both df1 and d are the same.


model.cuda() in pytorch If I call model.cuda() in pytorch where model is a subclass of nn.Module, and say if I have four GPUs, how it will utilize the four GPUs and how do I know which GPUs that are using?

model.cuda() by default will send your model to the "current device", which can be set with torch.cuda.set_device(device).

An alternative way to send the model to a specific device is model.to(torch.device('cuda:0')).

This, of course, is subject to the device visibility specified in the environment variable CUDA_VISIBLE_DEVICES.

You can check GPU usage with nvidia-smi. Also, nvtop is very nice for this.

The standard way in PyTorch to train a model in multiple GPUs is to use nn.DataParallel which copies the model to the GPUs and during training splits the batch among them and combines the individual outputs.


Multiclass classification with large number of classes but for each user the set of target classes is known Hi Data Science Stack Exchange! I'm new here but I'm familiar with some machine learning theory (took some courses in school) and my question is more about how to apply ML in a practical setting.

I have this project where I'm trying to design a system to predict which "store" a user is going to buy a given item from. However, the set of stores a user can potentially buy from is already known for each user (because this promotion only works at limited stores they signed up for). On average, the set of stores the average user can buy from is around 3 but the # of distinct stores across all users is about 10000.

In theory, for a single user this seems like a simple classification problem. We have historical information such as the time/day/month the user bought item Y from store X along with other features such as location (postal code) where the user lives and features related to the type of item they are buying (cost, weight, brand).

However, the issue is that there's currently around 6000 users so going with this approach seems like I would need a separate model for each user but that doesn't seem like an efficient solution to me or at least not how I've generally seen ML algorithms used. Unfortunately, I don't see any other way I could take into account the fact that the set of stores a user can already buy from is known for each user already. I suppose I could have a categorical variable for each buy-able store as a feature but then that would be equivalent to adding 10000 features and I'm not sure if this would scale as the number of distinct stores increases.

It would be really helpful if anyone has any insight on how to apply machine learning techniques to this kind of problem in general as this is sort my first time working on a "real" problem. Thanks!

so it sounds like you have historical transaction data. If you do, you probably want to train a model using store label as your dependent variable, and all the others independent variables.

set.seed(100)
items <- c("milk", "cheese", "steak", "apple", "eggs")
df <- data.frame(trans_id = 1:100,
                       cust_id = sample(1:10, 100, replace = TRUE),
                       item = sample(items, 100, replace = TRUE)
                       )

df$store <- as.factor(sapply(df$cust_id, function(x) {
    sample(letters[cust_id:(cust_id+2)],1)
  }))

head(df)
##  trans_id cust_id   item store
##1        1       4 cheese     e
##2        2       3 cheese     f
##3        3       6   milk     f
##4        4       1 cheese     f
##5        5       5  steak     d
##6        6       5  apple     f

#train General linear model
glm(formula = store ~., family = binomial(link = "logit"),
     data = df)

##all:  glm(formula = store ~ ., family = binomial(link = "logit"), data = df)
##
##Coefficients:
##(Intercept)     trans_id      cust_id   itemcheese
##   1.320095    -0.007399    -0.066363     0.024632
##   itemeggs     itemmilk    itemsteak
##   0.084018     0.047022     1.069399

library("e1071")
#train naive bayes
naiveBayes(formula = store ~., data = df)

Depending on business constraints you may or may not want to include customer id as a predictor variable (include if you will only be predicting stores for a finite known pool of customers; don't include if your model needs to generalize across customers). Similarly, if this were a real data set, you probably wouldn't include a transaction id as a predictor variable, because it won't generalize to new cases (transaction ids are unique)


Why doesn't LightGBM perform better than SVC (linear kernel)? [sentiment analysis] I've read and heard about the mighty XGBoost which is one of the most famous models people are using today to solve Kaggle challenge. This makes me interest to develop my own intuition about the model so I decided to try XGBoost on my dataset, but unfortunately, I experienced a lot of dependency related problems when I tried to install XGBoost on mac :(

However, I did a bit more researches and found out another related library called LightGBM from Microsoft which they claim to achieve better result (or at least, equivalent) over XGBoost. Therefore, I decided to go with the LightGBM instead since I didn't encounter any problems during the library installation.

I expected that the model should outperform, or at least, perform similarly compared to other models on the same dataset. My dataset contains 14k of text documents and has 0 or 1 values which refer to positive and negative sentiment respectively as target variable.

As a result showing below, LightGBM with count vectorizer as input got the 6th place with an accuracy of 94.60%. I'm quite curious since the accuracy of LightGBM is not only lower than SVC but also lower than Extra Tree model (300 esitimators).

I don't have enough foundation and solid understanding behind these models so I couldn't come up with an answer about these outcomes. Can someone please give me an idea or assumption according to the result?

PS. All models was performed with 5-fold cross validation

model             score
--------------  -------
extraT           0.9528
svc              0.9514
sgd_elas         0.9481
extraT_tfidf     0.9476
svc_tfidf        0.9473
lightGBM         0.9460
sgd_elas_tfidf   0.9458
lightGBM_tfidf   0.9420
randomF          0.9409
randomF_tfidf    0.9345
mulNB            0.9307
berNB            0.9087
berNB_tfidf      0.9087
mulNB_tfidf      0.9036

Xgboost usually gives a higher accuracy(this is just an observation, not a fact). But the trick with xgboost is people dont know how to best use it. The most common mistake made using xgboost is not letting it train enough. Xgboost is an algorithm with overfits more than the other algorithms(well overfit is not the right term, it should be xgboost has high variance than the others). Do not tune the other parameters until you have found a converging nrounds for appropriate learning rate(0.01 or 0.1 or 0.3 etc). If you see that the cross val accuracy is much much lower than train accuracy then tune gamma(the second order derivative) first, then other parameters. If the cross val and train accuracy have a significant but not a lot difference(lets say train acc goes from 90 to 99 and cross val is stuck at 85 after significant iterations) then try reducing the max depth of trees. If the cross val and train accuracy are running side by side then just let the algorithm converge for a particular eta and then tune the other parameters(all except gamma).


Feature extraction from web browsing history of one website I have a dataset of web browsing histories for users visiting a particular website over a period of time (say the last 90 days). Each user has a unique ID and several records showing when he/she visited a particular page on the website.

It looks like follow:

UserID,Timestamp,Path
U_1,2017-01-24 12:05:43,/sport/rugby/article_title
U_1,2017-01-24 12:06:56,/sport/football/article_title
U_1,2017-01-24 15:26:12,/finance/local/article_title
......

I do not have access to the content of the articles, I just know the path to the article.

My goal is to build a classifier to predict if a user will take an action or not. So I need to extract features from each user data.

Suppose that I have ground-truth information associated with each user, indicating when a user did the action.

My first guess is to aggregate all the records of each user and extract frequency features (hashing TF) from each level of the path.

So for demonstration, a particular user might visit the /sport category 5 times (first level category), and the /sport/football category 3 times (second level category), and the /sport/rugby 2 times (second level category).

So for each user I will have a feature vector representing the frequency of the first level categories, and another one for the second level categories and so on.

I can now train a classifier for each feature vector and do a late fusion of the results, or I can concatenate (early fusion) the different features and train a single classifier.

I can also extract the terms from the article titles and build a TFIDF feature.

What I am trying now is to extract the features from the N days proceeding taking the actions for the positive samples, and randomly select N consequent days from negative users.

What are the possible other features that I can extract, and is there any better ML techniques to use in order to model and learn the user web browsing behaviour?

What you describe uses the data that the paths can offer. You can easily generate features from the data and time. For instance, given the date, you can generate a categorical variable denoting the weekday (Monday, Tuesday, etc..). Given the timestamp, you can generate binary variables to partition the day in four or more partitions: is_morning, is_afternoon etc.. Somebody may only read in the morning or at night, and the aim of these features is to capture this.

Further, you can get the interactions between weekdays and day partitions. Such features may help to distinguish users that in the Sunday mornings read about sports while Monday mornings they are at work and read financial news. Be careful of the overfitting though. Note that trees have been shown to capture such complex interactions; given them explicitly is beneficial though.


Error of linear regression as a result of the coefficients I'm trying to resolve a paradox. Suppose that I have a bunch of data points $\{x_i,y_i\}$ and calculate a slope and intercept, $m$ and $b$ such that $y=mx+b$. Also, I can calculate the error in $m$, $\sigma_m$ and the error in $b$, $\sigma_b$. Suppose further that the true model is a line $y=ax+c +\epsilon$ where $\epsilon$ is an error term that is normally distributed and has a standard deviation of $\sigma$. If I were to calculate the error in my calculated value as a result of the errors in my coefficients I would get $error^2 = \sigma_m^2 x^2 +\sigma_b^2$. This error increases with increasing $x$, a very strange property given that the standard deviation of y at any given point x is constant, i.e. $\sigma$. What am I missing? What is the resolution of this paradox?

Update: The standard error of the slope $\sigma_m$ is given by $\frac{\sigma^2}{\sum_i (x_i -\bar{x})^2}$ so $\sigma_m$ goes down inversely with the standard deviation of the X values. I thought that perhaps this resolved the paradox. However, x,and $\bar{x}$ are not constrained by the standard deviations. All of the x values can be within two units of each other and yet x may be a thousand, million, billion or more.

You have some inherent error in $m$, say that you predict $m=m_0+a$, where $m_0$ is the true value and where you can control the size of $a$ in terms of your error bounds on $m$. Then you get:

$$y=(m_0+a)x+b=m_0x+ax+b.$$

The $ax$ term is what will cause your error to grow as you increase/decrease $x$ toward $\pm\infty$. In other words, your predicted line will ultimately be infinitely far from the true line (as can be visualized by drawing two lines with slightly different slopes).


The Israeli MOH databases on Covid patient numbers have a lower cutoff value on published numbers of patients. How does this protect privacy? The Israeli ministry of health reports many statistics on Covid patients (Numbers of confirmed cases, hospitalizations, people in quarantine - Divided by age and other demographics) available here (Albeit only in Hebrew)
In any category, numbers below 15 are reported as <15 (see e.g. this table, with statistics on medical personnel cases). Someone told me this was for privacy protection reasons, but I don't understand why.
Does this make sense? How is privacy compromised if exact low numbers are published?

It means any number lower than 15 is sufficiently small over a sufficiently large feature space so that identifiable data can be inferred by brute force reverse engineering.


In the context of machine learning, what is the relationship between "Normalization" and "norm" "Normalization" and "norm" are used a lot in machine learning

In statistics and applications of statistics, normalization can have a range of meanings.

  In the simplest cases, normalization of ratings means adjusting values
  measured on different scales to a notionally common scale, often prior
  to averaging. In more complicated cases, normalization may refer to
  more sophisticated adjustments where the intention is to bring the
  entire probability distributions of adjusted values into alignment. In
  the case of normalization of scores in educational assessment, there
  may be an intention to align distributions to a normal distribution. A
  different approach to normalization of probability distributions is
  quantile normalization, where the quantiles of the different measures
  are brought into alignment.

In linear algebra, functional analysis, and related areas of mathematics, a norm

  is a function that assigns a strictly positive length or size to each
  vector in a vector space—except for the zero vector, which is assigned
  a length of zero. A seminorm, on the other hand, is allowed to assign
  zero length to some non-zero vectors (in addition to the zero vector).
  A norm must also satisfy certain properties pertaining to scalability
  and additivity which are given in the formal definition below.

in the context of machine learning, what is the relationship between "Normalization" and "norm"

Let's take L2 norm as an example.

This figure from wiki illustrates "in a right-angled triangle the square of the hypotenuse is equal [to the sum of] the squares of the two other sides"

given a = 3, b = 4, then c = 5.

consider c as a vector,
$
c =
\begin{bmatrix}
3 \\
4 \\
\end{bmatrix}
$

the L2 norm of c is the length (magnitude) of c = 5.

to normalize the vector c, divide each component by its norm 5, then the normalized vector =
$
\begin{bmatrix}
3/5 \\
4/5 \\
\end{bmatrix}
$

this processing is called normalization.


Google TPU: when/how will it be available to me? Google recently announced the TPU custom chip.

They stated it is available in Google Cloud Platform, but only for their internal usage.

When will I be able to use it? Will it be possible to buy this chip or will it be only accessible through their cloud?

Google has apparently said it would not be available for sale outside of Google.

http://www.forbes.com/sites/tiriasresearch/2016/05/26/google-builds-its-first-chip-just-for-machine-learning/#58ab2bcb7a92

Google also did mention it would be available to developers for Tensorflow and Cloud machine learning but otherwise it has been light on details or when this will become available.


How to find correlations between data over time? I have daily data of my sales and a daily update of how the COVID-19 cases increase. My daily sales contains information on my customer and on my product. My end goal would be to see if certain products are correlated (sold more or less) after covid than before. Or if certain regions had more sales before or after COVID.

My initial thought would be to do some kind of event analysis and compare if I have more sales before and after that certain event. Does anyone has some experience with this or can link some papers/references on this topic?

You can read about time varying correlation, you can obtain the correlation at each point in time based on some fixed number of last days, enough days to make it smooth but not many so as to be able to see changes in correlation. You can also use exponential smoothers to give more weight to recent observations. Also the dynamic conditional correlation might be interesting to you.


Forgetting curve using Duolingo data I'm trying to replicate the forgetting curve using the open sourced Duolingo data for fun.
The problem is that my finding doesn't make any sense, namely that the longer you wait the better recall value. Anyone have any pointers?

# make our plot outputs appear and be stored within the notebook.
%matplotlib inline

import matplotlib.pyplot as plt # import matplotlib for scatterplot, use the alias plt
import numpy as np # import the numpy package with alias np
import pandas as pd
from scipy.optimize import curve_fit

df = pd.read_csv('learning_traces.13m.csv') # Load Duolingo data
df = df[(df['history_seen'] == 1) & (df['session_seen'] == 1)] # Seen only once before
df = df.sort_values('delta') # sort by: time (in seconds) since the last lesson/practice that included this word/lexeme
df['delta'] = df['delta'] / 60.0 # Seconds to minutes
minInMonth = 44640

def func(x, a):
    return np.exp2(-x / a)

xdata = df['delta']
ydata = df['p_recall']

plt.scatter(xdata, ydata)

popt, pcov = curve_fit(func, xdata, ydata)
print(popt) # Show the result
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit')

Expectation:

Reality

Rolling Mean

I think the most important problem you are facing is that you are trying to fit a function that somewhat erratically flips between 0 and 1 as if it were a smooth function. My Mathematical intuition says that won't be numerically stable.

To create a more smooth function you could calculate a running average of your y data and plot it against the corresponding time data.
It should descend as you forget more over a longer period of time and should be easy to plot to demonstrate your intuition.

It has hysteresis/history since initial successes will be remembered. Your fit function would be the integral:

$$\int_0^X e^{-x/a} dx = a - a\cdot e^{-X/a}$$

where $a$ will be the estimator of the forgetting rate.

Other problems could be in the time domain (just leave the unit of time at seconds), or you are mixing time scale/sample scale.

Both your expectation and reality plot should cover the complete time range on the x-axis to make them better comparable.
Maybe you should throw away data with exceptionally large time delta values anyway, as they are likely to disturb the calculations and the plots. The programs from your references remove data > 9 months and < 45 minutes if I recall correctly.

Another domain problem that could interfere is that the data mixed words that are easy to remember and words that are difficult to remember, e.g. 'cat' is in many languages easier to remember than 'butterfly'  (though that could also average out over much measurement data).


Building a large distance matrix I am trying to build a distance matrix for around 600,000 locations for which I have the latitudes and longitudes. I want to use this distance matrix for agglomerative clustering. Since this is a large set of locations, calculating the distance matrix is an extremely heavy operation. Is there any way to opimize this process while keeping in mind that I am going to use this matrix for clustering later. Below is the code I am using.

from scipy.spatial.distance import pdist
import time
start = time.time()
# dist is a custom distance function that I wrote
y = pdist(locations[['Latitude', 'Longitude']].values, metric=dist)
end = time.time()
print(end - start)

Have you considered that the following steps will be even worse?

The standard algorithm for hierarchical clustering scales O(n³). You just don't want to use it on large data. It does not scale.

Why don't you do a simple experiment yourself: measure the time to compute the distances (and do the clustering) for n=1000,2000,4000,8000,16000,32000 and then estimate how long it will take you to process the entire data set assuming that you had enough memory... You will see that it is not feasible to use this algorithm on such big data.

You should rather reconsider your approach...


How do attention mechanisms in RNNs learn weights for a variable length input Attention mechanisms in RNNs are reasonably common to sequence to sequence models.

I understand that the decoder learns a weight vector $\alpha$ which is applied as a weighted sum of the output vectors from the encoder network. This is used to produce a new input vector.

What I don't understand is that the learned weight vectors $\alpha$ must be a fixed size vector because it's treated as learned weights, but it's applied to a variable length sequence.

If someone could help me understand this particular mechanism I'd appreciate it.

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.

For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\alpha_{iT_x})$ is calculated for target element $y_i$  as follows
$$\alpha_{ij}=\frac{\text{exp}(e_{ij})}{\sum_{k=1}^{T_x}\text{exp}(e_{ik})}$$
where $e_{ij}$ is calculated by neural network $a$ that receives hidden state $s_{i-1}$ of decoder (decoder generates the target sequence element-by-element) and hidden state $h_j$ of encoder (encoder distills the source sequence into hidden states $h_j$, where $h_j$ is the concatenation of $j$-th hidden states of forward and backward RNNs) and outputs
$$e_{ij} = a(s_{i-1}, h_j).$$
In other words, network $a$ recieves a vector of size $|s| + |h|$ and outputs a number. Because of this network, the attention matrix $\boldsymbol{\alpha}$ is free in size.

Attention weights are then used to calculate context $c_i$ for target $y_i$ as follows
$$c_i = \sum_{j=1}^{T_x} \alpha_{ij}h_j,$$

and so on and so forth.

Consequently, for the next (source, target) pair with lengths ($T_{x_2}$, $T_{y_2}$), the size of $\boldsymbol{\alpha}$ would also be $T_{y_2} \times T_{x_2}$ with no problem.


Making sense of loss and accuracy curves This is an issue that I have come across over and over again. Loss (cross-entropy in this case) and accuracy plots that do not make sense. Here is an example:

Here, I’m training a ReNet18 on CIFAR10. Optimizer is SGD with 0.1 learning rate, 0.9 Nesterov momentum, 1e-4 weight decay. The learning rate is decreased to a ⅕ at epochs 60, 120, 160.

Initially the curves are all nice and dandy; meaning training and
validation loss are decreasing and accuracies are increasing.
Around epoch 65~70 you see signs of overfitting; as val. loss starts
increasing and val. accuracy starts decreasing (the red box). There
is still nothing strange here.

Now there are two things that don’t make sense to me:

After epoch 120 (where LR is decreased) val. loss and accuracy start
improving for a couple of epochs (the green box). Why would
decreasing the learning rate suddenly improve validation performance
of a model that was already overfitting?! I would expect the drop in
LR to actually accelerate overfitting.

After epoch ~125 (the blue box) loss starts going up but accuracy keeps
improving. I understand that loss could go up while accuracy stays
constant (by the model getting more confident in its wrong
predictions or less confident in its correct predictions). But I
don’t get how accuracy can improve while loss goes up.

Just a couple of points below

Red box: In strict terms, overfitting takes place when the model is doing better in the training than the validation set. And so, I would say here it starts earlier than what is denoted in red and it probably does relate to the model finding better parameters that only apply to the training set. Of note, no significant train loss decrease.
Green box: In training, significant drop in the loss. This discovery of a significantly better minima means it may also apply in the validation set and it is actually reflected in the val loss curve. Despite that, the loss is no better than the checkpoint at the beginning of the red box.
Blue box: Loss and accuracy are not co-dependent metrics. Whereas accuracy is a measure of the number of correct guesses (how many), the loss is a measure of distance from correct given a data point (how far off).

Generally, smaller LR means the model has less "freedom" to hop very far around in the feature space. And so there are fewer chances to see significant and fast loss decrease. As you suggest, there is a slight upward overfitting trend in your val loss but is as significant or fast as the smaller LR allows in training.
Also, note that your train loss is higher than val loss for a good amount of your training (~55th epoch). You may want to investigate how your model is regularised as this may affect your learning curves in ways that may help you infer diagnostics better. E.g. in PyTorch, train loss between model states (model.train() and model.eval()) differ significantly.


Significance shifts when cases are excluded I have a county level dataset with a dichotomous variable which represents whether or not any charter schools exist in a county.  I am performing a nested logistic regression. That variable is the response and the explanatory variables are social factors such as education level, language spoken at home, household type, etc.  When I include all counties in the U.S. (Including ten states which have no charter authorization laws and thus every county in those states is represented as a zero in the response variable) my results show significant associations between the response and the explanatory variables.  When I exclude those ten states the significance disappears.

Any ideas as to why this might be happening?

Sounds like you came across Simpson's Paradox. Roughly speaking you have a result in one group of data but not when you look at different groups or subsets. This often happens when the variables depend on hidden dimensions in the data.

The wikipedia page explains it a bit better and in more detail.


Completing MDS manually in R Given a matrix A, I want to complete Multidimensional Scaling by hand, instead of using any given R functions.

As such, I have calculated the centered matrix B with the following code:

 n<-nrow(A)
 id<-diag(n)
 e<-diag(id)
 H <- id - (1/n)*e %*% etranspose
 B <- (-1/2)* H %*% A %*% H

My question is: how can I use my B matrix to complete multidimensional scaling on my A matrix, without the cmdscale function or anything along those lines?

If you're interested in the classic MDS algorithm, it is spelled out very nicely on the Wikipedia page:

  Classical MDS uses the fact that the coordinate matrix can be derived
  by eigenvalue decomposition from $B=XX'$. And the matrix $B$ can be
  computed from proximity matrix $D$ by using double centering.

  Set up the squared proximity matrix $D^{(2)}=[d_{ij}^{2}]$
  Apply double centering: $B=-{\frac {1}{2}}JD^{(2)}J$ using the centering matrix $J=I-{\frac {1}{n}}11'$, where $n$ is the number of
  objects.
  Determine the $m$ largest eigenvalues $\lambda _{1},\lambda _{2},...,\lambda _{m}$ and corresponding eigenvectors $e_{1},e_{2},...,e_{m}$ of $B$ (where $m$ is the number of dimensions
  desired for the output).
  Now, $X=E_{m}\Lambda _{m}^{1/2}$, where $E_{m}$ is the matrix of $m$ eigenvectors and $\Lambda _{m}$ is the diagonal matrix of $m$
  eigenvalues of $B$.

I can't really tell what your $A$ matrix is, but if each entry is a measure of the distance from the $i^{th}$ to $j^{th}$ entry, then that would be your proximity matrix.

Also, $11'$ is an n-by-n matrix of all 1's. This should be fairly self-explanatory, however most of these terms can be Googled if you're unsure. It really shouldn't be any more than about 10 lines of code.


How to compare experiments run over different infrastructures I'm developing a distributed algorithm, and to improve efficiency, it relies both on the number of disks (one per machine), and on an efficient load balance strategy. With more disks, we're able to reduce the time spent with I/O; and with an efficient load balance policy, we can distribute tasks without much data replication overhead.

There are many studies on the literature that deal with the same problem, and each of them runs different experiments to evaluate their proposal. Some experiments are specific of the strategy presented, and some others, like weak scaling (scalability) and strong scaling (speedup), are common to all of the works.

The problem is the experiments are usually executed over entirely different infrastructures (disks, processors, # machines, network), and depending on what is being evaluated, it may raise false/unfair comparisons. For example, I may get 100% of speedup in my application running on 10 machines with Infiniband connection, whereas I could get the same or even worse results if my connection was Ethernet.

So, how can one honestly compare different experiments to point out efficiency gains?

This is a very good question and a common situation.

In my opinion there are three different factors that must be controlled:

Data: There exist already different benchmarks in order to evaluate algorithms and architectures. These data must be publicly available so that everybody can contrast their approaches.
Architecture: My suggestion is to test everything on the cloud, so that everyone can contrast their results and also there is no doubt the same machines and software is used.
Algorithms: If you have developed a distributed algorithm, it would be great to compare your algorithm on a specific data. In this case, algorithms must not be public.

So, answering your question, if you want to compare different experiments and state to what extent your distributed algorithm outperforms others, you should try to replicate as accurate as possible the same environment (data and architecture) where the experiments were carried out.

If this is not possible, my suggestion is that you test your algorithm with public data and cloud architecture so that you become a referent as you are facilitating the comparison of future algorithms.


Sentiment data for Emoji For experimenting we'd like to use the Emoji embedded in many Tweets as a ground truth/training data for simple quantitative senitment analysis. Tweets are usually too unstructured for NLP to work well.

Anyway, there are 722 Emoji in Unicode 6.0, and probably another 250 will be added in Unicode 7.0.

Is there a database (like e.g. SentiWordNet) that contains sentiment annotations for them?

(Note that SentiWordNet does allow for ambiguous meanings, too. Consider e.g. funny, which is not just positive: "this tastes funny" is probably not positive... same will hold for ;-) for example. But I don't think this is harder for Emoji than it is for regular words...)

Also, if you have experience with using them for sentiment analysis, I'd be interested to hear.

Total of 972 emoji is not really that big not to be able to label them manually, but I doubt that they will work as a good ground truth. Sources like Twitter are full of irony, sarcasm and other tricky settings where emotional symbols (such as emoji or emoticon) mean something different from normal interpretation. For example, someone may write "xxx cheated their clients, and now they are cheated themselves! ha ha ha! :D". This is definitely negative comment, but author is glad to see xxx company in trouble and thus adds positive emoticon. These cases are not that frequent, but definitely not suitable for ground truth.

Much more common approach is to use emoticon as a seed for collecting actual data set. For example, in this paper authors use emoticon and emotional hash tags to grab lexicon of words useful for further classification.