Elbow method on hundreds columns and rows So I have these vectors called matrix_ after I applied TF-IDF (term frequency-inverse document frequency), and I also converted it to dataframe matrixDF_ .

I wanted to cluster these matrix_ vectors with MiniBatchKMeans from sklearn. But since the vectors are large (569 rows and 829 columns), then I need to find the optimum K-Cluster first.

That is why I use KElbowVisualizer from Yellowbrick. At first, it works fine and I got the optimum k-cluster. However, when rerunning it again, it showed me a different number of k-cluster. And that is where I am confused then.

Anyone can help me with this problem?

here is my code:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import MiniBatchKMeans
from scipy.spatial.distance import cdist
from sklearn.metrics import silhouette_score
from yellowbrick.cluster import KElbowVisualizer

""" vectorize the words. getting the tf-idf of each words """
vec = TfidfVectorizer(
     lowercase = True,
     stop_words = 'english',
     use_idf=True
)

matrix_ = vec.fit_transform(df)
matrixDF_ = pd.DataFrame(matrix_.toarray(), columns=vec.get_feature_names())

model = MiniBatchKMeans(MiniBatchKMeans(init_size=559, random_state=None))
visualizer = KElbowVisualizer(model,k=(2, 11), metric='distortion',
timings=True, size=(1080, 720))
visualizer.fit(matrix_)        # Fit the data to the visualizer
visualizer.show()

I also tried fit the dataframe matrixDF_. to the visualizer, but each output after every run has different output number.

Here are some of output examples. First output:

second output:

You are using different random starting points each time you run k-means (random_state=None) This means you may get different clusterings, different clustering metrics each time. That's expected. What you may wish to do is average the results over several different runs to get a more reliable estimate of the loss at each k.


Problem with calculating error rate for KNN I am trying to validate the accuracy of my KNN algorithm for the movie rating prediction.

I have $2$ vectors: $Y$ - with the real ratings, $Y'$ - with predicted ones.

When I calculate Standard Error of the Estimate (is it the one I need to calculate?) using following formula:

$$\sigma_{est} = \sqrt{\frac{\sum (Y-Y')^2}{N}}$$

I'm getting result of $\sim 1.03$. But I thought that it can't be $> 1$. If it is not, then what does this number say to me?

results = load('first_try.mat');
Y = results(:,1);
Y_predicted = results(:,2);

o = sqrt(sum((Y-Y_predicted).^2)/rows(Y))

K-NN is a measure of distance, thus the result of your equation will depend on the scale of your data. If the ratings are in a scale from 0 to 100. Then if you always predict very poorly you are evidently going to have values much larger than 1.

For example, for a very bad predictor

import numpy as np

Y = [100, 90, 100, 90]
Y_p = [10, 10, 10, 10]

np.sqrt(np.sum(np.subtract(Y, Y_p)**2)/len(Y))

  85.14693182963201


Joining tables from different locations in Bigquery been trying to join two tables in different data set which are in different locations but in same project. Always getting an error that

  dataset not found in US location.

The datasets locations are US and us-east1

Here is what I am doing

select a.*, b.* from `project.dataset1.table1` a join `project.dataset2.table2`
on a.common_col = b.common_col

Please help me out in this?

You can't join two tables in different data set which are in different locations. As an alternative solution, you can copy datasets between regions using BigQuery Data Transfer Service. Here is the documentation link Copying Datasets:
bq mk --transfer_config \
      --project_id=myproject \
      --data_source=cross_region_copy \
      --target_dataset=dataset_us \
      --display_name='Copy Dataset' \
      --params='{"source_dataset_id":"dataset_eu","source_project_id":"myproject"}'


Implementing sklearn's FeatureHasher on Unseen Data For a little bit of background I have been working on a binary classification of health insurance claims and am implementing sklearn's FeatureHasher to vectorize categorical features, many of which are particularly high in cardinality with a high count of unique factor levels and sklearn's FeatureHasher has been a useful tool to encode all of this information.

I have trained an XGBoost classifier on my initial training set which was transformed such that each categorical feature was hashed into log(x) dimensions (x being the number of unique factor levels within in each feature).  I am planning to use the trained model to make predictions on new/unseen claims data which will be scored on a daily basis.  I will be hashing the daily extract of claims data with identical hashing dimensions used in the training set, such that there are no disparities between the number of features the model was trained upon and the number of features within the daily extract, allowing me to use the trained model to make predictions on incoming claims data.

I am wondering how sklearn's FeatureHasher encodes unseen data.  For example, if I had ten factor levels in a specific categorical feature within my original training set and only three factor levels within the same categorical feature in an incoming daily extract of claims data (likely to be a much lower volume of claims data as compared to the training set), does FeatureHasher encode this information in the same way such that each factor level can be identified and mapped back to its original value?  Conversely, will hashing encode factor levels within categorical features differently between training and test sets if the number of factor levels within a given feature are not equal to what the model was trained upon? If so would this cause issues with how the model interprets the factor levels of a given categorical feature within a daily extract of claims causing the model to confuse or misinterpret factor levels from how they were originally encoded in the training set?

Any feedback would be greatly appreciated!

P.S. I have looked into using CatBoost gradient boosting (catboost.ai/docs/concepts/python-installation.html) as an alternative but unfortunately am using a platform that does not support this and thus will not be an option for me

Feature hashing just applies a fixed hash function to its input strings; it doesn't need to have seen any data.  Note the docstring for the fit method:

No-op.
This method doesn’t do anything. It exists purely for compatibility with the scikit-learn transformer API.

There may be collisions, and you won't know it with unseen data; your model will be forced to consider these random collisions as the same, but this may not be an issue.  Feature hashing is really only meant for situations where you need to keep memory usage down.


Standardization with positive and negatives I have a data set that has a few columns such as:
Total cost: mean = 3,000,000
Percent complete: mean = 50
final profit %: mean = 14
I know with such different orders of magnitude before I fit a linear regression I should standardize the data (using python and sklearn). The problem is there are negatives in this data that I need to keep so I don't know which type of standardization I should use? The only two I am familiar with are log transformations and StandardScaler both of which I think get rid of negatives.

You can use Normalization. Normalization rescale your mean to 0 and standard deviation to 1 containing both positive and negative value.
$X_{Normalised} = \frac{X - \mu}{\sigma}$
Here $\mu$ is your original mean and $\sigma$ is your standard deviation.


weight training speed too slow in CNNs I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a seemingly converged manner). My initial weights: convolution kernels as non zero unit matrices; fully connected layer weights as 0's. The activation function is sigmoid. The data scale from 0 to 1. Why do the weights change so little?

In machine learning, the vanishing gradient problem is a difficulty
  found in training artificial neural networks with gradient-based
  learning methods and backpropagation. In such methods, each of the
  neural network's weights receives an update proportional to the
  partial derivative of the error function with respect to the current
  weight in each iteration of training. The problem is that in some
  cases, the gradient will be vanishingly small, effectively preventing
  the weight from changing its value. In the worst case, this may
  completely stop the neural network from further training.

Source :: https://en.wikipedia.org/wiki/Vanishing_gradient_problem

Thus, gradient is vanishing till it reaches initial layer of neural network and in turn very little change in weights.


Mapping output neurons to classes I have read few articles some say there is no need to have no. of units in output layer = no. of classes
why some say they both should be equal.
My questions are

If no. of neurons = no. of classes. How are the classes mapped to each neuron/unit in the output layer. To elaborate how does Neural Network decide which neuron/unit deals with which class.

After training a neural network using tensorflow on a multiclass classification problem using softmax in out put layer and no. of units in output layer = no. of classes. When I use this trained network on test sample, the output is a numpy array of probabilities of each class. How do I whther the first element of that array represents which class.

If no. of units is not equal to no. of classes can some one share a link of such an example for multiclass problems.

In multiclass classification problems, the number of output units (i.e. neurons) is equal to the number of classes. Only in binary classification, you have 2 classes but just 1 output unit.

The mapping between output units and classes is decided by you. You assign each class an index from $0$ to $N-1$, where $N$ is the number of classes. This values (or one-hot versions of them) are what you supply as expected output to the loss function. Output units form a vector of size $N$, with each unit being at a specific position in that vector. The unit at position $i$ represents the class with index $i$.

The output of the softmax is also a vector of $N$ probability values. The probability at position $i$ represents the probability of the input data belonging to the class with index $i$.


Machine learning algorithms and Computer Vision technologies for detecting 52 playing cards deck https://en.wikipedia.org/wiki/Standard_52-card_deck
https://en.wikipedia.org/wiki/Playing_card
https://www.google.com/search?q=playing+cards&client=ms-android-lava&prmd=sinv&sxsrf=ALeKk02cmtbG25vrqk59BybOJWHF7PG3Aw:1600237953234&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjcuIj2hu3rAhXtzjgGHZegCj0Q_AUoAnoECAwQAg
https://www.google.com/search?q=playing+cards&client=ms-android-lava&prmd=sinv&sxsrf=ALeKk02cmtbG25vrqk59BybOJWHF7PG3Aw:1600237953234&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjcuIj2hu3rAhXtzjgGHZegCj0Q_AUoAnoECAwQAg
https://www.instagram.com/p/CFMBjc6J7PN/?utm_source=ig_web_button_share_sheet
Are there Machine learning algorithms & Computer vision technologies which can view the 52 playing card deck?
Input : 52 playing card deck images as .jpg,.gif,.tiff
Output : Detecting the playing card names.
For example : Spade Ace, Three diamonds, Jack of Hearts, King of Clubs.
All the playing cards names should be detected by inputting playing cards images.
If no, what are the limitations? Do the 52 playing card images have to be modified?
One can also add one more playing card as "Joker" to the existing 52 playing cards added as a image as input.

Yes, you can recognize these cards. For easy implementation, you can check here and here. Also, you can build your custom neural network model with tensorflow, keras, pytorch etc. Recognizing visually "52 card deck" is already solved problem. Because, cards have good features & landmarks. You either can use neural networks or "old school" computer vision algorithms.


Why Scikit and statsmodel provide different Coefficient of determination? First of all, I know there is a similar question, however, I didn't find it so much helpful.

My issue is concerning simple Linear regression and the outcome of R-Squared. I founded that results can be quite different if I use statsmodels and Scikit-learn.

First of all my snippet:

import altair as alt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm

np.random.seed(0)
data = pd.DataFrame({
'Date': pd.date_range('1990-01-01', freq='D', periods=50),
'NDVI': np.random.uniform(low=-1, high=1, size=(50)),
'RVI': np.random.uniform(low=0, high=1.4, size=(50))
 })

Output:

          Date        NDVI        RVI
 0    1990-01-01    0.097627    0.798275
 1    1990-01-02    0.430379    0.614042
 2    1990-01-03    0.205527    1.383723
 3    1990-01-04    0.089766    0.142863
 4    1990-01-05    -0.152690   0.292427
 5    1990-01-06    0.291788    0.225833
 6    1990-01-07    -0.124826   0.914352

My independent and dependent variable:

X = data[['NDVI']].values
X2 = data[['NDVI']].columns
Y = data['RVI'].values

Scikit:

regressor = LinearRegression()
model = regressor.fit(X, Y)
coeff_df = pd.DataFrame(model.coef_, X2, columns=['Coefficient'])
print(coeff_df)
Output:
    Coefficient
NDVI    0.743

print("R2:", model.score(X,Y))

R2: 0.23438947208295813

Statsmodels:

model = sm.OLS(X, Y).fit() ## sm.OLS(output, input)
predictions = model.predict(Y)
# Print out the statistics
model.summary()

Dep. Variable:  y   R-squared (uncentered): 0.956
Model:  OLS Adj. R-squared (uncentered):    0.956
Method: Least Squares   F-statistic:    6334.
Date:   Mon, 18 May 2020    Prob (F-statistic): 1.56e-199
Time:   11:47:01    Log-Likelihood: 43.879
No. Observations:   292 AIC:    -85.76
Df Residuals:   291 BIC:    -82.08
Df Model:   1
Covariance Type:    nonrobust
    coef    std err t   P>|t|   [0.025  0.975]
x1  1.2466  0.016   79.586  0.000   1.216   1.277
Omnibus:    14.551  Durbin-Watson:  1.160
Prob(Omnibus):  0.001   Jarque-Bera (JB):   16.558
Skew:   0.459   Prob(JB):   0.000254
Kurtosis:   3.720   Cond. No.   1.00

And scatterplot of data:

How should I proceed with this analysis?

You need to add an intercept to statsmodels manually, while it is added automatically in sklearn.
import altair as alt
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm

np.random.seed(0)
data = pd.DataFrame({
'Date': pd.date_range('1990-01-01', freq='D', periods=50),
'NDVI': np.random.uniform(low=-1, high=1, size=(50)),
'RVI': np.random.uniform(low=0, high=1.4, size=(50))
})

X = data[['NDVI']].values
X2 = data[['NDVI']].columns
Y = data['RVI'].values

# Sklearn (note syntax order X,Y in fit)
regressor = LinearRegression()
model = regressor.fit(X, Y)
print("Coef:", model.coef_)
print("Constant:", model.intercept_)
print("R2:", model.score(X,Y))

# Statsmodels (note syntax order Y,X in fit)
X = sm.add_constant(X) # manually add a constant here
model = sm.OLS(Y, X).fit()
print(model.summary())

Results:
Sklearn:
Coef: [-0.06561888]
Constant: 0.5756540424787774
R2: 0.0077907160447101545

Statsmodels:
                            OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.008
Model:                            OLS   Adj. R-squared:                 -0.013
Method:                 Least Squares   F-statistic:                    0.3769
Date:                Tue, 19 May 2020   Prob (F-statistic):              0.542
Time:                        11:18:42   Log-Likelihood:                -25.536
No. Observations:                  50   AIC:                             55.07
Df Residuals:                      48   BIC:                             58.90
Df Model:                           1
Covariance Type:            nonrobust
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const          0.5757      0.059      9.796      0.000       0.457       0.694
x1            -0.0656      0.107     -0.614      0.542      -0.281       0.149
==============================================================================
Omnibus:                        5.497   Durbin-Watson:                   2.448
Prob(Omnibus):                  0.064   Jarque-Bera (JB):                3.625
Skew:                           0.492   Prob(JB):                        0.163
Kurtosis:                       2.122   Cond. No.                         1.85
==============================================================================


How can I count the number of matching zero elements between two numpy arrays? I have a function that returns the predicted accuracy of a time-series model. I have two equally-sized numpy arrays, one for the actual direction and one for the predicted direction. I'm classifying whether there is a change in the data's direction from the previous t-1 step.  '1' stands for an increase from t-1, '0' stands for no change in direction, and '-1' stands for a decrease from t-1. I'm trying to compare the elements of both arrays to determine if they both contain the same data to determine accuracy.
I can match indexes and count the number of '1s' and '-1s' that match,  but I cannot count the number of matching zeros. (It's kinda hard summing zeros). :-)  Anyway, I've tried the numpy sum function specifying '0' as the argument for both arrays but it only returns an array of zeros but no count.  I'm not trying to create a confusion matrix...the goal here is to create an accuracy score.  I plan to take all the matching ones, negative ones and zeros and divide that by the total length to get an accuracy score.
Thanks in advance.

You can compare the predictions with the expected results directly, using simple comparisons, in this case just ==. This returns boolean values - True or False, which you can sum up because True == 1 and False = 0.
Here is an example for your case using some randomly generated dummy data:
In [1]: import numpy as np

In [2]: y = np.random.choice([-1, 0, 1], 10)

In [3]: preds = np.random.choice([-1, 0, 1], 10)

In [4]: y
Out[4]: array([ 1,  1,  1, -1,  1, -1, -1,  1,  1,  0])

In [5]: preds
Out[5]: array([ 0, -1,  1,  0,  1,  1, -1,  1, -1,  0])

The real part that checks where your predictions are correct is then done using two checks:

where are the predictions equal to the ground truth i.e. where were you correct in predicting the direction, and
where are the predictions equal to the direction you are interested in i.e. downwards (-1), no change (0) or upwards (1)

This can be done as follows:
In [6]: (y == preds) & (preds == 0)   #   &   means we needs both checks to be True
Out[6]:
array([False, False, False, False, False, False, False, False, False, True])

We can see only the final position is True after both these checks, because that is the only place that the prediction was True and the value direction was 0.
You can then write a loop to check all values and do something with them:
In [7]: n = len(preds)    # the number of test samples (= 10 in my dummy example)

In [8]: for direction in [-1, 0, 1]:
            score = sum((y == preds) & (preds == 0))
            accuracy = score / n
            print(f"Direction {direction:>2}: {score}/{n} = {accuracy * 100:.1f}%")

Which gives:
Direction -1: 1/10 = 10.0%
Direction  0: 1/10 = 10.0%
Direction  1: 1/10 = 10.0%


Does image resizing lower the prediction accuracy of MLP? I am implementing a vanilla neural network (MLP) to do image classification in python using tensorflow on images of honey bees to detect their health status. The images in my dataset are of different shapes and sizes, so I decided to do image resize using cv2. All my images are now of the same size (64 by 64) but some of them have been stretched/shrieked due to resizing. Does this have an effect on the low prediction accuracy I am getting from my MLP?

Generally speaking, it highly depends on your data. If you have images of numbers for each image, it may not be that much bad but for images of cats or dogs, you completely put your information away by resizing to that size.

To answer your question, yes. The reason is that it leads to high Bayes error. It simply means that you as an expert can not say what they are. Consequently, it is not possible for the network to learn them. You can easily see the images and figure out that there is not anything to be learned. For instance, in that case, what is the difference between the sky and sea? Can a $64\times64$ image represent them? Can you as an expert find it out without any previous knowledge?


What is the difference between shuffle in fit_generator and shuffle in flow_from_directory? I am using Keras to create a deep learning model and I would like to know that what is the difference between shuffle argument in fit_generator() method and shuffle argument in flow_from_directory() method?

From fit_generator() documentation:

  shuffle: Boolean. Whether to shuffle the order of the batches at the beginning of each epoch. Only used with instances of Sequence
  (keras.utils.Sequence). Has no effect when steps_per_epoch is not
  None.

So if you are using a generator and set steps_per_epoch it would have no effect. In case of using a Sequence generator, it would get the batches in different order at each epoch. For example, in first epoch it might get my_seq[0] and then my_seq[1] and then my_seq[2] (and so on) and in the second epoch it might get my_seq[2] and then my_seq[0] and then my_seq[1] and so on for the next epochs (i.e. note that my_seq[i] is the i-th batch generated by the Sequence generator we have defined).

From flow_from_directory() documentation:

  shuffle: Whether to shuffle the data (default: True)

It is not clear when and how shuffling is done. So we must take a look at the source code. By doing so, we find out that flow_from_directory() method returns an instance of DirectoryIterator class which in its docstring we can see the following:

  shuffle: Boolean, whether to shuffle the data between epochs.

So it would shuffle all of the data (i.e. list of all the training images to be read at each epoch) at the beginning of each epoch.


R plot(surv(), newdata=...) draws same lines many times - why? I'm new to R and cannot make plotting work as desired. The problem is that R seems to draw the same four lines over and over again, redundantly. The details of the case I'm having are as follows.

I have a dataset:

> str(dataset)
'data.frame':   57641 obs. of  3 variables:
 $ duration : num  3 8 7 2 4 8 2 2 8 8 ...
 $ graduated: logi  FALSE TRUE TRUE FALSE FALSE TRUE ...
 $ group    : num  651 651 671 671 651 651 651 651 651 651 ...

Then, I fit a Cox proportional hazards regression model to it:

survObj <- Surv(time = dataset$duration / 2, event = dataset$graduated)
model <- coxph(survObj ~ group, data=dataset)

Next, following this example, I create a frame that would hopefully group the survival functions by the group number:

frame <- data.frame(group = dataset$group)

> str(frame)
'data.frame':   57641 obs. of  1 variable:
 $ group: num  651 651 671 671 651 651 651 651 651 651 ...

There's four groups in the data:

> unique(dataset$group)
[1] 651 671 652 681

Using this new frame, I create a fitted survival model:

fitObjGrouped <- survfit(model, newdata = frame)

Finally, I plot the thing:

color_set <- rainbow(4)
plot(fitObjGrouped, col=color_set)

The result has the correct lines, but drawn many times over each other:

As you can see, there's two red lines and two blue lines drawn last. They're the correct ones, one for each category, but closer observation reveals that there's a green or other color lines underneath each of them. When converting this to PDF the file size is 273 times larger than what it should be!

So the question is: why is R drawing the lines so many many times and how could I achieve correct model fitting and plotting at the same time?

Can somebody please help me to better understand the R commands I'm using? Thanks in advance!

Note that in the linked presentation, on the slide titled "Plotting the effects", the treat object has only 2 rows.

In your case, because frame has 57k rows, fitObjGrouped has predictions for each row of newdata.  You can verify this with fitObjGrouped$n.  To fix the problem, try:

frame <- data.frame(group = unique(dataset$group))


Smooth curve in Bokeh I have been trying to plot my data from a cvs file with Bokeh but I couldn't find a smoothing option to smooth my graph curve.

Is there any in bokeh.plotting for this task? If not, how else should I proceed?

bokeh library internally uses _glyph_function function to plot, if you take a look at their source code and which takes help from basic numpy, scipy library for defining arrays and other stuff and this so goes for curve smoothing too. So better do achieve this via scipy.


Preferred approaches for imbalanced data I am building a binary classification model with imbalanced target variable (13% Class 1 vs 87% class 0). I am considering the following three options to handle the data imbalance

Option1: Create a balanced training dataset where with 50% / 50% split of the target variable.
Option 2: Samples the dataset as-is (i.e., 87% / 13% split) and use upsampling methods (e.g., SMOTE) to balance the target variable to 50% / 50% split.
Option 3: Use learning methods with appropriate hyperparameters to account for data imbalance for example: scale_pos_weight in XGBoost, class_weight in LGBMRegressor, class_weight in RandomForestClassifier

Assuming I have enough available data, is the first option is always the best approach?
What are the Cons and Pros of each of the three methods? especially the 2nd and 3rd options (I assume that it is always preferred to avoiding creating new synthetic samples)

I think it mostly depends on your dataset type! are you dealing with text? or image? or... and your features will tell which option is the best fit for your case....but according to my experience in most of the cases, option 1 and 2 besides they depend on your dataset and power of your features they need to be judge based on your model high bias or variance and they should inform you they are good or no! you need to do some experiment to figure out them or know your dataset well to find out adding or reducing dataset will affect your model performance or not!

and what I like to tell is try to use upsampling and downsampling methods same time to make your dataset balanced in a fair way(kinda)!....in this case (87% class 0 and 13% class 1)....upsample class 1 and downsample class 0! how much you need to upsample or how much downsample it is all your choice and definition of fairness in your dataset! and this definition could differ!


How to reduce dimensionality of 3.2B categorical features? Background:

This means a dataset of 7,000 samples and 3.2B columns, which I would have to read into distributed Spark memory somehow.

Obviously I want to reduce the number of columns that gets fed into the final model, although I could start with a smaller amount of samples.

Approaches:

I keep coming back to this Multiple Factor Analysis for categorical features http://factominer.free.fr/factomethods/multiple-factor-analysis.html which looks like it would give me groups... but do I have to manually decide which groups to remove?

I've also read about autoencoders NNs being used for dimensionality reduction.

I've also read about including all inputs directly in the first layer of my desired model and then drastically reducing the number of inputs into the second layer as another form of dimensionality reduction.

More Background:

There are 3.2B positions in the human genome. Each position can contain either one of 4 different nucleotides or some larger insertions/ deletions of nucleotides.

I have 2,200 cases (prior to QC filtering) for a relatively common disease and I can and compare them to as many controls as I like so 4,800 controls seems like a reasonable balance. I've read that case-control ratios edging toward 90-10 are considered imbalanced.

What approach should I take?

Based on this article, I have decided to run with an autoencoder (tanh, tanh, tanh) neural net as opposed to PCA.

  "[for] thousands of dimensions the autoencoder has a better chance of unpacking the structure and storing it in the hidden nodes by finding hidden features ... [Also,] in contrast to PCA, the autoencoder has all the information from the original data compressed in to the reduced layer. [Finally,] the autoencoder is better at reconstructing the original data set than PCA when k is small, however the error converges as k increases. For very large data sets this difference will be larger and means a smaller data set could be used for the same error as PCA."

  https://www.r-bloggers.com/pca-vs-autoencoders-for-dimensionality-reduction/

Autoencoders for dimensionality reduction are also explained in OReilly's TensorFlow book.

UPDATE:
It looks like an embeddings layer as the first hidden layer would also be a good start to reducing dimensionality.

UPDATE: keras pooling layers also seem relevant


Visualizing ConvNet filters using my own fine-tuned network resulting in a "NoneType" when running: K.gradients(loss, model.input)[0] I have a fine-tuned network that I created which uses vgg16 as it's base.  I am following section 5.4.2 Visualizing CovNet Filters in Deep Learning With Python (which is very similar to the guide on the Keras blog to visualize convnet filters here).

The guide simply uses the vgg16 network.  My fine tuned model uses the vgg16 model as the base, for example:

model.summary()

Layer (type)                 Output Shape              Param #
======================================================================= vgg16 (Model)                (None, 4, 4, 512)         14714688
________________________________________________________________________ flatten_1 (Flatten)          (None, 8192)              0
________________________________________________________________________ dense_7 (Dense)              (None, 256)               2097408
________________________________________________________________________ dense_8 (Dense)              (None, 3)                 771
======================================================================== Total params: 16,812,867 Trainable params: 16,812,867 Non-trainable params: 0

I'm running into an issue when I run this line:
grads = K.gradients(loss, model.input)[0] where when I use my fine tuned network I get a result that's a "NoneType"

Here is the code from the guide:

> from keras.applications import VGG16 from keras import backend as K
>
> model = VGG16(weights='imagenet',
>               include_top=False)
>
> layer_name = 'block3_conv1' filter_index = 0
>
> layer_output = model.get_layer(layer_name).output loss =
> K.mean(layer_output[:, :, :, filter_index])
>
> grads = K.gradients(loss, model.input)[0]

To reproduce the on my fine tuned model, I've used the exact same code, except I obviously changed the model that I imported:

model = keras.models.load_model(trained_models_dir + 'fine_tuned_model.h5')

...and I also had to index into the nested Model object (my first layer is a Model object as is shown above) to get the 'block2_con1' layer:

my_Model_object = 'vgg16'
layer_name = 'block3_conv1'
filter_index = 0

layer_output =
model.get_layer(my_Model_object).get_layer(layer_name).output

any idea why running grads = K.gradients(loss, model.input)[0] on my fine tuned network would result in a "NoneType"??

Thanks.

SOLVED:  I had to use: grads = K.gradients(loss, model.get_layer(my_Model_object).get_layer('input_1').input)[0]

instead of just grads = K.gradients(loss, model.input)[0]

which is confusing because both model.get_layer(my_Model_object).get_layer('input_1').input)[0]
and
model.input[0] print the same thing and are of the same type.


Where do I find the .csv file for the submission after creating it? I am making a submission of a classificaition problem with CNN on Google Colab. So I have arrived at doing this:

subm.to_csv('submission.csv', index=False)

so in theory I should have finished. The only problem is that I don't know where to find the newly created csv file.

Can somebody please help me? Thanks in advance.

When you are done with Colab instince files saves will not persist. You have to configure saving them to google drive. Try something like this;

  from google.colab import drive
  drive.mount('/content/gdrive', force_remount=True)
  root_dir = "/content/gdrive/My Drive/"
  base_dir = root_dir + 'fastai-v3/'

so now you can access and save files to google drive directly.


How do I deal with data that has only limited target values? I'm currently working on a small project using the D1NAMO dataset (1). I want to predict the glucose level (that is given in the dataset) based on several features: accelerometer data, heartbeat (ECG) data, breathing data and some other features. I have several users (006, 007, 008 etc) that all have a few days of data.
The problem is that the target value (glucose) is measured only once in 5 minutes, whereas the other measurements are done every few milliseconds. That means that in my data I have many rows without the target value. For example, for one user (008) I have 214 rows where I have a glucose level and around 60.000 rows where glucose is NaN. I want to use a model without the notion of time, so that could be anything such as FFNN, CNN, Decision Trees (XGBoost), ensembles etc.
Snapshot of my data:

My questions:

How should I deal with this problem where I have only few target values? Should I summarize all values in a window of 5 minutes to match with the target value?
Should I make a model on the instance level or population level?

Thanks in advance for any replies.
(1) https://www-sciencedirect-com.vu-nl.idm.oclc.org/science/article/pii/S2352914818301059

I would try two different approaches:

interpolate the missing values on a user level.
work with the sunset of rows for which we actually have the glucose level.

Then, I would compare the test accuracy of the model built with both methods. Remember that your test set has to be composed of rows for which you have the glucose level - you cannot build it with interpolated data, that would be cheating!


classifier predicts only one class I have a classification that has to predict three different classes: gcc,icc, clang. The prblem is that if I use a blind test set to do a submission, when I look athe the prediction I have on it I find that most of the predictions are only of one type. So I have the following situation:

My code is the following:

#pakages
import numpy as np
import pandas as pd
import json as j
import re
import nltk
from nltk.tokenize import word_tokenize

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import CountVectorizer

from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

#for visualizing data
import matplotlib.pyplot as plt
import seaborn as sns; sns.set(font_scale=1.2)

%matplotlib inline

print('Libraries imported.')

dataFrame = pd.read_json('training_dataset.jsonl',lines = True)
dataFrame

dataFrame['opcodes'] = dataFrame['instructions'].apply(lambda x:[i.split()[0] for i in x])

df = dataFrame[["opcodes", "compiler"]].copy()
df

df['opcodes'] = [" ".join(opcode) for opcode in df['opcodes'].values]

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)

df_x = df['opcodes']

X_all = tfidf_vectorizer.fit_transform(df_x)
y_all = dataFrame['compiler']

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all,
    test_size=0.2, random_state=15)

from sklearn import svm
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100).fit(X_train,y_train)

y_pred = clf.predict(X_test)

acc = clf.score(X_test, y_test)
print("Accuracy %.3f" %acc)

Now, using a random forest classifier I get an accuracy of 0.89, but the problem is that the model predicts mostly the class gcc, so I guess something is wrong.

I was almost soing a submission when I noticed this.

Does this happen because I didn't consider the fact that I have a multiclass classification problem? I really don't understand.

Can somebody please help me? Thank's in advance.

[EDIT] The dataset is balanced.

[EDIT 2] When try to do the prediction on the blind test set I do the following:

test = pd.read_json('test_dataset_blind.jsonl',lines = True)
test

test['instructions'] = [" ".join(opcode) for opcode in
test['instructions'].values]

tfidf_vectorizer=TfidfVectorizer(ngram_range=(2,2),min_df=2,max_df=0.5)
df_x_new = test['instructions']

X_new = tfidf_vectorizer.fit_transform(df_x)

new_pred_class = clf.predict(X_new)[:3000]
len(new_pred_class)

sub_compiler = pd.DataFrame({'instructions':
test['instructions'],'compiler':new_pred_class})
sub_compiler.to_csv('sub_compiler.csv')
sub_compiler.head()

[EDIT 4]anctually the problem seems to be on how I select the data to test. Infact, if I do new_pred_class = clf.predict(X_new)[-3000:], I get:

`
so this time mostly the clang are predicted.

I have 30000 rows in the original dataset and 3000 in the blind test set.

[EDIT 5]So I think I have to reorganize the select the new_pred_class taking random values from clf.predict(X_new)[:3000], so what I mean is that I should take 3000 random values. How could I do that? Thank's again.

EDIT 6] if I do type(X_new)  I have scipy.sparse.csr.csr_matrix

It is a common problem that - with unbalanced classes - some model tends to predict mostly the majority class. You could try to oversample the minority classes. In addition, RF tends to perform weak here. Boosting or NN are often able to recover more details, which can be important to predict the minority classes.

Edit: Okay, now that you clarified that you face a balanced problem, I guess your problem is the classifier. You could try multinominal logit with lasso regulation to „shrink“ irrelevant features (words). You should also consider Boosting or NN (maybe LSTM sequential).


Obtaining consistent one-hot encoding of train / production data I'm building an app that will require user input. Currently, on the training set, I run the following code, in which data is a pandas dataframe with a combination of categorical and numerical columns.

dummified_data = data.get_dummies()
train_data = dummified_data[:10000]
test_data = dummified_data[10000:12000]

Currently, I have a hand-written function that takes user-inputted data and transforms it into a format like dummy data. This doesn't seem sustainable as the number of columns/the size of my categorical variables grows.

Is there a way to dummify training data and production data consistently?

If I understand your question correctly you want to make sure that the order of the encoding is always the same. Have you tried sklearn- to be more specific sklearn.preprocessing.OneHotEncoder?

The way it works is that you fit (or fit_transform) on your training sample. Then you save the state of your encoder (for example you can pickle it). In production you then load this encoder and transform. This should then provide the same results.

If for some reason you want to stay away from scikit you can also create a dict from your training sample and then apply this dict on production - this should be possible with pandas built in functionality.


Building a summary string in a Pandas groupby (Possibly cross-tab or pivot-table question) I'm a n00b in Python Pandas.
I want to do a particular aggregation / grouping / cross-tab but I know so little of the terminology that I don't even know how to look this up.
But here's what I want.
Say I have a table like this
Bob, Oranges,  5
Bob, Apples,  10
Bob, Bananas, 12
Tim, Oranges,  3
Tim, Apples,  20
Tim, Bananas,  5

I want to groupby fruit to find the total for each type of fruit. But produce another field containing a string which has the details sorted by their value in another column.
So I want an output something like this.
Oranges,  8, "Bob(5), Tim(3)"
Apples,  30, "Tim(20), Bob(10)"
Bananas, 17, "Bob(12), Tim(5)"

Where the string aggregates the values from the names column in a list sorted by the associated numeric value.
I know that there isn't something out of the box to do this, but what is this kind of operation or aggregation or pivoting (where you take one of the columns and turn it to be "horizontal" if that makes any sense) actually called?
How would I go about implementing it in Pandas functions?

Say you have a dataframe of your data in this format:
df = pd.DataFrame({
    'name': ['Bob', 'Bob', 'Bob', 'Tim', 'Tim', 'Tim'],
    'fruit': ['Oranges', 'Apples', 'Bananas', 'Oranges', 'Apples', 'Bananas'],
    'num': [5, 10, 12, 3, 20, 5]
})

You can perform a groupby on fruit and aggregate the sum of the num field. After that you can apply a function on your aggregated dataframe that leverages values from your non-aggregated data like so:
df_agg = df.groupby('fruit').sum()
df_agg.reset_index(inplace=True)

def desc(x):
    d = []
    for idx, row in df.loc[df['fruit']==x].sort_values(['num'], ascending=False).iterrows():
        d.append(f"{row['name']}({row['num']})")
    return ', '.join(d)

df_agg['desc_str'] = df_agg['fruit'].apply(lambda x: desc(x))

This gives you the following df_agg that you are looking for:
    fruit   num desc_str
0   Apples  30  Tim(20), Bob(10)
1   Bananas 17  Bob(12), Tim(5)
2   Oranges 8   Bob(5), Tim(3)


On minimizing matrix norm (AB-C) Given A, B and C are matrices with dim(A) = m x n, dim(B) = n x p and dim (C) = m x p, the problem asks to evaluate

I need to learn $$\tilde{A}$$ such that $$\min_{\tilde{A}}||\tilde{A}^TB-C||$$

and $$\min_{\tilde{A}}||\tilde{A}-A||$$

In general, you can't. You can find a matrix that will minimize $||\tilde{A}^TB-C||$, and another that minimizes $||\tilde{A}-A||$, but, in general, the two matrices you find won't be the same. In fact, the matrix that solves the second problem is always $\tilde{A} = A$, and, if $||{A}^TB-C||$ is not the minimum of the first function, then your problem doesn't have a solution.

However, what would a machine learning practicioner do? Instead of finding a matrix that solves both of the problems (that might not exist), you can aim to solve a combination of the two problems. You can aim to find the matrix that minimizes
$$||\tilde{A}^TB-C||^2 + ||\tilde{A}-A||^2
$$
or, in general, the matrix that minimizes
$$||\tilde{A}^TB-C||^2 + \lambda ||\tilde{A}-A||^2
$$
for some $\lambda >0$. If $\lambda \approx 0$, then you are just solving the first problem, and if $\lambda >>0$, then you are just solving the second one.

And, this is just a quadratic optimization problem, that can be solved using gradient descent and will provide a unique global maximum.


Does the phenomenon of over-fitting of data varies with training algorithms? Suppose I have a dataset which I want to train using Neural network and SVM. Is it possible that with my dataset after training, the Neural network is overfit while the SVM is not? Like can a dataset be overfit for one training algorithms and not be overfit for another training algorithm? Is it even possible? Or if it is not overfit for one training algorithm, can be assume that it will not be overfit for other training algorithms too?

A model is over-fitting if it makes good predictions on a test set but bad predictions on new data. This is generally a good indication that the used model is too complex.

The complexity of the model is often quantified as the number of free parameters. These are the parameters that need to be set in order to fit the data. More parameters allow more flexibility in what can be expressed but also increase the change of over-fitting.

The type of model also restricts what functions can be learned. For example, linear models can only learn linear functions (really).

The simple answer to your question is yes. For example, a SVM with little parameters might fit data well while a ANN with many parameters might over-fit on the same data.

A fair comparison however compares models with the same number of parameters. But the answer remains yes. One model might be better suited than another to fit the intrinsic structure of the data. For example, on the same data, a decision tree with 100 nodes might over-fit, a linear model might under-fit and a ANN might work perfectly. It all depends on the underlying structure that you want to model.


Split text into phrases of a person and an operator I have about 5,000 texts without punctuation marks.
Each text is a conversation between an operator and a person.
For example: "Hello hello how can I help you how can I find out how much money I have in my account"

How can one separate phrases between a person and an operator using machine learning methods?

I think the task you're looking for is called "Punctuation restoration" or "Punctuation prediction". An example of this is described in this paper, or this other paper.
Unfortunately I am not aware of available per-trained model that you might apply to your dataset. Honestly I think that it is unlikely that something trained specifically for costumer service calls exists, unless you want to annotate your own dataset and train a model out of it.


Calculation of distance between samples in data mining I am confused about a little issue related to distance calculation. What I want to know is, while calculating the distance between samples in classification or regression, is the label or output class also used or the distance is calculated using all other attributes excluding the label attribute?

The way that various distances are often calculated in Data Mining is using the Euclidean distance. You can read about that further here. If I understand your question correctly, the answer is no. The Euclidean distance can only be calculated between two numerical points. Therefore it would not be possible to calculate the distance between a label and a numeric point.


Data augmentation parameters When I use data augmentation to increase the train dataset, should I use all augmentation techniques (parameters in keras)?
Which data augmentation parameters should use with flow_from_directory?

This entirely depends on your data!

Generally, the more augmentation, the more situations your model will be exposed to during training and therefore the more robust it will be when being tested on unseen data.

However, what if we for example were working on a model for self driving cars? Using the vertical_flip just doesn't make sense, because the car will (hopefully!) be er be driving along on its roof.

I would suggest starting with no augmentation and slowly adding one possibility at a time. For example, you record an accuracy of 80% with no augmentation. Then addfeaturewise_normalizatiom and featurewise_std_normalization giving you an accuracy of 85%. Then adding horizontal_flip gets you to 90%. Finally you try adding zca_whitening and that send you back down to 86%.

The reverse approach may also work well for you, starting with all augmentation parameters turned on and removing them one by one. In any case, it is completely dependent on your specific problem and your available data.  Keras' ImageDataGenerator has a long list of parameters, so having a think about what makes sense will save you a lot of time.


NLP Text Summarization - which metrics to use in evaluation? I'm trying to implement Text Summarization task using different algorithms and libraries. To evaluate which one gave the best result I need some metrics. I have read about the Bleu and Rouge metrics but as I have understand both of them need the human reference summaries as a reference. Is it so that no automatic evaluation can be done in text summarization, i.e human generated summary is needed beforehand? What is the common practice for the tasks like this? How often is the text summarization accuracy measured at all?

So, your question is talking about whether human reference summaries are required to evaluate summarisation models.

The short answer is yes at the moment. The most important things about an output summary that we need to assess are the following:

The fluency of the output text itself (related to the language model aspect of a summarisation model)
The coherence of the summary and how it reflects the longer input text.

The problem with have an automatic evaluation system for a text summarisation model is that, although we can assess fluency from a language model, we can't really assess whether the model has pulled "the most salient" pieces of information from the original, longer text (and this can subjective from person to person). Hence why we need multiple human reference summaries to compute ROUGE and BLEU. However, as you are aware, these metrics have their limitations.

ROUGE is essentially a further development of BLEU, which have been commonly used a dubious proxy for output text fluency in research to compare summarisation and translation models. These metrics are dubious because they simply look at how much they overlap with reference texts from humans (https://rxnlp.com/how-rouge-works-for-evaluation-of-summarization-tasks/#.Xt1ewy-ZOi4).


Are there established good algorithms for incremental feature learning for a neural network? Do any python ML libraries implement such algorithms? I'm working with a high-dimensional dataset, and have found that my attempts at dimensionality-reduction hurt the accuracy of downstream classifiers, suggesting that I am effectively losing information in the course of feature-compression.

Furthermore, when I sort the features by importance (per estimation via random forest) and then train neural networks on different numbers of the most important features, I find that the resulting classifier accuracy stops increasing before I have exhausted all of the useful-seeming features.  I suspect that simultaneous training on many thousands of features is the problem, and I'd like to simplify the problem by learning from the features incrementally, that is:

I'd like to explore training a neural network adding a few features at a time: e.g. train using the top 100 features, then add in the next 100, train again (using coefs from previous round plus random coefs for the new part of the input layer), rinse, repeat.

NOTE that this "incrementalness" is across features, not samples; it doesn't affect whether training is incremental ("online") with respect to samples.

I gather this sort of thing has been tried before with some success but I haven't been able to find implementations in the Python ML libraries I've checked, nor can I find good explanations of the details of successful algorithms.

I have about 70K examples, divided over 6 balanced classes. For most of my work I care only about distinctions between a total of 4 classes, so I merge two pairs of the initial classes and end up with a total of 4 (I end up with unbalanced training sets as a result, but since the two larger classes are the ones whose classification is most important, I'm fine with that for now). I have a total of about 28K features, but am finding that I get best classifier performance when using no more than the top 4K features (pre-sorted by importance via Random Forest).

Does my approach seem reasonable?
Are there accepted published detailed algorithms (e.g., is it reasonable to add the features as extensions of the original input layer, or is another approach preferred)?
Do any Python libs, particularly any compatible with scikit-learn, implement such an algorithm?

The opposite of what you describe is used more often. You start with many features and then prune the neural network at the end.

I don't believe any python neural network package does what you want.

But it's very simple to implement.

For example, using sklearn, you can just pass warm_start=True, so that it doesn't reinitialize weights every time you call fit(), and then you can increase the number of nodes in your feature layer, or any layer you wish.

from sklearn import datasets
from sklearn.neural_network import MLPClassifier
import numpy as np

X, y = datasets.load_iris(True)
X = (X-np.mean(X, 0))/np.std(X, 0)  # normalize

nhidden = 10
m = MLPClassifier(nhidden, max_iter=1000, warm_start=True).fit(X[:, :1], y)

# every 1,000 iterations, throw in a new feature
for nfeatures in range(2, X.shape[1]+1):
    m.coefs_[0] = np.r_[m.coefs_[0], np.random.randn(1, nhidden)]
    m.fit(X[:, :nfeatures], y)

# in the end, we have a neural network trained with all features
print(m.score(X, y))


How to make an MNIST classifier work with blank images? I am trying to make a Sudoku solver and for the image recognition I trained a CNN but the problem that I am facing is that I don't know how to make it see a clear distinction between numbers and blank images.
(My neural network is trained for MNIST data set only)
For example in a Sudoku like this :

I want the classifier to classify the blank spaces as "0"
Here is what I have already tried:
import numpy as np
import cv2
from PIL import Image
import pytesseract
import matplotlib.pyplot as plt
from tensorflow import keras

#open the image
img = Image.open(r'D:\\D_Apps\\Sudoku Solver\\image\\1_9Tgak3f8JPcn1u4-cSGYVw.png').convert('LA')

#take only the brightness value from each pixel of the image
array = np.array(img)[:,:,0]

#invert the image (this is how MNIST digits is formatted)
array = 255-array

#this will be the width and length of each sub-image
divisor = array.shape[0]//9

puzzle = []
for i in range(9):
    row = []
    for j in range(9):
        #slice image, reshape it to 28x28 (mnist reader size)
        row.append(cv2.resize(array[i*divisor:(i+1)*divisor,
                                    j*divisor:(j+1)*divisor][3:-3, 3:-3], #the 3:-3 slice removes the borders from each image
                              dsize=(28,28),
                              interpolation=cv2.INTER_CUBIC))
    puzzle.append(row)

model = keras.models.load_model(r'C:\Users\Ankit\MnistModel.h5')

template = [
    [0 for _ in range(9)] for _ in range(9)
]

for i, row in enumerate(puzzle):
    for j, image in enumerate(row):
        #if the brightness is above 6, then use the model
        if np.mean(image) > 6:
            #this line of code sets the puzzle's value to the model's prediction
            #the preprocessing happens inside the predict call
            template[i][j] = model.predict_classes(image.reshape(1,28,28,1) \
                                                   .astype('float32')/255)[0]

print(template)

(read about this in a blog)
This algorithm took the average brightness and checked if the other cells had brightness less than 2 and classified them as blanks.
But this algorithm dose not work if the image did not have a white background for example the output for this image:

the output was :
[[7, 7, 0, 7, 7, 7, 1, 7, 1], [2, 1, 8, 1, 8, 1, 1, 0, 8], [7, 7, 1, 8, 8, 1, 7, 7, 7], [7, 1, 1, 1, 1, 1, 1, 0, 1], [7, 7, 1, 1, 7, 8, 1, 7, 7], [8, 7, 8, 1, 7, 7, 7, 4, 9], [7, 1, 1, 1, 1, 0, 7, 8, 7], [7, 4, 7, 8, 8, 7, 7, 7, 4], [2, 7, 7, 7, 8, 0, 4, 7, 7]]

What can I do to improve this? Should I retrain my model to work with other image colors?
Or Should i retrain the model with blank spaces? If so how can I find the dataset?
I have done a lot of research but can't find a clear answer to my questions

Try thresholding on the image. I believe, you will get ~95% of what is required.
Then try other classical image processing techniques depending on the issue.
import cv2 as cv
import numpy as np
from matplotlib import pyplot as plt
img = cv.imread('/content/sample_data/issue_image.png',0)
ret,thresh1 = cv.threshold(img,127,255,cv.THRESH_BINARY)

_, ax = plt.subplots(1,2,figsize=(12,6))
ax[1].imshow(thresh1,'gray')
ax[0].imshow(img)

Ref - OpenCV py_thresholding


tensorflow: is there a way to specify XLA_GPU with tensorflow? following code is used to specify device on which tf node is running on

with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

i have already known:
this post, tensorflow doc and xla demo

what i want to know is:

is there a way to specify XLA_GPU as the device on which tf node is running on

with tf.device('/XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')

executing the code above gives ValueError: Unknown attribute: 'device' in '/job:localhost/replica:0/task:0/device:/XLA_GPU:0'

this is 100% reproducible on google colab.

try 'device:XLA_GPU:0'

with tf.device('device:XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')


Keras custom loss using multiple input I have a model in keras with a custom loss. this loss is calculated using actual and predicted labels(or values) and is also based on some input value.

(an example would be to define loss based on reward or advantage as in a policy gradient method in reinforcement learning context )

example code:
note: the input value used in loss calculation is named 'in2' and variable is input_2

input_1=keras.layers.Input(...,name="in1")
output= keras.layers.Dense(...)(input_1)

input_2=keras.layers.Input(...,name="in2")
def loss_func(y_true,y_pred):
    # some custom loss i define based on input_2
    loss = keras.layers.Dense(..)(input_2)
    return loss

my_model=keras.Model(inputs=[input_1,input_2],outputs=output)
my_model.compile(...,loss=loss_func,)

Now the problem is that when I want to just predict an output, I have to give a dummy input for the unused input_2. The input_2 is only used for computing the loss and it is not used for computing the model output. but when i use this code, an error ocures for not providing 'in2':

# this throws error for not providing input value for 'in2'
my_model.predict({'in1':some_value})

so I have to provide some dummy value for in2 so the model outputs its value, although it does not use the in2 value provided. note that providing None value does not work either. I thought this functionality is implemented inside keras.

or maybe I am doing it the wrong way? if yes, what is the appropriate way?

idea:
An idea came to my mind was that maybe we can define two models. one model for training and another for predicting:

my_model_for_training = keras.Model(inputs=[input_1,input_2],outputs=output)
my_model_for_predicting = keras.Model(inputs=[input_1],outputs=output)

and when I want to predict, I just use the second model which does not require the input_2 (or in2) as in example. both models share the same layers so when train the first model, the weights are updated for both models as explained in docs (section: Shared layers). is there a better solution?

When you are in the inference phase, there is no loss to be calculated unless you are using a validation set to fine tune your model on.
Using two models with shared layers in between is fine yet I suggest to start training the model and don't waste your time writing perfect code.


Correcting for one of multiple strong batch effects in a dataset I am wondering which statistical tools to use when analysing data that have multiple strong batch effects (distributions vary from one batch to another). I would like to correct batch effect when it originates from one variable, without taking off the potential batch effect from other variables.

If this is unclear, taking a short example is probably the best way to go to explain my problem:

Imagine that we have 10 persons taking part in an experiment. The experiment is as follows:

each person is given a set of a 1 000 numbered tennis balls, all of which we know the physical properties (for instance, weight, diameter and colour of the ball)
each person is then asked to throw the 1 000 tennis balls one by one as far as they can, and to record their results

Following the experiment, we will know for all 10 000 tennis ball:

the distance at which the ball was thrown
who threw the ball
the ball's weight
the ball's diameter
the ball's colour

Now, because not everyone has the same capabilities for throwing tennis balls (be it in terms of muscle strength or something else), we can expect to see some strong batch effects within the data (for instance, we could observe that a ball thrown by the first person will, on average, have been launched farther than a ball with the same weight and diameter when it's been thrown by the second participant, etc...).

Correcting for this kind of batch effect can be done in multiple ways if everyone had been given the same set of balls (in a setting with normal distributions, standardisation would probably work fine). Now, imagine that when organising the experiment, we did not pay enough attention and ended up giving some people heavier tennis balls, some others smaller tennis balls, etc...

At the end of the experiment, we realise indeed using a Chi2 test (or, say, Kruskal-Wallis H test) that everyone was not given a set of balls coming from a random sampling of all 10 000 balls.

How can we then correct for who threw the ball, without taking off the batch effects originating from the fact that the set of balls were different?

The main problem is that by correcting for batch effect using regular standardisation (for instance), we will probably end up removing the effect due to the fact that some people were given heavier or larger balls.

Or, in other words using the example, how could we account for the difference in terms of strength between the first and the second participant while not correcting for the fact that the first participant had in average heavier tennis balls than the second?

At first, I was thinking of running a Generalized Linear Model with the dependent variable being the distance at which the balls were thrown, and all the other variables as regressors, and then subtracting to the dependent variable only the effect of the variable for who threw the ball. I am however unsure of whether this would or not make statistical sense, which is why I ask if other techniques can be used, or if this one would work.

My first try to this would be to cluster my independent variables and use the labels in a supervised learning predictor. But I think this can be a good research problem beyond the scope of a stackoverflow short answer. I would recommend you to start with the Exceptional Model Mining paper.

Hope this helps!


Group K-fold with target stratification I have a pd.DataFrame
pd.DataFrame({
    'person': ['a', 'b', 'c', 'aa', 'bb', 'cc', 'aaa', 'bbb', 'ccc'],
    'group': [10, 10, 10, 20, 20, 20, 30, 30, 30],
    'target': [1, 2, 2, 3, 2, 3, 1, 2, 3]
})

I want to split df into train and test by group several times (K-Fold), so train and test contains examples from mutually exclusive group subsets. For example, if test contains examples from groups [10, 20] test should contain examples only from group [30]. I also want to keep stratification by target. So target distribution is also considered for groups subsets creation. How is it possible? Didn't find anything in scikit-learn that may help.
In practice I have a df with about 500k rows, 2.5k different persons, 30 groups and following target distribution

To use both groups and stratification, it seems that you should write your own code. Note that you will inevitably loose samples in training and/or in testing set (unless you are lucky).
One way of achieving it is to:

do your split by groups (you could use the GroupKFold method from sklearn)
check the distribution of the targets in training/testing sets.
randomly remove targets in training or testing set to balance the distributions.

Note: It is possible that a group disappear using such algorithm. You might prefer to not randomly remove the targets when balancing the training/testing sets.

Here is a sample code
import pandas as pd
import numpy as np
from sklearn.model_selection import GroupKFold

df = pd.DataFrame({
    'person': ['a', 'b', 'c', 'aa', 'bb', 'cc', 'aaa', 'bbb', 'ccc'],
    'group': [10, 10, 20, 20, 20, 20, 20, 30, 30],
    'target': [1, 2, 2, 3, 2, 3, 1, 2, 3]
})

X = df['person']
y = df['target']
groups = df['group'].values
group_kfold = GroupKFold(n_splits=3)
group_kfold.get_n_splits(X, y, groups)

# First split by groups
for train_index, test_index in group_kfold.split(X, y, groups):
    print("Groups split: TRAIN:", train_index, "TEST:", test_index)
    y_train_grouped, y_test_grouped = y[train_index], y[test_index]

    final_train_index = []
    final_test_index = []
    # Then balance the distributions for each target
    for target in df['target'].unique():
        target_train_index = y_train_grouped[y_train_grouped == target].index.tolist()
        target_test_index = y_test_grouped[y_test_grouped == target].index.tolist()
        n_training = len(target_train_index)
        n_testing = len(target_test_index)
        print("Target:" + str(target) + " - n_training:" + str(n_training) + " - n_testing:" + str(n_testing) +
              " | target_train_index:" + str(target_train_index) + " - target_test_index:" + str(target_test_index))

        # Shuffle to remove randomly
        np.random.shuffle(target_train_index)
        np.random.shuffle(target_test_index)

        # Check if we need to remove samples from training or testing set
        if n_training > n_testing:
            while n_training > n_testing:
                target_train_index.pop(0)
                n_training = len(target_train_index)
        if n_training < n_testing:
            while n_training < n_testing:
                target_test_index.pop(0)
                n_testing = len(target_test_index)

        # Append new indexes to global train/test indexes
        final_train_index.append(target_train_index)
        final_test_index.append(target_test_index)

    # Flatten for readability
    final_train_index = [item for sublist in final_train_index for item in sublist]
    final_test_index = [item for sublist in final_test_index for item in sublist]
    print("FINAL split: TRAIN:", final_train_index," TEST:", final_test_index, "\n")

EDIT
Using stratified cross-validation seems not to be mandatory (see links below), so you might reconsider using it.
You might find this links usefuls:

Why use stratified cross validation? Why does this not damage variance related benefit?
Understanding stratified cross-validation
A study of cross-validation and bootstrap for accuracy estimation and model selection

Regarding over/undersampling, I think it is hard to answer without more details of the data distribution and how imbalanced your classes are.


SAS procedure IF THEN ELSE loop did not recognize the Roman letters or numbers as value I have written the following SAS procedure to display Grades based on the salary:

Not Working: This displays only 'I' for all observations irrespective of the else condition that says any salary less than 35000 should have a Grade as 'II'

DATA Test (Keep = FirstName LastName Salary Grade);
set orion.sales;
if salary < 35000 then grade = 'I';    /* Did not interpret 'I' */
else grade = 'II';    /* Did not interpret 'II' */
run;

proc print data = test;
run;

Working:

DATA Test (Keep = FirstName LastName Salary Grade);
set orion.sales;
if salary < 35000 then grade = 'G1';    /* interprets 'G1' */
else grade = 'G2';    /* interprets 'G2' */
run;

proc print data = test;
run;

So basically, the SAS did not interpret the Roman letters or numbers. Does anyone has an idea of why this is happening or any workaround to display the Roman numbers or letters?

SAS initializes the length and type of the variable, I.e. creates and assigns a position for it in the program data vector, at the first reference to the variable it sees.

If (condition) then output='I'; defines it as length 1.

Workarounds

Your discovered solution of having the same lengths.
The first reference to the variable being LENGTH output $2;
Assigning the longest length first.
The first reference to the variable being If 0 then output='zz';

You can confirm the issue, which also pops up when concatenating datasets, by looking at variable properties with proc contents or proc datasets.

From the documentation:

  Note:   When you create character variables, SAS determines the length of the variable from its first occurrence in the DATA step. Therefore, you must allow for the longest possible value in the first statement that mentions the variable. If you do not assign the longest value the first time the variable is assigned, then data can be truncated.

http://support.sas.com/documentation/cdl/en/basess/58133/HTML/default/viewer.htm#a001336069.htm

Sample code:

data a;
    input firstname $ lastname $ salary;
    cards;
Billy Bob 20000
Bob Graham 50000
Graham Alice 30000
Alice Graham 40000
;

proc print data=a; run;

*Did not work;
DATA test;
set a;
if salary < 35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

*Worked;
DATA test;
length grade $ 2;
set a;
if salary<35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

*Worked;
DATA test;
set a;
if salary>=35000 then grade = 'II';
else grade = 'I';
run; proc print data = test; run;

*Worked;
DATA test;
if 0 then grade='ZZ';
set a;
if salary<35000 then grade = 'I';
else grade = 'II';
run; proc print data = test; run;

Also, as I stated, you can run PROC CONTENTS DATA=TEST; RUN; after each data step to confirm the data length of each column.


Predicting next number in a sequence - data analysis I am a machine learning newbie and I am working on a project where I'm given a sequence of integers all of which are in the range 0 to 70. My goal is to predict the next integer in the sequence given the previous 5 integers in the same sequence. There isn't much more information on the sequence of integers itself (for example, how was the sequence obtained, etc).

The following are the things I tried.

The first thing that came to mind was to use a LSTM regression model with 5 input time steps and one output (corresponding to the next integer in the sequence - in Keras this would be return_sequences=False). I passed 5 previous integers themselves as the input. This resulted in the model predicting pretty much the average (~30) all the time.
I tried the model in (1) above but with more input time steps (say 100), but still no improvement.
I then tried (1) and (2), but this time using the difference between consecutive integers as the input and trying to predict the difference to the next integer in the sequence. The results with this are still bad.
I then tried a LSTM classification model by one-hot encoding the input and output since I know that all integers in the sequence are in the range 0 to 70. Again, no improvement.
I then tried a seq2seq (encoder-decoder) LSTM model with 5 inputs in the encoder and 5 outputs in the decoder with the correct outputs also being fed into the decoder (teacher forcing). Still the results are bad.

At this point I started doubting whether I can train a model on the given data and whether the given data is just a bunch of random integers.

I looked for statistical tests to determine whether the data is random or not and found out about pandas autocorrelation plot. This is what the plot looks like when plotted on the difference between consecutive integers (looks similar when plotted using the actual integers themselves).

As I understand, since the values are very close to zero, it means that the data is random. Is that right ?

I also used statsmodels "plot_acf" and the following is the plot I got for the difference between consecutive integers.

I see that there is some negative correlation when the lag is 1. Why doesn't this show up in the plot using pandas' autocorrelation_plot()?

I tried building a AR (auto regression) model as well, but still the results are bad.

The histogram of the integers in the sequence also seems to suggest that the integers are random (all values have about the same count except some higher integers).

Am I wasting my time trying to build a machine learning model to predict the next integer in the sequence ?

Interesting problem. The pandas autocorrelation plot suggests that the data is random.

How much do you know about the source?  Is it believable that the sequence is, in fact, random?

Have you gone through the data and plotted a histogram of the integer appearance counts? Do they appear uniformly or are some more frequent than others?

One thing I think you should try is to reformulate your LSTM model. I don't think this is a regression problem, even though the target is a single integer. It's a classification problem with 70 classes. Try treating it that way and using a categorical crossentropy loss function. In such a scenario, ~1.5% accuracy would represent random guessing. Can the model do better than that?


Training Keras model with multiple CSV files I'm currently trying to train a Keras model on several large CSV files. I can fit one in memory, but not all combined. From my point of view, there are several ways to deal with this problem.
I could merge all of the data into a single CSV file and then read the CSV by the chunks of the data. This would simplify the process, however, I would not be able to shuffle the data each epoch.
Other approach that I can think of is to create a custom fit_generator. However I'm not sure how to actually implement this. Should I create a special generator for each dataset, and then cycle them? This would allow me to shuffle each dataset every epoch and even their order.
I believe that it would be more elegant to implement just a single generator, and manage all of the files inside. However to make this work, I would need to know the number of batches prior to training. That would require me to first get number of samples in each file, which would take a while. Also, I would need to sort out the issue of some lines being incorrect after reading the file with pandas. That limits me from using some fast os-level functions to get the number of lines.
Or is it possible to actually create a fit_generator without setting the number of batches prior to training? What approach would you recommend?

I would use tensorflow 2.0 with tf.data
import tensorflow as tf

filenames = ["filename1", "filename2", ...]

dataset = tf.data.Dataset.list_files(filenames, seed=42, shuffle=True)

# this reads 5 text files at a time, skips the first row of each file
dataset.interleave(lambda filename: tf.data.TextLineDataset(filename).skip(1), cycle_length=5, num_parallel_calls=tf.data.experimental.AUTOTUNE)

for line in dataset.take(5):
    print(line)

You can then pass in the dataset objects directly into tensorflow.keras for training.
There are some good examples here but please beware that this is NOT for tensorflow 2.0, and somethings may have changed


How to use data generator for regression keras? I am using the Keras data generator to load data from a directory. I am basically dealing with a regression i.e there is a numerical value for each of my images in the range [-1 1]. I am using the following code to load images from the directory:
df=pd.DataFrame(columns=['Image','Steer'])
df['Image']=filenames
df['Steer']=steer

from sklearn.model_selection import train_test_split
train_x,test_x,train_y,test_y=train_test_split(df['Image'], df['Steer'], test_size=0.2, random_state=1)

train_df=pd.DataFrame(columns=['id','steer'])
train_df['id']=train_x
train_df['steer']=train_y

test_df=pd.DataFrame(columns=['id','steer'])
test_df['id']=test_x
test_df['steer']=test_y

images_dir = '/content/original/compressed/All_data/'
train_datagen = ImageDataGenerator(rescale = 1./255)
train_generator = train_datagen.flow_from_dataframe(dataframe=train_df, directory=images_dir,
                                              x_col="id", y_col="steer", has_ext=True,
                                              class_mode="other", target_size=(110, 220),
                                              batch_size=32)

test_datagen = ImageDataGenerator(rescale = 1./255)
test_generator = test_datagen.flow_from_dataframe(dataframe=test_df, directory=images_dir,
                                              x_col="id", y_col="steer", has_ext=True,
                                              class_mode="other", target_size=(110, 220),
                                              batch_size=32)
  import numpy as np
import pandas as pd

from skimage.io import imread
from skimage.transform import resize

import keras
from keras.models import Sequential, Model, load_model
from keras.layers import Input, Conv1D, Conv2D, MaxPooling1D, MaxPooling2D, Dense, Dropout, Activation, Flatten
from keras.layers.normalization import BatchNormalization
from keras.utils import to_categorical
from skimage.transform import resize
from tensorflow.keras import layers
import tensorflow as tf

  from keras.applications import VGG16

# include top should be False to remove the softmax layer
pretrained_model = VGG16(include_top=False, weights='imagenet')

base_model = VGG16(input_shape = (110, 220, 3), # Shape of our images
include_top = False, # Leave out the last fully connected layer
weights = 'imagenet')

for layer in base_model.layers:
    layer.trainable = False

from tensorflow.keras import layers
import tensorflow as tf

# Flatten the output layer to 1 dimension
x = layers.Flatten()(base_model.output)

# Add a fully connected layer with 512 hidden units and ReLU activation
x = layers.Dense(512, activation='relu')(x)

# Add a dropout rate of 0.5
x = layers.Dropout(0.5)(x)

# Add a final sigmoid layer for classification
x = layers.Dense(1, activation='linear')(x)

model = tf.keras.models.Model(base_model.input, x)
opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss = 'mean_absolute_percentage_error',optimizer=opt,metrics = ['mae'])

batch_size=128
history=model.fit(train_generator,
                   epochs = 15,
                   verbose = 1,
                   validation_data = test_generator
                  )

When I train the model using this code, a value of 0.221 is returned for the MAE for each epoch no matter what I do. I have changed the activation functions, optimizers, learning rate, etc but the result is the same every time. Am I loading in the data in a correct way?

When you are using a pre-trained model, you should use it's specific pre-processing function,
Below is an example for resnet50.
from keras.applications.resnet50 import preprocess_input
train_datagen = ImageDataGenerator(#rescale = 1./255
                                    preprocessing_function=preprocess_input)

Always display/print your image/label for a sanity check.
You may use the below snippet.
#Quick check the DataGenerator

import matplotlib.pyplot as plt
for image, label in train_generator:
    plt.imshow(image[0])
    print(label[0])
    break;


Implementing Complementary Naive Bayes in python? Problem
I have tried using Naive bayes on a labeled data set of crime data but got really poor results (7% accuracy). Naive Bayes runs much faster than other alogorithms I've been using so I wanted to try finding out why the score was so low.
Research
After reading I found that Naive bayes should be used with balanced datasets because it has a bias for classes with higher frequency. Since my data is unbalanced I wanted to try using the Complementary Naive Bayes since it is specifically made for dealing with data skews. In the paper that describes the process, the application is for text classification but I don't see why the technique wouldn't work in other situations. You can find the paper I'm referring to here. In short the idea is to use weights based on the occurences where a class doesn't show up.
After doing some research I was able to find an implementation in Java but unfortunately I don't know any Java and I just don't understand the algorithm well enough to implement myself.
Question
where I can find an implementation in python? If that doesn't exist how should I go about implementing it myself?

Naive Bayes should be able to handle imbalanced datasets. Recall that the Bayes formula is

$$P(y \mid x) = \cfrac{P(x \mid y) \, P(y)}{P(x)} \propto P(x \mid y) \, P(y)$$

So $P(x \mid y) \, P(y)$ takes the prior $P(y)$ into account.

In your case maybe you overfit and need some smoothing? You can start with +1 smoothing and see if it gives any improvements. In python, when using numpy, I'd implement the smoothing this way:

table = # counts for each feature
PT = (table + 1) / (table + 1).sum(axis=1, keepdims=1)

Note that this is gives you Multinomial Naive Bayes - which applies only to categorical data.

I can also suggest the following link: http://www.itshared.org/2015/03/naive-bayes-on-apache-flink.html. It's about implementing Naive Bayes on Apache Flink. While it's Java, maybe it'll give you some theory you need to understand the algorithm better.


Why does GPU speed up inference? I understand that GPU can speed up training for each batch multiple data records can be fed to the network which can be parallelized for computation. However, for inference, typically, each time the network only processes one record, for instance, for text classification, only one text (i.e., a tweet) is fed to the network. In such a case, how can GPU speed up?

Although what you describe is correct, such online/real-time usage is far from being the only (or even the most frequent) use case for DL inference. The keyword here is "batch"; there are several applications where the inference can be also run in batches of incoming data instead of on single samples.
Take the example mentioned by NVIDIA in their AI Inference Platform technical overview (p.3):

Inference also batch hundreds of samples to achieve optimal throughput on jobs run overnight in data centers to process substantial amounts of stored data. These jobs tend to emphasize throughput over latency. However, for real-time usages, high batch sizes also carry a latency penalty. For these usages, lower batch sizes (as low as a single sample) are used, trading off throughput for lowest latency. A hybrid approach, sometimes referred to as “auto-batching,” sets a time threshold—say, 10 milliseconds (ms)—and batches as many samples as possible within those 10ms before sending them on for inference. This approach achieves better throughput while maintaining a set latency amount.

Although, as correctly pointed out in another thread comment, it is on NVIDIA's best interest to convince you that you need GPUs for inference (which is indeed not always true), hopefully you can see the pattern: whenever we want to emphasize throughput over latency, GPUs will be useful for speeding up inference.
Practically, any application that runs on existing archives of data (videos, audio, music, text, documents) instead of waiting for incoming streams in real-time can meaningfully rely on GPUs for inference. And here "archives" does not imply necessarily time spans of years or months (although it can be so, e.g. in astronomy applications); the archive consisting of the photos uploaded to Facebook in the last 3 minutes (or since I have started writing this...) is huge, and it, too, can benefit from GPU-accelerated inference.
Videos, in specific, since they are usually broke up in frames to be processed, may benefit from sped up GPU inference even in near-real time applications.
Bu if you want to just set up your web app that will process low-traffic incoming photos or tweets to respond in real-time, then indeed GPU may not offer anything substantial in performance.


Has the Random Forest algorithm ever been used in Reinforcement Learning applications? I've seen a research paper describing a "Reinforcement Learning Tree", which the authors say that it has a better convergence than random forests.

However, I couldn't find anything related to utilizing random forests for a table in Reinforcement Learning. Have tests ever been conducted on using random forests in reinforcement learning?

I expect someone somewhere has used a RF estimator inside RL to approximate action values, if only to assess it as a comparison to other function approximators. However, it does look like from a web search that this is not used widely, and I could not find an example either.

The main problem with a RL/RF hybrid with RF as the value estimator is that the random forest base algorithm is not an online one - it works across a finalised data set and processes the whole batch in order to do things like bagging. Even when used as an estimator using experience replay, support for online learning is a desirable feature, and values are typically fed into supervised learning part in small or medium batches. That is because the action values that are learned by the internal estimation function in reinforcement learning are non-stationary.

Once you have learned the action values for the current policy, and in most RL well before estimates for the values have converged, you change the policy. This changes the expected action values, so your estimator must be able to forget older data and bias towards most recent values. Algorithms that can be made to work online can do this, purely offline ones cannot.

However, there are online Random Forests which have the necessary alterations to work with non-stationary data. I don't see any reason why they could not work. Decision trees in general have been used successfully as estimators in RL - see Reinforcement Learning with Decision Trees. It looks like that in principle it could be done.


How to use Scikit-Learn Label Propagation on graph structured data? As part of my research, I am interested in performing label propagation on a graph. I am especially interested in those two methods:

Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002 http://pages.cs.wisc.edu/~jerryzhu/pub/CMU-CALD-02-107.pdf
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schoelkopf. Learning with local and global consistency (2004) http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.115.3219

I saw that scikit-learn offer a model to do that. However, this model is supposed to be applied on vector structured data (i.e. data points).

The model builds an affinity matrix from the data points using a kernel, and then run the algorithm on the constructed matrix. I would like to be able to directly input the adjacency matrix of my graph in place of the similarity matrix.

Any idea on how to achieve that? Or do you know any Python library that will allow to run label propagation directly on graph structured data for the two aforementioned methods?

Thanks in advance for your help!

Answering my own question here, as I hope it will be useful to some readers.

Scikit-learn is primarily designed to deal with vector structured data. Hence, if you want to perform label propagation/label spreading on graph-structured data, you're probably better off reimplementing the method yourself rather than using Scikit interface.

Here is an implementation of Label Propagation and Label Spreading in PyTorch.

The two methods overall follow the same algorithmic steps, with variations on how the adjacency matrix is normalized and how the labels are propagated at each step. Let's, therefore, create a base class for our two models.

from abc import abstractmethod
import torch

class BaseLabelPropagation:
    """Base class for label propagation models.

    Parameters
    ----------
    adj_matrix: torch.FloatTensor
        Adjacency matrix of the graph.
    """
    def __init__(self, adj_matrix):
        self.norm_adj_matrix = self._normalize(adj_matrix)
        self.n_nodes = adj_matrix.size(0)
        self.one_hot_labels = None
        self.n_classes = None
        self.labeled_mask = None
        self.predictions = None

    @staticmethod
    @abstractmethod
    def _normalize(adj_matrix):
        raise NotImplementedError("_normalize must be implemented")

    @abstractmethod
    def _propagate(self):
        raise NotImplementedError("_propagate must be implemented")

    def _one_hot_encode(self, labels):
        # Get the number of classes
        classes = torch.unique(labels)
        classes = classes[classes != -1]
        self.n_classes = classes.size(0)

        # One-hot encode labeled data instances and zero rows corresponding to unlabeled instances
        unlabeled_mask = (labels == -1)
        labels = labels.clone()  # defensive copying
        labels[unlabeled_mask] = 0
        self.one_hot_labels = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)
        self.one_hot_labels = self.one_hot_labels.scatter(1, labels.unsqueeze(1), 1)
        self.one_hot_labels[unlabeled_mask, 0] = 0

        self.labeled_mask = ~unlabeled_mask

    def fit(self, labels, max_iter, tol):
        """Fits a semi-supervised learning label propagation model.

        labels: torch.LongTensor
            Tensor of size n_nodes indicating the class number of each node.
            Unlabeled nodes are denoted with -1.
        max_iter: int
            Maximum number of iterations allowed.
        tol: float
            Convergence tolerance: threshold to consider the system at steady state.
        """
        self._one_hot_encode(labels)

        self.predictions = self.one_hot_labels.clone()
        prev_predictions = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)

        for i in range(max_iter):
            # Stop iterations if the system is considered at a steady state
            variation = torch.abs(self.predictions - prev_predictions).sum().item()

            if variation < tol:
                print(f"The method stopped after {i} iterations, variation={variation:.4f}.")
                break

            prev_predictions = self.predictions
            self._propagate()

    def predict(self):
        return self.predictions

    def predict_classes(self):
        return self.predictions.max(dim=1).indices

The model takes as input the adjacency matrix of the graph as well as the labels of the nodes. The labels are in the form of a vector of an integer indicating the class number of each node with a -1 at the position of unlabeled nodes.

Label Propagation algorithm is presented below.

$$
\begin{array}{l}{
\mathbf{W} \text {: adjacency matrix of the graph} \\
\text { Compute the diagonal degree matrix } \mathbf{D} \text { by } \mathbf{D}_{i i} \leftarrow \sum_{j} W_{i j}} \\ {\text { Initialize } \hat{Y}^{(0)} \leftarrow\left(y_{1}, \ldots, y_{l}, 0,0, \ldots, 0\right)} \\ {\text { Iterate }} \\ {\text { 1. } \hat{Y}^{(t+1)} \leftarrow \mathbf{D}^{-1} \mathbf{W} \hat{Y}^{(t)}} \\ {\text { 2. } \hat{Y}_{l}^{(t+1)} \leftarrow Y_{l}} \\ {\text { until convergence to } \hat{Y}^{(\infty)}} \\ {\text { Label point } x_{i} \text { by the sign of } \hat{y}_{i}^{(\infty)}}\end{array}
$$

From Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002

We get the following implementation.

class LabelPropagation(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1 * W"""
        degs = adj_matrix.sum(dim=1)
        degs[degs == 0] = 1  # avoid division by 0 error
        return adj_matrix / degs[:, None]

    def _propagate(self):
        self.predictions = torch.matmul(self.norm_adj_matrix, self.predictions)

        # Put back already known labels
        self.predictions[self.labeled_mask] = self.one_hot_labels[self.labeled_mask]

    def fit(self, labels, max_iter=1000, tol=1e-3):
        super().fit(labels, max_iter, tol)

Label Spreading algorithm is:

$$
\begin{array}{l}{
\mathbf{W} \text {: adjacency matrix of the graph} \\ \text { Compute the diagonal degree matrix } \mathbf{D} \text { by } \mathbf{D}_{i i} \leftarrow \sum_{j} W_{i j}} \\ {\text { Compute the normalized graph Laplacian } \\ \mathcal{L} \leftarrow \mathbf{D}^{-1 / 2} \mathbf{W} \mathbf{D}^{-1 / 2}} \\ {\text { Initialize } \hat{Y}^{(0)} \leftarrow\left(y_{1}, \ldots, y_{l}, 0,0, \ldots, 0\right)} \\ {\text { Choose a parameter } \alpha \in[0,1)} \\ {\text { Iterate } \hat{Y}(t+1) \leftarrow \alpha \mathcal{L} \hat{Y}^{(t)}+(1-\alpha) \hat{Y}^{(0)} \text { until convergence to } \hat{Y}^{(\infty)}} \\ {\text { Label point } x_{i} \text { by the sign of } \hat{y}_{i}^{(\infty)}}
\end{array}
$$

From Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schoelkopf. Learning with local and global consistency (2004)

The implementation is, therefore, the following.

class LabelSpreading(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)
        self.alpha = None

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1/2 * W * D^-1/2"""
        degs = adj_matrix.sum(dim=1)
        norm = torch.pow(degs, -0.5)
        norm[torch.isinf(norm)] = 1
        return adj_matrix * norm[:, None] * norm[None, :]

    def _propagate(self):
        self.predictions = (
            self.alpha * torch.matmul(self.norm_adj_matrix, self.predictions)
            + (1 - self.alpha) * self.one_hot_labels
        )

    def fit(self, labels, max_iter=1000, tol=1e-3, alpha=0.5):
        """
        Parameters
        ----------
        alpha: float
            Clamping factor.
        """
        self.alpha = alpha
        super().fit(labels, max_iter, tol)

Let's now test our propagation models on synthetic data. To do so, we choose to use a caveman graph.

import pandas as pd
import numpy as np
import networkx as nx
import matplotlib.pyplot as plt

# Create caveman graph
n_cliques = 4
size_cliques = 10
caveman_graph = nx.connected_caveman_graph(n_cliques, size_cliques)
adj_matrix = nx.adjacency_matrix(caveman_graph).toarray()

# Create labels
labels = np.full(n_cliques * size_cliques, -1.)

# Only one node per clique is labeled. Each clique belongs to a different class.
labels[0] = 0
labels[size_cliques] = 1
labels[size_cliques * 2] = 2
labels[size_cliques * 3] = 3

# Create input tensors
adj_matrix_t = torch.FloatTensor(adj_matrix)
labels_t = torch.LongTensor(labels)

# Learn with Label Propagation
label_propagation = LabelPropagation(adj_matrix_t)
label_propagation.fit(labels_t)
label_propagation_output_labels = label_propagation.predict_classes()

# Learn with Label Spreading
label_spreading = LabelSpreading(adj_matrix_t)
label_spreading.fit(labels_t, alpha=0.8)
label_spreading_output_labels = label_spreading.predict_classes()

# Plot graphs
color_map = {-1: "orange", 0: "blue", 1: "green", 2: "red", 3: "cyan"}
input_labels_colors = [color_map[l] for l in labels]
lprop_labels_colors = [color_map[l] for l in label_propagation_output_labels.numpy()]
lspread_labels_colors = [color_map[l] for l in label_spreading_output_labels.numpy()]

plt.figure(figsize=(14, 6))
ax1 = plt.subplot(1, 4, 1)
ax2 = plt.subplot(1, 4, 2)
ax3 = plt.subplot(1, 4, 3)

ax1.title.set_text("Raw data (4 classes)")
ax2.title.set_text("Label Propagation")
ax3.title.set_text("Label Spreading")

pos = nx.spring_layout(caveman_graph)
nx.draw(caveman_graph, ax=ax1, pos=pos, node_color=input_labels_colors, node_size=50)
nx.draw(caveman_graph, ax=ax2, pos=pos, node_color=lprop_labels_colors, node_size=50)
nx.draw(caveman_graph, ax=ax3, pos=pos, node_color=lspread_labels_colors, node_size=50)

# Legend
ax4 = plt.subplot(1, 4, 4)
ax4.axis("off")
legend_colors = ["orange", "blue", "green", "red", "cyan"]
legend_labels = ["unlabeled", "class 0", "class 1", "class 2", "class 3"]
dummy_legend = [ax4.plot([], [], ls='-', c=c)[0] for c in legend_colors]
plt.legend(dummy_legend, legend_labels)

plt.show()

The implemented models work correctly and allow to detect the communities in the graph.

Note: The propagation methods presented are meant to be used on undirected graphs.

The code is available as an interactive Jupyter notebook here.


how to make decision based on users reports I have users reports about an accident, i want to know how to make sure that the number of reports is enough to take that accident as a true accident not a spam.

My idea is to consider a minimum number of reports in a specific time interval, for example 4 reports in 20 minutes are good enough to believe the existence of that accident.

My question is how can I choose the number of minimum reports and that time interval? Is there another logic to take that decision?
I will appreciate your answers .

You don't need a prediction model for this. Maybe if you have had users' data. But without anything else, then you just need labeled data. Historical data that you know if it was a real accident or not.

When you have your labeled data, then you can follow a process like this, which is still heavily dependent on the kind of your data.

Start iterating on your labeled dataset and calculate the accuracy of a real accident's report for different combinations (5, 10, 15, 20, 25, 30 ... mins) and (1, 2, 3, 4, 5, 6, 7, etc users).

You will have a 2D matrix. I guess, acting fast on an accident is important in your case. Set an acceptable accuracy and choose the combination with the smallest interval, above that accuracy.


What is the name for this chart which splits a quantity by allocation to a class hierarchy? I need to create this type of chart, where you can see a quantity is split into a hierarchy of classes or taxonomy. In the example below the quantity is a household monthly income, and the classes are different monthly expenditures, classified broadly to the left, and progressively split into more detailed sub-classes.

However, I don't know what it is called and thus cannot find which tool to use.

This is called a Sankey-Diagram:
"Sankey diagrams are a specific type of flow diagram, in which the width of the arrows is shown proportionally to the flow quantity." (wiki

An easy package to make these graphs would be "d3Network" in R. But many other options are available.