Comparing multi-class results with binary classification results We used machine learning to discriminate the following five disease classes:

Normal (N)
Myocardial Infarction (MI)
Coronary Artery Disease (CAD)
Congestive Heart Failure (CHF).

In the past, these diseases were only discriminated with binary classifiers: N and MI, N and CAD, N and CHF.

What is the best way of comparing the multi-class results with the three individual binary classification results?

Access to the confusion matrix is available for all cases.

(As everything in life) It depends.

Often, a single person (a sample) can be within more than a single class. You do have a set of 3 binary classifiers that are known, these are understood as three models to which we can ask the questions:

classifier 1, do this person have MI?
classifier 2, do this person have CAD?
classifier 3, do this person have CHF?

What is not obvious from the above is that (ignoring any physical constraints - I'm no expect in heart diseases) a person may have MI, CAD and CHF at once.  Or just two of them at once, or only one, or none of them.

If you build a classifier that will say that a person has: MI OR CAD OR CHF OR nothing (N); then the binary classifiers cannot be compared against that new classifier because the three classifiers produce a different set of results from the multiclass classifier.  But not all is lost, let's work a couple of options that can be explored:

There are physical limitations that allow us to assume that the classes are disjoint

If one can argue that it is physically impossible for someone to have two of: MI, CAD or CHF at once then we can compare the classifiers.  Note that this is an assumption that needs a proof outside of the ML process: e.g. the definition of the diseases may exclude the possibility of having them at once.

In this case we need to deal with the case where multiple binary classifiers may output a positive class for the same sample.  For example if both the MI and CAD classifiers outputs the positive class for a sample then we need to select either MI or CAD as the true classification.  Most binary classifiers can output (or at least estimate) class probabilities, one can use these probabilities to decide for the winning classifier (the output with the higher probability).

(Note: If the classifiers are different model types from each other it is likely that they will require probability calibration but this is beyond this answer.)

With this process you then have a simple set of 4 classes (N, MI, CAD, CHF) for all samples in the dataset.  You can use this set to compare against the output of the multiclass classifier.

It is (physically) valid for a person/sample to be in multiple classes at once

In this case the multiclass classifier is not classifying four classes, instead it needs to classify eight classes (N, MI, CAD, CHF, MI+CAD, MI+CHF, CAD+CHF, MI+CAD+CHF).

Finding the samples for each of these classes is likely to be difficult (and it is likely you will not find any samples at all for some of them).  Moreover you will need to rebuild your model to include more classes.

On the other hand this is the perfect case for the use of a multiclass and multilabel classifier.  Instead of creating the eight classes (which will likely be unbalanced) several models can predict multiple classes at once.  Notably Decision Tress (and forests) and Neural Nets are good at multiclass with multilabel classification.

If you can go for the multilabel classification you will have matrices as output of your model, for example for some 6 samples you may get:

MI  CAD CHF
0   0   0
1   0   0
1   1   0
0   0   1
0   1   1
0   0   0

This can be directly compare against the output of all three binary classifiers on the same 6 samples.

If you go for the 8 classes (and I strongly advise against that - it is always a pain to deal with very unballanced classes) you can directly compare the output of the binary classifiers against the list of results.  (e.g. class MI+CAD means that both the 1st and 2nd binary classifiers did output a positive result for a sample).

P.S. Why one would prefer multilabel classifiers over building the classes by hand?  Multilabel classifiers allow one to predict multiple classes at once, which do work as a "new class", yet they perform this as part of their decision algorithm.  As a rule of thumb a multilabel classifier is less likely to impact the separability of other classes by adding this "new class" as a combination of existing classes.  A new class built by hand can affect the separability of the other classes if the algorithm (or hyperparameters) weight classes based on their (low) support (number of elements in class against total number of samples).


Neural network for Multiple integer output I have a data set that contains 135 input features and 132 output values to be predicted. The input features are all numeric floating point values and each output value would be an integer between [0,1,2,3,4].

I am basically new to data science and machine learning, therefore I need to understand what kind of neural network model (regression or classification) would fit best for this kind of data. On one hand the output values represent different classes but I am not sure how a single neural network can predict multiple classes from the same input data.

welcome to the site!

I think the key word you need to know that defines your task is: multi-target classification or regression.

You can find an explanation and some possible techniques at this link.

For neural networks:

The key is to remember that the last layer should have linear activations (i.e. no activation at all).

As per your requirements, the shape of the input layer would be a vector (135,) and the output (132,).

The usual loss function used for regression problems is mean squared error (MSE). Here's an example of multidimensional regression using Keras:

model = Sequential()
model.add(Dense(200, input_dim = (135,)))
model.add(Activation('relu'))
model.add(Dense(200))
model.add(Activation('relu'))
model.add(Dropout(0.3))
model.add(Dense(132))
model.compile(loss='mean_absolute_error', optimizer='Adam')


Manual Data Cleanup Tools I am writing an ETL pipeline for geospatial data of the form
place_name,address,longitude,latitude,id_linking_to_other_dataset
As the last step in the pipeline, I would like to apply manual transformations submitted by reviewers. Some of these transformations might be (borrowing from Google maps suggest edits docs):

Change a place's name, location, or the id linking it to another dataset
Mark a place private or non-existent
Mark a place as moved or duplicated

I don't have a ton of records (about 5000) but would like to manage this manual correction using best-practices. Ideally, corrections could be version controlled and applied as the last step in an ETL pipeline (even if other parts of the pipeline change).
There are lots of good tools handing off data annotation for ML, but I'm not seeing resources for this type of correction. Thoughts on useful tools?

i would recommend loading it into a dataframe and then use standard pandas functionality.
(str.replace,loc,iloc)
My answer is a bit vague, sorry for that but I would need to know a bit more of the technical details about your ETL pipeline and which format the data is which you want to change.


Keras functional api explanation of activation() layer? With the keras functional api it is possible to write something like this:

x = Activation('relu')(x)
x = Dense(8, activation='softmax')(x)

My question is whether the Activation() function is a separate Layer (equivalent to Dense(128, activation='relu') or if not... why and when this notation is used?

As stated in the docs, the activation layer in keras is equivalent to a dense layer with the same activation passed as an argument.

This would be equivalent

x = Dense(64)(x)
x = Activation('relu')(x)

is equivalent to

x = Dense(8, activation='relu')(x)

As per your example if the activation layer is used as a layer, this will act as a transformation of the outputs of the previous layer.

As you can see in the model

inputs = Input(shape=(784,))
x = Dense(32, activation='tanh')(inputs)
x = Activation('relu')(x)
predictions = Dense(8, activation='softmax')(x)

The output of the first layer is the result of a densely connected layer with a tanh function. Then these outputs will each be transformed by a relu function. You can see how this is simply a transformation and does not introduce any new model parameters using

model.summary()


Math behind 2D convolution for RGB images I read many threads discussing why 2D convolutional layer is typically used for RGB images in neural network.
I read that it is possible to use 3D conv layer.
What I do not understand is the math behind it.
Say your image is 300 by 300, and the kernel_size = (3, 3) and filter = 16 for the Conv2D layer. Input_shape would be (300, 300, 3) because there are 3 channels(RGB).

Since the kernel is 2D, the convolution can only be done at 1 channel at a time. Is that correct?
Are the same kernel applied/convolved for the 3 channels? If so there should be 3 output but the dimension of the output would be (298, 298, 16). Is it averaged over the 3 channels?

If your image is 3D then your kernel should be 3D too. Of course, you can also apply the 2D in which the same filter will be applied to all channels.
Image Source (Content is also well).

However, normally you apply a 3D filter to a 3D image. So if you apply 16 filters of size 3x3x3 to an image of size 6x6x6, then you will get 16 outputs of size 4x4. If you would apply 16 filters of size 3x3 filters, you would get 16 outputs of size 4x4x3. It would be treating each channel separately. But when you use a 3D filter, your output of convolution operation depends on all three dimensions. In other words, you multiply your 27 points from your 3x3x3 filter with the corresponding 27 points (3x3 pixels and their 3 channels) from the image, and then add them to get the result. Thus, 1 more dimension would be there for you to handle (16x4x4x3 instead of 16x4x4).
The answer to your question 1 is Yes, you would apply the filter 1 channel at a time.
Check the link for a very good explanation by Andrew NG.


How to properly use oversampling without inflating results? I am using with a tiny private dataset (over 192 samples) with 4 classes. A preprocessing step is trivial in order to do any classification. Among feature selection and extraction techniques, i decided to apply oversampling (SMOTE). Here is what i did:

Using the entire dataset (original 192 samples):
Create synthetic samples for each class using SMOTE, so i get a total of 500 samples per class (2000 in total)

I have a big suspicion about this procedure because when i apply SMOTE i get very high accuracy rates (even 100% in some cases) with the simplest models such as a 15-neuron MLP. So, i have some questions to assert the correctness of my experiments:

Is it ok to oversample the entire dataset or should i apply the SMOTE only in train data (keep in mind that this will leave few samples for test)?

SMOTE is originally used to deal with imbalanced datasets, creating samples for the classes with few sample. Is it ok to use it for generating samples for ALL classes in order to enlarge the entire dataset?

First of all - have you tried to do modeling without oversampling? If you dont have imbalanced classes, why do you need oversampling? You dont always need tons of data to get a good model as a baseline.
Regarding should i apply the SMOTE only in train data - always leave some test data out, dont apply any transformation on the whole dataset (even normalization or missing values imputing). If you have some preprocessing - do it on train data set, train model. Then apply the preprocessing pipeline on test data and predict on it.
simplest models such as a 15-neuron MLP. - it doesnt sound as simplest solution to me. have you tried any other approaches? also, have you checked confusion matrix to see the classification error on test data for each of the class?


Is an image with 5 labels equivalent to 5 images with 1 label? I am collecting data to train an object detection model using and was wondering if 5 labels in the same image and 5 images with 1 label each provided the same quality of input training data. Example: an image with 5 labeled apples vs. 5 images with 1 apple each.

No. These are two separate problems. Multi-label classification and multi-class classification. In general, when we talk about classification we mean multi-class classification i.e. there are a certain number of categories and the input training samples fall into only one of these which is your case of 5 images with 1 label. In the case of 1 image with 5 labels, it is a multi-label classification, which is a generalization of multi-class classification. Both of these are different tasks and the use case depends on the problem that you are trying to solve!


COVID: How is the R coefficient calculated? I'm having a hard time trying to figure out what the formula for estimating R is.
I assume it's a coefficient used in some sort of regression type?
If that is the case is it an exponential, logarithmic or another type of regression from which I extrapolate this coefficient?
Thanks, hope someone can help this ignoramous :)

This article provides a good explanation.
Realistic models are much more complicated, but the basic ideas are:
$R_0$ is the basic reproductive number. If you put an infected individual in a population that is fully susceptible to the disease, then that individual is expected to infect $R_0$ people.
$R_t$ is the effective reproductive number (same as $R_e$). People include the subscript to distinguish $R_t$ from $R_0$ because $R$ is ambiguous. In a population $P$ at time $t$, an infected individual causes $R_t$ new infections on average. Therefore, given the susceptible population $S\subseteq P$ (i.e. the people who are not immune at time $t$), we have $R_t=R_0\frac{|S|}{|P|}$. Again, the actual $R_t$ is affected by many other factors, but if we only considered $R_0$ and the fraction of susceptible people in a population at a given time, then this is the interpretation of $R_t$.
Therefore, the growth ($R_t>1$) or decay ($R_t<1$) in the number of cases is exponential.

For live updates on COVID $R_t$ values in the US per state, you can check out: https://rt.live


How to generate bulk graphics using R I have a dataset of Key Performance Indicator (KPI) and for each KPI I have a current level of achivement and 2 targets : Target1 and Target2.

How can I automatically generate one graphic for each achievement as in the file attached here.

As I have many KPIs I would like to generate my graphics in a batch process

There are multiple ways you can achieve this. I guess the easiest would be to create a function that saves a bar chart for your KPI to a file:

save_KPI_plot <- function(fn, kpi_data) {
    png(paste0(fn, ".png"))
    # Plot code
    dev.off()
}

You can call the function either as follows,

save_KPI_plot("kpi_1", kpi_1_data)

or store your file names in one list and your data in another and loop over both:

kpi_fns <- list("filename_1", "filename_2")
kpi_data <- list(kpi_1_data, kpi_2_data)
for (i in seq_along(kpi_fns)) {
    save_KPI_plot(kpi_fns[i], kpi_data[i])
}

If you prefer to have a different image format you can change png() to bmp() or jpeg().


Weights and Bias set to NaN I am performing linear regression on one of UCI Machine learning repository data set. Below is the code :-

import tensorflow as tf
import matplotlib.pyplot as plt
import csv
import os
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

wine_data = pd.read_csv('data/winequality-red.csv', delimiter=';')
n_samples = wine_data.shape[0]

train_x = wine_data.iloc[:, :11].values
train_y = wine_data.iloc[:, 11].values

training_epochs = 1000
learning_rate = 0.03
n_input = 11
n_classes = 1
display_step = 10

X = tf.placeholder(shape=[None, n_input], dtype=tf.float32)
Y = tf.placeholder(shape=[None], dtype=tf.float32)

# Weights and Biases.
W = tf.Variable(tf.truncated_normal([n_input, n_classes]))
b = tf.Variable(np.random.randn())

# Construct Linear Model.
prediction = tf.matmul(X, W) + b
loss = tf.reduce_sum(tf.pow(prediction - Y, 2))/(2 * n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    for step in range(1, training_epochs + 1):
        sess.run(optimizer, feed_dict = {X: train_x,
                                         Y: train_y})
        if step % display_step == 0 or step == 1:
            cost, x, y = sess.run([loss, W, b], feed_dict = {X: train_x,
                                                             Y: train_y})
            print (x)
            print (y)
            print ('Step ' + str(step) + ', Minibatch Loss = ' + \
                   '{:.4f}'.format(cost))
    print ('Training Done!!!')

Now, weights and biases get set to Nan only after single step.

[[14430.979  ]
 [  918.5443 ]
 [  484.25635]
 [ 4654.437  ]
 [  151.41441]
 [31196.336  ]
 [97207.76   ]
 [ 1732.7361 ]
 [ 5743.218  ]
 [ 1149.8185 ]
 [18074.775  ]]
1735.9697
Step 1, Minibatch Loss = 32277641302114304.0000
[[nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]
 [nan]]
nan
Step 10, Minibatch Loss = nan

What all reasons could result weight and biases carry Nan?

I think there is some NaN values in your data set.

First time it is printing the random values which are generated for the weights.  But after first step, that means, after the first update of weights, they are becoming NaN. That means there is some NaN values or some missing data in the training data.

Check your training data. I am not clear how tensorflow handles the NaN or missing data.


How do CNNs use a model and find the object(s) desired? Background: I'm studying CNN's outside of my undergraduate CS course on ML. I have a few questions related to CNNs.

1) When training a CNN, we desire tightly bounded/cropped images of the desired classes, correct? I.e. if we were trying to recognize dogs, we would use thousands of images of tightly cropped dogs. We would also feed images of non-dogs, correct? These images are scaled to a specific size, i.e. 255x255.

2) Let's say training is complete. Our model's accuracy seems sufficient, with no problems. From here, let's have a large, HD image of a non-occluded dog running through a field with various obstacles. With a typical NN and some data, we just take the model, cross it with some input, and bam it's going to output some class. How will the CNN view this large image, and then 'find' the dog? Do we run some type of preprocessing on the image to partition it, and feed the partitions?

Though there can be a very detailed explanation for this question but I will try to make you understand much minimal words.

1) Cropping the images to a particular size isn't a necessary condition and neither is scaling. But put this way, it doesn't matter whether a dog is represented in a B&W image or  RGB image because a convolution network learns features in the images which are independent of colors. Scaling and resizing help to limit the value of pixels between 0 and 1.

2) Once you have trained your CNN model, it has learned all the features like edges,etc. to recognize a dog in the image. Because the model has learned the features, it acquires certain properties like translation invariance which means that no matter where you position a dog in the image, it's still a dog and have the same features. How the model recognize it? It checks for the features of a dog, learned during training, no matter what the size of the new image is or where the dog is in the image or what the dog is doing.

For getting a in-depth understanding you can refer to the following resources:

http://neuralnetworksanddeeplearning.com/chap6.html

http://cs231n.github.io/convolutional-networks/


How do I figure out if subsets can be plotted in a normal Euler diagram? In some cases, it may be impossible to draw Euler diagrams with overlapping circles to represent all the overlapping subsets in the correct proportions. This type of data then requires using polygons or other figures to represent each set. When dealing with data that describes overlapping subsets, how can I figure out whether a simple Euler diagram is possible?

If you are interested in an empirical solution, you might consider the Venneuler package in R. It provides residuals (percentage difference between input intersection area and fitted intersection area).

https://www.rforge.net/doc/packages/venneuler/venneuler.html


LSTM with target outside the timeseries Say I want to predict the final size of a flower depending on the raining and temperature during 200 days. The flowers are in differents towns, so each flower has its own conditions of rain and temperature.

I have :

500 flowers

For each flower :
- 1 time serie with 200 points corresponding to raining volume each day
- 1 time seris with 200 points corresponding to temperature on each day

known target : final length for each flower (500 data)

what should be the input dimensions of my LTSM network ?

What you're looking for is covered in the keras LSTM examples:

from keras.layers import LSTM, Dense
import numpy as np

data_dim = 16
timesteps = 8
num_classes = 10

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))

# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))

model.fit(x_train, y_train,
          batch_size=64, epochs=5,
          validation_data=(x_val, y_val))

Source: https://keras.io/getting-started/sequential-model-guide/#examples

In your case, timesteps = 200, data_dim = 2.
It sounds like you have a continuous outcome and don't have classes, so you probably want to switch out the last layer to a Dense(1) with a linear activation or something, and switch out your loss and metrics appropriately (probably mean_squared_error and mae or something).

Also please note the architecture is purely an example, you may not need stacked LSTMs (or even LSTMs) for your example, you may want dropouts, you may want more dense layers, etc. This is just an example to get you started,


How is the output of a maxpool layer window size=1x2 and stride=2 calculated? I'm looking at the architecture proposed in the following paper: Baoguang Shi et al, An End-to-End Trainable Neural Network for Image-based Sequence
Recognition and Its Application to Scene Text Recognition.

In the proposed architecture of the model, a MaxPooling Window:1 × 2, s:2 layer is mentioned. I'm not sure what the size of the output of this layer would be.

If i have an input of size (32 x 8), then the output would be:

(32-1)/2 + 1 = 16.5, <- this part doesn't make sense to me

(8-2)/2 + 1 = 4

*ignoring depth and batch size here

According to the paper, maybe "s" represents stride in row, while the stride in column equals 1.


How do I get Kera's to use more than 1 core? I've read that keras supports multiple cores automatically with 2.2.4+ but my job only runs as a single thread.  I'm running inside a VM else I'd try to use the GPU I have which means the solution I'm working with is CPU based.
Here's my snippet of code
import numpy as np
import tensorflow as tf
from tensorflow import keras
epochs_ = 1000
batch_size_ = 150

np.random.seed(42)
tf.random.set_seed(42)

from keras.layers import Dense, SimpleRNN, GRU, LSTM
from keras.optimizers import SGD

#simple RNN
data_ = Lagged_Set

model6 = keras.models.Sequential([
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True, input_shape=[None, len(data_.columns)]),
    keras.layers.SimpleRNN(32, return_sequences=True),
    keras.layers.TimeDistributed(keras.layers.Dense(n_ahead))
])

model6.compile(loss="MAPE", optimizer="rmsprop",metrics=['MAPE'])
history = model6.fit(X_train, Y_train, epochs=epochs_,batch_size=batch_size_,validation_data=(X_valid, Y_valid))

I tried this which did nothing
session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=8, inter_op_parallelism_threads=8)

#tf.compat.v1.ConfigProto.set_random_seed(1)
sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)
tf.compat.v1.keras.backend.set_session(sess)

If you are using tensorflow >= 2.0.0, you can try set the number of threads
tf.config.threading.set_intra_op_parallelism_threads(1)
tf.config.threading.set_inter_op_parallelism_threads(112)

or you can look thread limit number of cores


What is the differences in the Gini Index, Chi-Square, and Information Gain splitting methods? I am looking through decision trees, and I do not understand what makes each of these methods different. Could someone explain clearly what the difference between these is? Thank you.

As I understand it, all three want to minimize the false classified data points in your data set. (Logically, if you look for what decision trees are used)

But each of them comes from another side to this problem.

gini impurity wants "better as random"

It compares the "I label random data with random labels" against the labeling after possible split by decision tree (Wish is, that you can split the tree with better outcome than "random random random")

information gain wants small trees

It uses knowledge from information theory. It models the difference between "good" and "bad" split with criteria "simple/small trees preferred". As a result of this, it want to split the data in a way, that the daughters are "pure as possible".

For the chi-square ... I have found two things: CHAID, a (seemingly complex) decision tree technique and the chi square to prune decision trees after their building.

The chi square in general has its roots in biological statistics. It gives a characteristic number how the observed distribution conform with the null hypothesis one have about this distribution. (Biology have to act like this a lot. "I observe something, I search for an explanation, I form a hypothesis, I probe if this is statistical confirmable")

For formulas please look in Wikipedia and other sources.


Why an eigenvector might be reasonable notion of centrality Using adjacency matrix to represent a connection between node i and node j, 1 means connected and 0 means not connected.

Using eigenvector to represent the centrality means the more nodes with high centrality values a node is connected to, the higher that node's centrality value is.

Eigenvector itself is just a vector after applying a transformation, the result vector has the same direction or totally opposite direction as the original one.

I really can't see the relationship between those two properties.

Let the adjacency matrix of our network be $A∈\{0,1\}^{n×n}$ with an empty diagonal ($A_{ii} = 0 ∀i$).

Direct approach

Let’s start with the approach that a node’s centrality ($C_i$) shall be proportional to the sum of the centralities of its neighbours with a proportionality constant $\frac{1}{λ}$ (chosen thusly with some foresight):

$$ C_i = \frac{1}{λ} \sum_{j=1}^{n} A_{ij} C_j.$$

This is nothing but a line-wise formulation of the matrix–vector multiplication:

$$ λ \vec{C} = A·\vec{C},$$

which is exactly the definition of an eigenvector. Now, to see why the largest eigenvector is chosen, we can turn to the Perron–Frobenius theorem, which tells us that for this eigenvalue (and in case of a connected network only for this eigenvalue), we can find an eigenvector all of whose components, i.e., the eigenvector centralities are positive.

Iterative approach

Alternatively, we can interpret the above ansatz iteratively:

Assign random positive values to the components of $\vec{C}$.
Update these values according to:

$$\vec{C} ← \frac{A·\vec{C}}{\left| A·\vec{C} \right|}.$$

This means that each component is updated according to:

$$C_i ← \frac{1}{\left| A·\vec{C} \right|} \sum_{j=1}^{n} A_{ij} C_j,$$

i.e., you say the new centrality of a node is the sum of the centralities of its neighbours – times some normalisation to avoid values getting very big.
Repeat Step 2 until the centralities converge. The idea is that if this converges to a unique result, this result does not only fulfil $C_i \propto \sum_{j=1}^{n} A_{ij} C_j$ but is also robust in that respect.

For almost all initial choices of $\vec{C}$, this will converge to the positive, length-1 eigenvector to the largest eigenvalue (which exists and is unique for a connected network, see above). The reason for this is that components along the eigenvector to the largest eigenvalue will be most enlarged by the multiplication and thus dominate the others over the iterations. (Having no components along the eigenvector to the largest eigenvalue does not happen in reality and is the reason why it’s almost all above.)

Note that such an iteration can also be used to numerically determine the largest eigenvalue.


Remove all characters following a certain character in a column of a dataset I have a data set like the following, and the first column contains the groupings. However, some are labelled slightly differently. I need to remove all characters following the punctuation used (bracket, semicolon, comma).

groups <- c("Group1", "Group1", "Group1;Group1", "Group1(subset)", "Group1,ex" )

I would like this to present all of these just as Group1 (so they would all appear the same as the first two) - so to remove all characters in the string following the punctuation. I then need to repeat this for 1000s of groups, all in that same column of my dataset).

I know gsub is an option, but I'm not clear on how to use it to remove all of a string following a number of different characters, or on how to use it on just one column of a very large dataset.


Running H2O in databricks I'm trying to run H2O in databricks. However, when I do the following:

hc = pysparkling.H2OContext.getOrCreate(spark)

I get the following error:

java.lang.AbstractMethodError

Does anyone know what could be the problem? Thanks!

I'm not sure that it going to work.  Take a look at the links below.

https://spark.apache.org/docs/latest/ml-classification-regression.html
https://spark.apache.org/docs/latest/ml-clustering.html

Find an algo from something in there and run it.  It should work fine.


Python (Pytorch) loss function syntax I have seen many examples of this syntax that is being used for the loss function specifically:
loss = nn.BCEWithLogitsLoss()(pred, y)
Can anyone explain me what does the (pred, y) do exactly, that it directly computes the loss, instead of calling loss as a function of these 2 arguments again?

This is an example of Python's builtin __call__ method, as described here. In short: BCEWithLogitsLoss is a class. The first set of parentheses (empty, in your case) provides any needed arguments to the class initializer. Then the second set of parentheses are passed to the call method. So, this is convenient syntax that allows you to instantiate the class and evaluate one of its methods in one line.
Confirming this in the source code is actually a bit difficult. You can see the source for BCEWithLogitsLoss here, which confirms it is a class. But its only method is forward; where is __call__? For that, we notice that the BCEWithLogitsLoss class inherits from the _WeightedLoss class, which inherits from Loss, which inherits from Module. We can then see that this base class implements the mapping between __call__ and forward.


Text classification based on n-grams and similarity I have tried to cluster hundred texts using k-means clustering. I would like to consider other algorithms to group text based on their content and try to spot news not related to other news (topic different).
I would like to know if there is some other method/algorithm that could be used to classify text into two or more groups based on their level of similarity.
Let’s say I have thousands of text that use the same keywords but in different context.
Some text may have the same sentence structure, for example:

The pen is on the table
The black pen is on the table.
Where is my pen? Your pen is on the table.
I cannot find my pen. I was using it a few minutes ago.

I would like to be able to cluster the sentences above based on their structure, i.e. using n-grams with n manually set.

In this way I should be able to include the first three sentences in the same cluster, as their level of similarity in using same/similar words is higher compared to the last sentence.
It is about compute for each sentence the similarity with others.

Do you have any idea on how I could do that?

First of all, both in your question and your tags, you used Clustering and Classification interchangeably. Be careful as they are totally different problems. I give you a comprehensive tour on how to encode your texts and find similarities so you will be fine doing any of them.

Classic Solution

As you mentioned, n-gram modeling and using Bag of Words variants (for example Tf-Idf) is the classic solution. There are points about this approach:

It does not model the order of words in sentences. So each document/sentence is modeled as a set of vocabularies regardless of the sequence of their positions. With n-gram model where n is larger than 1, you capture some concurrences of the words but still far from understanding the text.
TF-IDF is sparse by nature (there are huge amount of dimensions and a huge fraction of zero values in the matrix). For short documents (like your example sentences), it will be end up in vectors with a few values and a huge amount of zeros in other dimensions. You will need to reduce that dimensionality as I will explain.
It can not handle out of vocabulary words in case you will use it in a classification or text search problem.
The good thing about it is that it is pretty simple and works as a basic algorithm for such a task. The information theoretic concepts behind TF-IDF, beautifully weights words in the document e.g. stop-words will be automatically ignored.

How To

Generate your TF-IDF matrix
Apply Singular Value Decomposition to your matrix to reduce the dimensionality to k dimensions where k is the number of different topics in your corpus. You get back a $m\times k$ matrix back which is the reduced version of your TF-IDF. $n$ is the number of your documents and $k$ the number of different topics in it
You need to simply run your clustering algorithm on this matrix and get your results.

What you did above is called Latent Semantic Analysis. The best topic modeling out of BoW model.

How To Choose $k$
Just visualize! Use different values for $k$ and using t-SNE or UMAP (actually use UMAP! I just mentioned t-SNE for sake of completeness of my answer) for visualizing your data in 2 dimensions and try to guess what is the best $k$. Please note that Clustering problem has no evaluation. At the end of the day it is a smart search.

Classic Solution with Modern Tools

Increase the usage of UMAP in the above pipeline. It is a pretty strong nonlinear dimensionality reduction and visualization algorithm. You can drectly feed your TF-IDF to UMAP and see the results or after applying SVD, try to use UMAP for visualization and dimensinality reduction.

Modern Approaches to NLP/NLU

Neural-based language modeling has a long history which is out of scope here, but in 2018, Google released an algorithm which was a breakthrough in NLP and is widely used nowadays. It is called BERT. A startup called HuggingFace implemented an easy-to-use software package for BERT in Pytorch and Tensorflow called Transformers. For your use-case, I recommend another package from Technical University of Darmstadt in Germany called sentence-transformers which is based on SBERT (Sentence-BERT). You can easily find sample codes for document similarity search in their repo, for example this is an exact solution to your problem.

In case of further questions please just drop a comment.

Good Luck!


mysql could not recieve table from Citespace I created was database through commands (see lower)

Started Citespace, clicked DATA - IMPORT/EXPORT
Citespace-Built-in-database
Database-Connect to mySQL
Input directory - choosed directory with WOS files
Project - import data (WOS)

Now check:

click Count records
Recieved:
projects  0
articles    0
authors  18020
refs   144499
keywords  64260
phrases    0
institutions  11695
verbs

so tables

- articles
- bursts
- clusters
- phrases
- verbs

are empty
and system see no projects

CAN YOU help me to understand what i do wrong?

=== database schema

CREATE DATABASE wos;
USE wos;
CREATE TABLE articles
(
    id int,
    uid VARCHAR(100),
    project VARCHAR(100),
    author VARCHAR(100),
    title VARCHAR(100),
    abstract VARCHAR(1500),
    source VARCHAR(200),
    j9 VARCHAR(100),
    volume VARCHAR(100),
    issue VARCHAR(100),
    bp VARCHAR(100),
    ep VARCHAR(100),
    page VARCHAR(100),
    dt VARCHAR(100),
    doi VARCHAR(100),
    year int,
    month int,
    date int,
    citations int,
    editor VARCHAR(100),
    tagged boolean
);
/*  TABLE articles
id(int), uid, project, author, title, abstract, source, j9, volume, issue, bp, ep, page, dt, doi, year(int),
month(int), date(int), citations(int), editor, tagged(boolean)
*/
CREATE TABLE authors
(
id int,
lastname VARCHAR(100),
firstname VARCHAR(100),
initials VARCHAR(100),
project VARCHAR(100),
uid VARCHAR(100),
pos VARCHAR(100)
);
/*  TABLE authors
id(int), lastname, firstname, initials, project, uid, pos */
CREATE TABLE refs
(
id int,
bibcode VARCHAR(100),
ref VARCHAR(100),
doi VARCHAR(100),
author VARCHAR(100),
year VARCHAR(100),
source VARCHAR(100),
volume VARCHAR(100),
page VARCHAR(100),
citer_uid VARCHAR(100),
project VARCHAR(100)
);
/*   TABLE refs
id(int), bibcode, ref, doi, author, year, source, volume, page, citer_uid, project */
CREATE TABLE keywords
(
id int,
keyword VARCHAR(100),
uid VARCHAR(100),
year VARCHAR(100),
project VARCHAR(100),
type VARCHAR(100)
);
/* TABLE keywords
id(int), keyword, uid, year, project, type */
CREATE TABLE phrases
(
id int,
phrase VARCHAR(100),
isTitlePhrase boolean,
project VARCHAR(100),
uid VARCHAR(100),
year int,
month int,
date int,
freq int
);
/*  TABLE phrases
id(int), phrase, isTitlePhrase(booelan), project, uid, year(int), month(int), date(int), freq(int)  */
CREATE TABLE verbs
(
id int,
project VARCHAR(100),
uid VARCHAR(100),
verb VARCHAR(100),
freq VARCHAR(100)
);
/*  TABLE verbs
id(int), project, uid, verb, freq  */
CREATE TABLE bursts
(
id int,
project VARCHAR(100),
term VARCHAR(100),
weight  double,
start int,
end int
);
/* TABLE bursts
id(int), project, term, weight(double), start(int), end(int) */
CREATE TABLE institutions
(
id int,
name VARCHAR(100),
country VARCHAR(100),
uid VARCHAR(100),
year int,
project VARCHAR(100)
);
/* TABLE institutions
id(int), name, country, uid, year(int), project */

The problem may be is that you created data schema manually. You can erase wos database and just use CiteSpace. It creates dataschema automatically. Hope it will be OK.


remove special character in a List or String Input_String is Text_Corpus of Jane Austen Book

output Should be :
['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

But getting this Output :
['to', 'be,', 'or', 'not', 'to', 'be:', 'that', 'is', 'the', 'question!']

Regular expressions can be used to create a simple tokenizer and normalizer:
from __future__ import annotations
import re

def tokens(text: str) -> list(str):
    "List all the word tokens in a text."
    return re.findall('[\w]+', text.lower())

assert tokens("To be, or not to be, that is the question:") == ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

Otherwise, use an established library like spaCy to generate a list of tokens.


Converting VOC2012 annotation .xml to segmentation .png Does anyone know a good way to convert VOC2012 annotations files in xml to SegmentationClass (class annotations) files in png?

This is Deeplab's bash script for preprocessing the dataset which may help - https://github.com/tensorflow/models/blob/master/research/deeplab/datasets/download_and_convert_voc2012.sh

Some people reported running into some problems reading the annotations, which is covered here https://stackoverflow.com/questions/51676447/python-use-pil-to-load-png-file-gives-strange-results?noredirect=1#comment90316222_51676447


xgboost GridSearchCV take too long or does not goes to the next step just strange

%%time
xgb = xgb.XGBRegressor(n_estimators=500, learning_rate=0.07, gamma=0, subsample=0.75, colsample_bytree=1,
                            max_depth=7, tree_method='gpu_exact')

this code takes around Wall time: 866 ms.

but when I do the gridsearchCV it does not goes to the next step
even though I gave only one parameter

    %%time
    xgb = XGBClassifier(tree_method='gpu_exact',verbose_eval=True, silence=False)
    kfold = StratifiedKFold(n_splits=10, random_state=0)
    xgb_param_grid = {
                    'learning_rate': [0.08,0.09],
                    'random_state': [0],
                    'max_depth': [8,9],
                    'n_estimators': [400,500]
    }
    xgbGrid = gsRFC = GridSearchCV(xgb,param_grid = xgb_param_grid, cv=5, scoring="neg_mean_squared_error", n_jobs= 10, verbose = 1)
    xgbGrid.fit(X,y)
    xgb_best = xgbGrid.best_estimator_

for my understanding, this should not take that long.<br/>
it d

does not go to the next step I do not sure this even working or not
it stop with

  Fitting 5 folds for each of 8 candidates, totalling 40 fits
  [Parallel(n_jobs=10)]: Using backend LokyBackend with 10 concurrent
  workers.

data set size is  (15035, 22)
am I doing something wrong?

Based on the combinations of learning parameters, learning rate(2), max_depth(2) and n_estimators(2), it seems the algorithm is exactly doing what it's supposed to do. with cross validation set to 5 it's performing 40 fits (2*2*2*5).
Can you add some more details to clarify like what configuration you are using for doing this task?


Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them? I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 X=combine_6_non;
 y=target;
 c = cvpartition(y,'k',10);
 opts = statset('display','iter');
 [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)

The function fun is as follows:

function err = fun(XT,yT,Xt,yt)
  model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);
  err = sum(svmclassify(model, Xt) ~= yt);
end

Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.


Dueling DQN what does a' mean? what does $a'$ mean in the "combining" equation in Dueling DQN? (top of the page 5)

$$Q(s,a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \biggl( A(s, a; \theta, \alpha) - \frac{1}{N}\sum_{a'}^{N}A(s, a'; \theta, \alpha) \biggr)$$

Where there are $N$ actions to choose from;

$s$ is the incoming state (the input vector)
$a$ is the action taken? (the chosen action)
$a'$ I don't know what it represents in this context
$\theta$ represents the weights of the convolutional layers
$\alpha$ are the weights of the "Advantage stream" which outputs a vector
$\beta$ are the weights of the Value stream (which outputs a scalar)

Why not to simply use $a$ everywhere, why is $a'$ used in the average?

It is just a type of namespacing, because $a$ is already assigned the chosen action. There are two contexts of action being considered in the equation, so there needs to be a symbol for each context. Using $a'$ is an obvious choice as the letter $a$ is implicitly linked to representing an action already.

The sum over $a'$ is a sum over all possible actions in state $s$, irrespective of the chosen action $a$.

So both $a$ and $a'$ represent actions. $a$ is the current action, supplied on the LHS of the equation. $a'$ represents the iterator of a sum over all actions $[\forall a' \in \mathcal{A}(s)]$, only used in the calculation on the RHS. Sometimes you will see a completely different letter chosen, or some subscripting or other way to show these represent different actions.

It is also quite common to see $a$ representing current action, and $a'$ representing the next action (taken when in state $s'$). But that is not what is happening here.


How to build a classifier for determining if two pairs are a match? I'm trying to build a classifier that can determine if two addresses are a match or non-match.

Let's assume I have a data set of address pairs that have a match or non-match label.

I'm new to ML so something may be going over my head but is this how I'm thinking of approaching the task:

Approach for building classifier

Segment each address into it's subcomponents such as streetname, zipcode etc
Construct a comparison vector with one dimension for each field. The fields holds the value from a string similarity method such as jaro-winkler.
All my comparison vectors with an associated match label can now be fed into training a classifier

Predicting if a pair is a match

Build a list of candidate address pairs by using a method such as blocking
Construct a comparison vector the same way as the classifier was built above
The comparison vector for each candidate is now fed into the classifier and this will resolve to either a match or a non-match

My approach was inspired by research paper  Machine learning innovations in address matching: A practical comparison of word2vec and CRFs but a lot of the things are vaguely described for a beginner.

To sum up I'm looking for any input on whether this approach holds up or I'm misunderstanding something completely

Generally your approach looks good to me. Here are a few comments/suggestions:

You didn't mention how the set of labelled pairs is obtained. This part can be tricky when doing record linkage among a large set of N addresses, since it's very difficult to manually annotate all the N*N pairs. Bootstrapping is a common approach afaik.
The way the data is obtained may also have consequences about the proportion of positive/negative cases. In general the proportion positive cases is very low, and this can cause the usual problems of class imbalance. It's important not to train the model with a positive/negative ratio completely different from the one in the test set/in production.
It's often useful to diversify the similarity measure: instead of using only Jaro-Winkler, you can think of cosine-TFIDF, Levenshtein edit distance, and a lot of variants. In the same logic it's useful to vary the levels of comparison, e.g. characters, characters bigrams/trigrams, words.


Clustering 3D multivariate data I am looking for a clustering procedure that will group a number of 3D points on the basis of their spatial relations and multivariate dimensions. Dimensions are mostly represented with (interval) metric variables and few composed of categorical variables.
My questions:

Is there any clustering procedure that also factors in weighting
coefficients assigned (by user) to certain dimensions or spatial
relation (distance)?
Is there any clustering procedure that analyses both categorical and
metric dimensions?
Is there a clustering procedure thatr fullfill these requirements
and it is already implemented in any of the commonly-used
statistical software such as     R, MATLAB, Python?

My data looks like this:

  0.0303   0.0763   0.1363    0.1753    0.1916    0.2411    0.2661
  0.0000   0.0000   0.0000    0.0084    0.0176    0.0393    0.0482
  0.3287   0.3794   0.4887    0.7151    1.0220    4.8060    7.4140
  0.2310   0.2692   0.3563    0.4384    0.4836    0.6694    0.8040

Is there a method that considers both numerical and alphabetical values? For example if I add to this dataset a column with values such as A, B, C, AC, CF,....
Thanks

It would be nice show a few example data points to make sure that my answer is what you may be looking for, but anyways,

1) I think so, you may wanna look at feature weighting for k-means in this paper

3) For the libraries commonly used for this, you may wanna look at this answer or this code

2) I'm unclear what you mean here, maybe you could elaborate, or probably someone's going to pick up on it and answer you on this point.


Partial least squares (PLS) I am relatively new to Orange, trying to utilise it for linear regression, in particular partial least squares (PLS). My statistics knowledge is in the moment not good enough to know whether I could compose an equivalent by combinding PCA with ordinary linear regression, but I would anyhow have expected a dedicated PLS widget - or as an option in the Linear regression widget.

I even found a description of PLS functionality in Orange2 here, but this is not available (or moved to where I don't find it) in Orange3.

Well, I have a python/numpy PLS implementation that I could probably adapt into a python script block, but is this necessary? Please tell me that it's just a mouse-click/import away.

Well, I have a python/numpy PLS implementation that I could probably
  adapt into a python script block, but is this necessary? Please tell
  me that it's just a mouse-click/import away.

Welcome to the site! I wish I could tell you that what you want is a simple point-and-click but I don't think that it will be.

Generally speaking, most people use Orange as a tool for the visualization layer while you are looking for a tool for your modeling layer. Python has tons and tons of tools that you can use for modeling and should fit what you need. So, while that will be more steps for you, it will ultimately provide you more flexibility for other projects you may do in the future; just use Orange for the visualizations.


Time series data: How I measure influence of new product sales on existing product sales (statistically)? Here my goal is…

Find Product 5 (New Product) is really influencing other product sales (product 1 to 4) or not?
If it is influencing other product sales, how much?

New to R and tried several related posts but didn’t find exact answer to my question. I love R and learning every day something new which helping us in taking data driven decisions.

My sample Dataset is like below (Week and Product 1 to Product 5 Sales per each week)
Here my new product is Product 5 and launched on Week 5.

Week    Product-1   Product-2   Product-3   Product-4   Product-5
1   2   4   5   5
2   4   4   6   4
3   4   4   6   5
4   4   4   6   6   4
5   4   6   5   3   5
6   2   7   6   4   3
7   3   8   7   5   6
8   2   9   9   3   6

Here my questions are

What is the best process or model to show the influence of product 5 (statistically)?
Do I need to run co-integration tests before I run correlation?
Example some of these products are never be correlated with Product 5
(example: growth in cockle growth vs. growth in electricity demand)
How I know correlation vs. causation in this mix?
Since my new
product launched on week 5, where I can start my correlations? Is it
from week 5 or from earlier weeks?
Do I need to test for stationarity first? and bring the data to stationary?

You could build an ARIMAX model. This would permit to include autoregressive (AR) terms as well as well as the sales in product 5 as an Exogenous Input (X). This would give you a potential model where the sales for a product $i$ at time $t$ is given by $s^i_t$ then,

$s^1_t=\alpha_1 s^1_{t-1} + \alpha_2 s^1_{t-2} + \ldots + \beta_0 s^5_t + \beta_1 s^5_{t-1} + \ldots $

Note that you may need to make the series stationary first, but see more on that below. You could estimate this model with the seasonal R package that relies on the X-13ARIMA-SEATS software developed by the US Census Bureau.

I would recommend to ensure that your time series are all stationary, see for example this post before you use X13. I would also run cointegration tests. See for more explanation this excellent post.

Since you only have data on week 5 I would start modeling in week 5 but you could include autoregressive (AR) terms related to the sales of product 1 prior to week 5.


Training Doc2Vec and Word2Vec at the same time As far as I can tell the typical Doc2Vec implementation (e.g. Gensim) first trains the word vectors and afterwards the document vectors were the word vectors are fixed.

If my goal is that conceptually similar vectors (regardless of whether they represent documents or words) are close to each other, then would it be better to train them simultaneously?

For instance, a document about Godzilla should be similar to the word Godzilla (as well as to other documents about Godzilla) as well as to the word Japan.

Furthermore, could such a result being achieved with a typical implementation, when including single words as documents into the training data?

You need to be careful with the assumptions you make about the doc2vec implementation. Here are some useful concepts:

Word2vec has two different model implementations (Skip-gram and Continuous-bag-of-words) and Doc2vec has analogous PV-DM and CBOW models:

Word2vec

Continuous-bag-of-words (CBOW)
Skip-gram

Doc2vec (Paragraph Vector)

Distributed Memory (PV-DM)
Distributed Bag-of-words (PV-DBOW)

In skip-gram the inputs are pairs of words generated by a moving window of size n trough the data where we use one word to predict the other, whereas in CBOW the task is predicting a center word from the surrounding words. In the second case the context word vectors are typically averaged.

The Doc2vec extension works analogously. The DBOW model is analog to skip-gram and what it does is calculating the probability of words in a paragraph given a randomly selected word from that paragraph. The DM however, tries to predict the central word based on a context of words and the context paragraph.

As you can see, not necessarily the doc2vec models need to calculate word embeddings first.  In fact, in the gensim doc2vec function there is an parameter for the dbow model that says:

  "dbow_words (int {1,0}) – If set to 1 trains word-vectors (in
  skip-gram fashion) simultaneous with DBOW doc-vector training; If 0,
  only trains doc-vectors (faster)."

Meaning that during the training phase not even the dbow requires producing the word vectors.

There is a great notebook from rare-technologies that gives a lot of insights about how these models are trained.

Regarding your specific questions, you can train a word2vec and doc2vec model and use the results to compare words and paragraphs (look at this paper). But take into account that doc2vec is not training words and then doing some fancy stuff with them. The training process involves calculating paragraphs directly.


Dataset with some mislabeled data (around 1%) I have a dataset with around 1% of mislabeled data, it is a multi label problem and i want to find a way to correct those incorrect labels.
Assuming that the amount of mislabeled data is low i divided the dataset in Train/Test and trained a classifier taking care that the classifier does not overfit.
After that i know that the accuracy on the Test set is as high as possible i evaluated the whole dataset using the classifier and the result is a new set of labels which i assume that are the corrected labels.
Is this the correct approach to solve a problem like this?

It think it's a reasonable approach, but currently it seems that you have no way to check whether the new labels are correct or not. I think you should at least check that the new labels don't introduce more errors than they solve.
Ideally you would re-annotate a random sample of instances, keeping both the old (possibly erroneous) labels and the new ones. Then you can use this sample as a test set and evaluate the two following points:

most/all instances for which the new label is the same as the old label should be predicted with this label (otherwise it means your method changes correct labels)
most instances for which the new label is different from the old label should be predicted with the new label (otherwise it means your method doesn't fix the wrong labels)

The problem with this approach is that you need to annotate a large sample, since you need a reasonable number of wrong labels which are only present in 1% of the data.
If re-annotating a large sample is not possible, you could try a kind of boostrapping approach: run your method, then take a sample of instances which are predicted as different from the old label. Among these labels changes, count how many are correct. This approach requires less manual annotation effort since you don't need a large random sample, however it would miss the cases of wrong label which is not changed by the classifier.


What is the meaning of: Non-trainable params: 454? Should this ideally be 0? If so, how can this be made 0.

I guess it is due to using batch normalisation. Try to ascertain by investigating the math of that. If you see the paper, you can easily find out that the bias terms and $\beta$, if I remember, both get added. Consequently, one can be ignored and won't be trained because it is not needed.

  Dense layer should be added at the end (in the part you mentioned in your comment) or can it be added in previous layers also?

Not really, dense layers should be employed after conv layers. What they do is classifying the extracted features obtained by conv layers. About conv layers, they are employed for reducing the number of parameters and finding local patterns.

There is no consensus on how to change the number of filters in convolutional layers, at least as far as I know. But there is a point here. In the following lines of your code, you've employed a kind of pooling layer just before dense layer. If the number of activations coming from conv layer is many, you can use it but consider that by doing so, you ignore important features. I suggest you not doing that, especially, for the last conv layer. Also, try to increase the number of neurons in a dense layer or add extra layers for better accuracy.

conv2d_4 (Conv2D)            (None, 16, 16, 64)        73792
_________________________________________________________________
global_average_pooling2d_1 ( (None, 64)                0
_________________________________________________________________
dense_1 (Dense)              (None, 13)                845
=================================================================

About using batch norm, it is used due to a kind of problem which is called Covariat Shift. It simply tries to keep the distribution of the outputs of different layers in order to facilitate the learning process.

Based on your questions, I highly recommend you watching professor Andrew Ng's course about ConvNets in Coursera.


Evaluation of recommendation systems I have developed a content-based recommendation system and it is working fine. The input is a set of documents={d1,d2,d3,...,dn} and the output will be Top N similar documents for a given document output={d10,d11,d1,d8,...}. I eyeballed the results and found it to be satisfactory, the question I have is how do I measure the performance, accuracy of the system.
I did some research and found that recall, precision, and F1-score are used to evaluating the recommendation systems that predict user ratings. For this, we should no the original ratings and then the system should predict the ratings later we can plot the confusion matrix and then compute the aforementioned metric. However, in my case, I don't predict anything instead I measure the cosine similarity score sort it in descending order and pick the top N.
In this use case, how do I evaluate the system?
Thanks

There's some confusion about different kinds of output and their corresponding evaluation:

One can consider the top N results as predicted positive, any result lower than N as predicted negative. In this option one can use binary classification evaluation measure: precision, recall, f1-score would be the standard measures in this case.
One can consider the ratings/scores assigned to the full set of results. In this case there are two options:

if the numerical results are comparable, e.g. same kind of rating, then standard regression evaluation measures can be used, for instance RMSE.
if not, then it's still possible to compare the order of the results. Spearman rank correlation is a common evaluation measure in this case.

It seems that in your case you could use either the classification or the ranking evaluation measures. Of course, any of these evaluation methods requires gold standard results in order to compare the predictions against them.


How do I find output size of a network? The input to a convolutional layer of a neural network is an image of size $128\times 128\times 3.$

$ 40$ convolutional filters of size $5\times 5$ are applied to it. Would you get an output? If no then why not? If yes, what will be the output size?

Yes, and it would be $124 \times 124 \times40$.

To calculate the output dimensions:

$(dim_x - filterdim_x + 1) \times (dim_y - filterdim_y + 1) \times filters$

in your case:

$(128 - 5 + 1) \times (128 - 5 + 1) \times 40 = 124 \times 124 \times40$


Best tool for analyzing inventory that has binary sales outcome of yes or no I am trying to determine a good tool that will help me generate a probability of a sale for a list of of 300,000 products. I have a table of historical sales data (with about 300,000 records) that contains around 10 continuous variables along with a dependent variable that has a yes/no (i.e., binary outcome) value indicating whether product in the list has had a sale in the past 12 months.

The historical data essentially looks like this.

Product1,2,3 etc
Variable 1
Variable 2
Variable 3
Variable 4
Variable 5
Variable 6
Variable 7
Variable 8
Variable 9
Variable 10
Sold in past 12 months (Yes or No)

The last variable in the list is of course the dependent variable.

All I want to do is to find a tool that is going to be the best or easiest to use, so that I can assign a probability to each product in the list, essentially giving me the chance to condense my list to the products that are the highest likelihood to generate a sale, so that I can list those products instead of the others that have lower probability of generating a sale.

Ideally, the tool could do a quick logistic regression, or some other probability calculation based on the available variables, and thereby give me a (RVU-like) number (perhaps a probability ranging from 0 to 1) for each product, allowing me to quickly select the top 50,000 products to list on a website, since they have the higher probability of generating a sale according to the available variables.

I am of course assuming that the variables are somehow correlated to the outcome, but perhaps the tool will help me determine that.

Does anyone have any suggestions of a good tool to accomplish this? I would presume that there is a simple way to set this up in Microsoft Excel, but if not, then a piece of software that does this would of course be great too.
I am also open to suggestions as to which type of regression analysis (or other analysis) is the best to accomplish this.

Thanks for any suggestions.

Logistic regression would be an ideal candidate for assessing the probability of a sale, but it would be wise to take Emre's comments on 'how many' into account.  In this case even a basic aggregation of your top 50000 with a 'yes' dependent variable would get you there-- but why stop there?

You may want to isolate whether or not seasonal (or other cyclical) factors are influencing purchasing behavior (e.g. if you were attempting to optimize inventory levels for the summer purchasing habits).  In this case, linear regression with dummy variables accounting for the season or month may be a good way to delve deeper (this touches on the 'classification' point made by Emre).  The more historical data you have here, the better.

Back to your point on 'fast'-- you mention Excel so I am assuming Python or R are being ruled out.

RapidMiner or Knime would fit your requirement for 'fast' in any of these modeling contexts.  Otherwise, if you're curious, scikit-learn (python) has about 3 dozen other general linear models freely available (R has a vast library as well).

Attached is a link on how to do this kind of modeling within RapidMiner--

Logistic Regression Example (RapidMiner)

https://www.youtube.com/watch?v=0npgGPK0zR8

Happy modeling!


Making an ensemble model for high F1 score I presently have 2 algorithms that have a numerical output. Using a threshold of 0.9, I get the classification output. Let's say they are:

P (high precision, low recall)
R (high recall, low precision)

Individually, they have poor F-1 scores. Is the naive way of creating a classifier C as:
C(*) = x.P(*) + (1-x).R(*)
And optimizing for x and threshold a good approach to improve the F-1 score?
Or is there some alternate approach I must try.
Note: I can't vary functions P() and R(). Their outputs are provided as a black-box function.

In general this would mean that P predicts only a small number of instances higher than 0.9 whereas R predicts most instances higher than 0.9. Therefore a weighted average of the two scores will fall somewhere in the middle, likely resulting in a moderate precision and moderate recall.
This can give significantly better results but only if the two classifiers are complementary, i.e. they predict instances in a way different enough from each other. Otherwise it's equivalent to tuning the threshold on a single classifier.


Cross-validation: K-fold vs Repeated random sub-sampling I wonder which type of model cross-validation to choose for classification problem: K-fold or random sub-sampling (bootstrap sampling)?

My best guess is to use 2/3 of the data set (which is ~1000 items) for training and 1/3 for validation.

In this case K-fold gives only three iterations(folds), which is not enough to see stable average error.

On the other hand I don't like random sub-sampling feature: that some items won't be ever selected for training/validation, and some will be used more than once.

Classification algorithms used: random forest & logistic regression.

If you have an adequate number of samples and want to use all the data, then k-fold cross-validation is the way to go. Having ~1,500 seems like a lot but whether it is adequate for k-fold cross-validation also depends on the dimensionality of the data (number of attributes and number of attribute values). For example, if each observation has 100 attributes, then 1,500 observations is low.

Another potential downside to k-fold cross-validation is the possibility of a single, extreme outlier skewing the results. For example, if you have one extreme outlier that can heavily bias your classifier, then in a 10-fold cross-validation, 9 of the 10 partitions will be affected (though for random forests, I don't think you would have that problem).

Random subsampling (e.g., bootstrap sampling) is preferable when you are either undersampled or when you have the situation above, where you don't want each observation to appear in k-1 folds.


Irregular Precision-Recall Curve I'd expect that for a precision-recall curve, precision decreases while recall increases monotonically. I have a plot that is not smooth and looks funny. I used scikit learn the values for plotting the curve. Is the curve below abnormal? If yes, why and how can I correct it considering scikit learn automatically sorts the true and predicted labels. If the plot is OK, how best do I explain this behaviour?

This is definitely possible. When you are reducing the threshold, you will never decrease the recall (you can only flag more of the positive examples as positive). Precision is looking at all the examples that you flag positively, and of those the fraction that are truly positive. This means when you are reducing the threshold, you might not add any true positives but only false positives, thereby lowering your precision. Let's take a look at a fabricated example, where P is positive and N is negative. The samples are ranked by score/probability. Everything before the threshold is flagged as positive:

PPNPNNPNNN

If we put the threshold between items 2 and 3, we get a precision of $1$ and a recall of $0.5$:

PP - NPNNPNNN

When we put it between items 3 and 4, we keep the same recall but our precision drops to $0.6667$:

PPN - PNNPNNN


what is a good performance measure for comparing different neural network architectures in unsupervised clustering task? What is a good measure to use when trying to decide between picking unsupervised clustering NN architectures?
There seems to some ideas here, but i am trying to find out feedback/suggestions from people who've used it before and seen the measures to be meaningful.

A solution is using some metrics like silhouette measure to measure the performance of ANN in clustering.


Counting indexes in pandas I feel like this is a rudimentary question but I'm very new to this and just haven't been able to crack it / find the answer.

Ultimately what I'm trying to do here is to count unique values on a certain column and then determine which of those unique values have more than one unique value in a matching column.

So for this data, what I am trying to determine is "who" has "more than one receipt" for all purchases, then determine the same information based on each product category.

My approach so far:

We have a dataset like this:

receipt,name,etc,category
1,george,xxx,fish
1,george,xxx,cat
2,george,xxx,fish
3,bill,xxx,fish
3,bill,xxx,dog
4,jill,xxx,cat
5,bill,xxx,cat
5,bill,xxx,cat
5,bill,xxx,dog
6,george,xxx,fish

So then I can do this:

df.set_index(['name','receipt'])

And get the more interesting

                etc category
name   receipt
george 1        xxx     fish
       1        xxx      cat
       2        xxx     fish
bill   3        xxx     fish
       3        xxx      dog
jill   4        xxx      cat
bill   5        xxx      cat
       5        xxx      cat
       5        xxx      dog
george 6        xxx     fish

At this point it feels to me like the data is easy to work with, but I haven't figured it out.

One thing that is interesting to me is that if I sort the data by name before indexing it, the data displays grouped by name. In both cases the index is the same, so I don't know how to play with the representation of the data after indexing.

It is easy to find the data by category using

>>> orders.loc[orders['category'] == 'fish']
                etc category
name   receipt
george 1        xxx     fish
       2        xxx     fish
bill   3        xxx     fish
george 6        xxx     fish

But what I can't figure out is how to tell pandas "Find me the list of names that have more than one receipt".

Smaller questions:

What is the "pandas way" to get the length of the names part of the index? I'm supposing I could just turn the name column into a set and get the length of that. But I'm curious about indexes.

Edit / Update

Thanks for those answers! Here is clarification on what I am looking for:

I'm trying to find "repeat customers": people with more than one receipt.

So my set of all customers would be:

names: ['george','bill','jill'], ratio: 1.0

My repeat customers:

names: ['george','bill'], ratio 0.66

All 'fish' customers:

names: ['george','bill'], ratio: 0.666

My repeat 'fish' customers:

names: ['george'], ratio: 0.333

I think the examples given look helpful, but feel free to add anything.

I think maybe you are looking for:

receipts_by_name_x_cat = df.groupby(['name','category']).count()

Or, if you just want the total across all categories:

receipts_by_name = df.groupby(['name']).count()

Then, you can search those who have more than one:

receipts_by_name[receipts_by_name['receipt']>1]

And, you can find the length of an index by typing:

len(df.index.get_level_values(0))

Assuming the name was the first index column (otherwise substitute 1, 2, etc.)


Do I need to include a squared and linear variable in a random forest to achieve a parabolic effect? I'm using a random forest model. One of my independent variables almost certainly has a parabolic effect on the dependent variable. In a linear regression, I would include the variable as both linear and squared in order to capture this effect. Should I do the same in a random forest?

By extension, same question for variables that have logistic effects (age, for example)?

Random Forest uses a ensemble of Decision Trees using CART algorithm.

Since CARTs (Classification And Regression Tree) are a non-parametric algorithm, they should be able to find interactions between variables and non-linear behaviors.

Nevertheless, building polynomials can help them have a better performance.


What is C in sklearn Logistic Regression? In sklearn.linear_model.LogisticRegression, there is a parameter C according to docs

Cfloat, default=1.0
Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.

I can not understand it? What does this mean? Is it λ we multiply when penalizing weights?

Exactly as it says in the documentation; it's $1/\lambda$, where $\lambda$ is the regularization strength.


Grouping high dimension Y-space to lower dimensions I have a ML problem with 300 variables to predict, Meaning I have a multi-label binary classification problem with a Y space of 300 and with about 2000 rows only.
Thus we assume that there are 2^300 permutations. However, it is not so.
Let me explain with an example.
X1,....X400 | Y1,Y2,Y3...........Y299,Y300
'healthy',..'sad'| 0,1..1,0
'unhealthy'...'sad'|1,0,....0
However when i aggregate across the rows,there are maybe about 100 combinations far less than 2^300 combinations such as this.
so Y1,Y2,Y100 => C1 OR Y33,Y44,Y291,Y299,Y300 =>C2 ... till C100
Thus, I can change it to a multi-class problem to predict these 100 classes instead.
However, there are a lot of classes with count of 1 or 2. More than 50%. Thus, this approach doesnt work so well.
Is there an algorithmic way to still do the multi-label but by grouping the variables in the most optimal way like from Y space to Z space with dimension 5 for example.
{Y1,Y10,Y222,Y232}=> Z1 can be grouped together
{Y2...} => Z2 can be grouped together
Thus,we can predict on the Z space instead. At this point, I am not sure whether the Y variables need to be exclusive to one Z group or not.
But I am open to all suggestions at this moment ..
Cheers

You may find this resource helpful : https://xang1234.github.io/multi-label/

One possibility is to cluster similar labels together so that they are
processed together by the multilabel classification algorithms.
Community detection methods such as the Louvain algorithm allow us to
cluster the label graph. This is implemented in the
NetworkXLabelGraphClusterer with the parameter methods = louvain.

A resource for splitting your data into (hopefully) balanced train,val,test sets (not so easy because you are in a multi label setting) : http://scikit.ml/stratification.html


Why do we want the variance of the layers to remain the same throughout a deep network? I've been reading the literature on vanishing/exploding gradients and specifically how they connect to weight initialization. An idea I've come across a few times, which seems very important in this area, is that we want the variance to remain the same throughout the layers of a neural network, that is, if $v_n$ is the variance of the $n$-th layer, we want all the $v_n$ to be about equal. For example in Kumar 2017:

In this paper, I revisit the oldest, and most widely used approach to the problem with the goal of resolving some of the unanswered theoretical questions which remain in the literature. The problem can be stated as follows: If the weights in a neural network are initialized using samples from a normal distribution, $N(0,v^2)$, how should $v^2$ be chosen to ensure that the variance of the outputs from the different layers are approximately the same?

The paper goes on to claim that "the first systematic analysis of this problem was conducted by Gloriot and Bengio", citing Gloriot & Bengio (2010). But that paper seems to just assume that the reader already accepts keeping the variance stable is a good idea, I can't find anything like an explanation in the paper. They just make this claim:

From a forward-propagation point of view, to keep information flowing we would like that
$$\forall(i, i′), Var[z^i] =Var[z^{i′}]$$
From a back-propagation point of view we would similarly like to have
$$\forall(i, i′), Var\left(\frac{∂Cost}{∂s^i}\right)=Var\left(\frac{∂Cost}{∂s^{i′}}\right)$$

What variance is actually being taken here? Variance with respect to what?
What exactly is helped by the variance being stable in this way?
Whatever the answer to (2) is, what is the proof or evidence that that's the case?

What variance is actually being taken here? Variance with respect to what?

$Var(W^i)\rightarrow$ The variance of the random initialization of the weights
$Var(x)\rightarrow$ The variance of each feature
$Var(z^i)\rightarrow$ This is the variance of the values of $z^i_k$ for each neuron $k$ of the layer $i$ which is the same for all the neurons in the layer $i$

What exactly is helped by the variance being stable in this way?

Following the notation of the article, let's compute the gradient of the cost function w.r.t. the parameters in two consecutive layers (that we are going to call: layer $i$ and layer $i+1$). In this layers, the quantity used to update their respective weight matrices, is given by:
$$\text{Layer 1}\rightarrow \frac{\partial Cost}{\partial W^i}= \frac{\partial Cost}{\partial s^i}\frac{\partial s^i}{\partial W^i}=\frac{\partial Cost}{\partial s^i} (z^{i-1})^T$$
$$\text{Layer 2}\rightarrow \frac{\partial Cost}{\partial W^{i+1}}= \frac{\partial Cost}{\partial s^{i+1}}\frac{\partial s^{i+1}}{\partial W^{i+1}}=\frac{\partial Cost}{\partial s^{i+1}} (z^{i})^T$$
There we can see that if we consider:
$$ Var\left(\frac{\partial Cost}{\partial s^i}\right) = Var\left(\frac{\partial Cost}{\partial s^{i+1}}\right) \,\,\,\,\,\leftrightarrow\,\,\,\,\, Var(z^i)=Var(z^{i-1})$$
Then we would have:
$$Var\left(\frac{\partial Cost}{\partial W^i} \right) = Var\left(\frac{\partial Cost}{\partial W^{i+1}} \right)$$
This is a good thing because having the same variance in the updates of both layers means that the updates are globally spread in the same way, so assuming that the mean value of $\partial Cost/\partial W$ in both layers is the same, then this would mean that globally these layers are learning at the same rythm.

Whatever the answer to (2) is, what is the proof or evidence that that's the case?

A good advantage of the previous reasoning is that if we could achieve this happening in the whole neural network, then all the layers in the NN would be learning at the same rythm! So problems like vanishing or exploding gradient would be avoided.


Got some troubles with using OneHotEncoder to multiple categories I'm trying to get the final pipeline on the titanic dataset(Example was taken from the 'Hands-on ML' book).
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelBinarizer

num_pipeline = Pipeline([
    ('selector', DataFrameSelector(list(df_num))),
    ('imputer',SimpleImputer(strategy='median', fill_value='num',missing_values=np.nan)),
    ('std_scaler',StandardScaler())
])
cat_pipeline = Pipeline([
    ('selector', DataFrameSelector(list(df_cat))),
    ('imputer',SimpleImputer(strategy='most_frequent', fill_value='categorical',missing_values=np.nan)),
    ('cat_encoder', OneHotEncoder(sparse=False)),
])

from sklearn.pipeline import FeatureUnion

full_pipeline = FeatureUnion(transformer_list=[
        ("num_pipeline", num_pipeline),
        ("cat_pipeline", cat_pipeline),
    ])

df_prepared = full_pipeline.fit_transform(df)
df_prepared.shape
df_total = pd.DataFrame(df_prepared, columns=df.columns)
df_total

Where
df_num = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']
df_cat = ['Sex', 'Embarked']

The problem is that I got the ValueError:
ValueError: Shape of passed values is (668, 10), indices imply (668, 7)

I tried dropping  'Embarked'  from the df_cat
and it created the DF somehow(but it was inaccurate). I assume it didn't work because of 'Embarked'
had 3 categories(10-3 = 7 - that's the number of columns I wanted.)
If I drop 'Sex'
category, I get ValueError: Shape of passed values is (668, 8), indices imply (668, 7) (10-2=8)
How to encode all two categories using Pipeline()
and which wrong steps I've made?

df.columns is incorrect because after One-hot-encoding you have more columns with different names. In your example instead of Sex you have two columns for the actual values and instead of Embarked you have three.
The whole point of the pipeline is to create those additional columns so why do you want to drop them? I do not know where you define df.columns but it seems to me the error is happening in this line:
df_total = pd.DataFrame(df_prepared, columns=df.columns)


Link prediction with input data I have a list of files, an I use the KNN algorithm to classify these files.

dataset = pd.read_csv(file)
training_samples = get_sample_number(dataset)
X_train = dataset.iloc[:training_samples, 5:9]
y_train = dataset.iloc[:training_samples, 9]
X_test = dataset.iloc[training_samples:, 5:9]

# Feature Scaling
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.fit_transform(X_test)

# Fitting classifier to the training set
classifier = KNeighborsClassifier(n_neighbors=5, metric='minkowski', p=2)
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

Now I have my categories in my y_pred array. But I want to save the result in the file where I read the dataset. How can I link a prediction to the right row in the file (or dataset)?

First as "timleathart" mentioned, you need to fix your code by changing this line :

X_test = sc.fit_transform(X_test)

to:

X_test = sc.transform(X_test)

For your question :

you have already the number of samples (training_samples) used for training. so all you need is to iterate over the y_pred and save the values in new column in the dataset starting from "training_samples" as row index.


How does Batch Normalization in Machine Learning address covariate shift and speed up training? In this video and this answer, it's mentioned that batch normalization doesn't allow the mean and variance of the parameters of any particular hidden layer to vary too much with change in previous layer's node values. Let's say we have a network like this (taken from the video):

As we change the mini-batch that the neural network is trained on, it's possible that the distribution of the input features  with respect to different mini-batches is different. As far as I understand, this is called covariate shift. An issue with covariate shift is that maybe the distribution of input features with respect to the validation set examples is different from the training set (ref. black cats / colored cats example in the linked video), and the distributions w.r.t. the different mini-batches are different.

Consequently, since the learned weights connecting the input layer to the next layer depend on the input features distribution, the learned weights will accordingly change with each mini-batch. Thus the activations in the next hidden layer will also vary in their distribution w.r.t. different mini-batches.

First question: Is my understanding so far correct?

Now I can understand that limiting the mean and variance of input features or that of the hidden layer activations will address the covariate shift problem only if the distributions across mini-batches (or between training and validation sets) are fundamentally the same but differ in parameters (not the neural network parameters - I mean mean and stddev). Crudely speaking, if the distribution of input features w.r.t different mini-batches is normal, but the parameters (mean and stddev) are different, then batch normalization seems fine.

Question 2: But what if the input features have normal distribution w.r.t. one mini-batch and a completely different distribution w.r.t. another? I don't see how limiting the mean and variance will help. Maybe I'm ignoring some key assumptions.

Question 3: Why does limiting the variation in the distributions of the hidden layers speed up the convergence?

So for the first question, you are almost correct. Your description of covariate shift is correct, but batch normalization isn't focused on fixing input layer covariate shift, but instead internal covariate shift. It's not a huge difference, since the internal covariate shift of a hidden layer acts as the input to another layer, but internal covariate shift can occur even if there isn't significant input layer covariate shift.

The internal shift happens when weight updates occur while learning, after a couple epochs the output distribution of a hidden layer will likely change. This slows learning, because now the next hidden layer will have to learn the new distribution. Batch normalization tries to minimize this change in distribution so that less of the learning is focused on trying to learn the changes in hidden layer distributions.


What is sum encoding? I've heard sum encoding mentioned as a method of encoding categorical variables, but I haven't been able to find a clear explanation of what it actually is.
I found the following explanation on Towards DataScience:

Just like OneHot except one value is held constant and encoded as -1
across all columns.

But this confused me as it doesn't involve any summing.
Is it really just?:
Value      Variable 1. Variable 2.
Red            1            0
Blue           0            1
Green         -1           -1

Sum encoding is similar to one-hot encoding but the difference is that in sum encoding we take one value as '-1' and it is not compared to other value. Whereas in one-hot encoding we create one column for each value to compare against all other values.
It is referred as sum encoding because the intercept represents the grand mean and the contrast estimate is the mean for level 1 minus the grand mean.
In one-hot encoding intercept represents the mean for the baseline condition and contrast estimate represents simple the difference between one particular condition and the baseline.


Dropping Missing Observations under MAR Assumption Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and non-missing data but there is no theoretical background supporting that there are unobserved factors that can determine if the data is missing or not.
I want to drop the missing observations instead of using an imputation method. However, I am not sure how to prevent bias. I can control for the factors that explain if the data is missing or not simply by adding them into regression but I am not sure if this would be enough.

One option is to run the regression analysis both ways (imputation vs dropping) and see which results are more useful. For your problem, find the best empirical estimate of bias and see if that estimate changes because on the feature engineering choice.


How to define a neural model as linear or non linear TL;DR: in what sense is the model of a neuron seen in the image above nonlinear?

In chapter 1, section 1.3 MODELS OF A NEURON of Simon Haykin's Neural Networks book, the standard model of a single neuron is described and visualised in the picture above.

Haykin states that this model, which consists of a set of inputs x1..xm, their corresponding weights w1..wm, a linear combiner that sums the weighted inputs and the bias (b) and an activation function that takes that sum and produces the output, is nonlinear. So, my question is, isn't the output linearly dependent on the input? For example, if the neuron only takes one input, x1, then the linear combiner takes the form v = x1 + b and the activation function is φ(v). So, the only way that I can see this model being nonlinear is if the activation function is nonlinear. But there are clearly cases where the activation function is linear (like the piecewise-linear function described in that same section of the book). So how can this model be inherently nonlinear?

I realise that this isn't a major concern, but I'd like to understand every part of the book before moving on, and this has been bugging me since I saw it.

Thanks to everyone in advance for your answers.

You are right, for the model to be non linear, the activation function must be non linear.

If the activation function is for example the identity function, your model won't be non linear, even if you stack multiple hidden layers. Indeed, the output of the neurons will only just be a linear composition of the inputs combined with weights.

I think the author says this neuron model is non-linear because in practice a linear activation function is almost never used. Some functions look like linear functions, but in fact these are not. Here is a demonstration for the ReLU activation function, which look like a linear function but is mathematically not : https://datascience.stackexchange.com/a/26481/73209

I guess it is the same for the Piecewise Linear Unit (PLU) activation function.


Can I use confident predictions to correct incorrect labels? From visual inspection of a sub-portion of my data, I estimate that around 5-6% of the labels are incorrect.

My classifier still performs well and when I take prediction probabilities that are above .95 for a given class that are in contrast to the actual label, I find that 92% of the time the classifiers prediction is correct. So, for example, a review text with the label positive that has an estimated probability of >= .95 of being negative, is in fact negative most of the time.

Can I therefore use the most confident predictions to correct some of the noisy labels? And not retrain on the corrected data, but use the probabilities to correct the validation and test sets to get more accurate final performance and also to calibrate the estimates (as noisy labels may be particularly harmful to calibration).

EDIT: Follow up to answer below.

When I train on the uncorrected labels and then predict the test set I get:

When I train on the corrected labels for the corrected test set I get:

Likewise, the model trained on the uncorrected data performs better on the original uncorrected test set than the model trained with the corrected data. Somehow removing only incorrect labels with high confidence from the train set appears to decrease performance on unseen data. I would probably have to remove all incorrect labels and not only those identified through high confidence.

EDIT EDIT:
After experimenting for a while I've reached the following conclusion:

It seems that predictions can be used for label corrections, but a different estimator should be used to try and determine which labels are incorrect. When I used the same estimator, despite most of the label corrections being valid (I determined 92-93% based on visual inspection of a 500 sample subset), it nevertheless caused the new prediction estimates to be biased. The new estimates were overly confident (drastically tended towards zero and one). This is either due to the correction, or it may be due to too little noise in the dataset (I considered the possibility that the noise is actually helping the estimator not to overfit. Neural networks have been found to be poorly calibrated, the author of this article suggests that overestimation may in fact be a form of overfitting).

At the moment, you don't really know the performance of your model because you have quite a few wrong labels in your test set.

You mentioned that you want to use the new probabilities to correct the validation and test sets. However, if you do that, you will of course get higher results because you are using the labels coming from your own model. However, if you verify by hand than it is ok.

Therefore, what I would suggest is:

Train the classifier with all your data. Don't do any splits.
After training, pass all your data through the model. Separate all the data that the model gets wrong.
Manually check all the mistakes. You will have a few scenarios: The model gets is wrong but the label was correct) in this case, leave the label as it was. The model gets it wrong but was wrong) this are the interesting cases for you, where the model is wrong but in fact it is correct since the original labels were wrong. Correct these.
Now perform train/validation/test splits.
Retrain your model with the train/validation splits.
Test your model with the test split.


How do I factor in features which are IDs? I am an absolute beginner in data science and I had this (possibly stupid) question on my mind, while reading a problem in Kaggle: Say I'm given IDs of some clients, IDs of products that they sell, and quantity of the product sold, and I'm asked to predict the quantity of a product, given the client ID, and the product ID.

Now, say the client IDs are in the range 10000 - 50000 and the product IDs are in the range 1-10.

Suppose, for a moment that the client IDs were random integers from 1-1000000 and the product IDs were random integers from 1000-2000. This isn't supposed to make the slightest change in the results, is it? After all, IDs are mere tags.

But thinking data-wise, I've just bloated up two entire columns in my data to a higher scale, and these are two features as well.

So how do I think about this? How do I factor in features which are IDs? How do I 'normalize' them?

I hope I'm not being vague here. I just don't know a better way of phrasing this question.

These IDs should not be represented as numerical values to your model. If you would, your model thinks 2 and 3 are closer together than 2 and 2000, while it's just an ID, the number is just a name.

Some models can deal with them but then they need to be factors or categories (like decision trees). However most models cannot deal with categories at all, there are however numerous ways to solve this problem. The most used one is probably one-hot encoding, which means for every category in your feature you add a column, and you put a 1 if it's that category and a 0 otherwise. Example:

ID | target
1  | 0
1  | 1
2  | 3
3  | 2

To:

ID_1 | ID_2 | ID_3 | target
1    | 0    | 0    | 0
1    | 0    | 0    | 1
0    | 1    | 0    | 3
0    | 0    | 1    | 2

This work very well if you have few categories, however in the case of thousands of IDs this will increase your dimensionality too much. What you can do is collect statistics about the target and other features per group and join these onto your set and then remove your categories. This is what is usually done with a high number of categories. You have to be careful not to leak any information about your target into your features though (problem called label leaking).


Model accuracy changes when target attribute change I have a dataset. For a target attribute for example:person, I get an accuracy which is different when I use for example:price as target attribute and so on.

Why this accuracy keep changing?

There is not reason to believe that the accuracy wouldn't change as you change the target. After all, the features that is able to explain a target distribution could be very different from the features that is able to explain another target distribution.

It is similar to how human learn. We can't expect a human who is just trained to speak English to suddenly speak fluently in another language.


Treating continuous data as a classification problem by predicting bins or quintiles I currently have a model that has several numeric Y or predicted variables
Sample Data:

Y1
Y2
...
YN

2710
0.32
...
31231

1710
0.52
...
51231

I am currently using regression (multioutput regression with gradient boosting) to predict the Y variables.
Currently I am not happy with the accuracy and was looking at ways to increase accuracy. I was recommended using discretization so I treat Y as bins or quintiles. i.e:

Y1
Y2
...
YN

2000 - 3000
0.3 - 0.4
...
30000 - 40000

1000 - 2000
0.5 - 0.6
...
51000 - 50000

How can I go about doing this? What algorithms or libraries perform this? I am finding online how to discretize features but not so much the data itself. Do I need to manually shape the trained data into bins and then train on it?
I guess what I'm asking is where to go from here coding wise.

One option is scikit-learn's KBinsDiscretizer which bins continuous data into intervals.