Why compressed image size is greater than original one in kmeans algorithm? I have a png image as shown below.

And I use kmeans algorithm to compress the image by color quantization. I compressed the image to use 64 colours. The code is:

ncolor = 64
rimage = image.reshape(image.shape[0]*image.shape[1],3)
kmeans = KMeans(n_clusters = ncolor, n_init=10, max_iter=200)
kmeans.fit(rimage)
centers = kmeans.cluster_centers_

labels = np.asarray(kmeans.labels_).reshape(rows, cols)
compressed_image = np.zeros((rows, cols,3),dtype=np.uint8 )
for i in range(rows):
    for j in range(cols):
            compressed_image[i,j,:] = centers[labels[i,j],:]

fig, ax = plt.subplots(1, 2, figsize=(16, 6),
                       subplot_kw=dict(xticks=[], yticks=[]))
ax[0].imshow(image)
ax[0].set_title('Original Image', size=16)

ax[1].imshow(compressed_image)
ax[1].set_title(f'{ncolor}-color Image', size=16);

io.imsave('compressed_tiger.png',labels);

The image is the origin image. And the resulte shown on jupyter notebook is:

since I compressed the image to 64 colours but the size of the saved file compressed_tiger.png (588KB) is larger than the original one (435KB). I don't understand why it becomes larger.

Your are saving the same color format... if they are 24bits RGB you are currently saving also in 24 bits RGB. You should convert to RGB-8 that uses 8 bits per pixel.

I have used pypng and PIL to get the following result

from PIL import Image
import png

    s = []
    for i in range(rows):
        s.append(tuple(labels[i].astype(int)))

    palette = []
    for i in range(ncolor):
        c = (centers[i][0].astype(int), centers[i][1].astype(int), centers[i][2].astype(int))
        palette.append(c)

    w = png.Writer(width=cols, height=rows, palette=palette, bitdepth=8)
    f = open('tiger8bits3.png', 'wb')
    w.write(f, s)

Look at PNG formats: http://www.patrickhansen.com/2011/02/04/png-8-24-32-what/

The result is 269KB:


scoring argument in scikit-learn LassoCV, LassoLarsCV, ElasticNetCV I was looking at the arguments in the linear regularization methods with cross validation within scikit-learn. RidgeCV has an argument scoring which is None by default but one can use a custom scorer (e.g. RMSE). I noticed that no such option is given for the rest of the classes (LassoCV, LassoLarsCV, ElasticNetCV). Why is this? Is MSE used by default in all of them?

RidgeCV implements CV by GridSearchCV, which supports custom scorer.

However, LassoCV and ElasticNetCV implement CV by LinearModelCV, with MSE as scoring hard-coded. LassoLarsCV impletments CV by LarsCV, with MSE as scoring hard-coded too.

If you want to use other scorers, maybe you can use GridSearchCV directly.

References

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/ridge.py#L1092

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/coordinate_descent.py#L1181

https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/least_angle.py#L1143


Need of maxpooling layer in CNN and confusion regarding output size & number of parameters In my CNN architecture for binary classification, I have 2 convolutional layers, 2 maxpooling layers, 2 batchnormalization operations, 1 RELu and 1 fullyconnected layer.

Case1: When the number of channels, $d=1$:

In the first layer an input of size $[28*28*d]$, $d=1$ channel is convolved with $M_1=20 $ number of filters applied over all the input channels of size {$f_h \times f_w \times d$} = $[3\times 3\times 1]$ having the step size (stride) as 1 that creates a feature map of size ${(h-f_h+1) \times (w - f_w +1)\times d \times M_1} = (28-3 +1)\times(28-3+1)\times 10 = [26\times 26\times 20]$.

The second convolutional layer contains twice the number of filters = 40 of same size $[3\times 3 \times 1]$. So, the number of parameters becomes $[23 * 23 * 1 * 40]$ as the output from the second convolutional layer. So total number of parameters =  $[26\times 26\times 20]+  [23 * 23 *  40]$

Case 2: When $d=2$ and all other sizes are same. The filter size become $[3 \times 3 \times 2]$. The output of the first convolutional layer will contain: $(28-3 +1)\times(28-3+1)\times 2 \times 20 = [25\times 25\times 40]$. For the second convolutional layer, the output will contain $[23 \times 23 \times 2 \times 40]$ parameters.

Question1) Is my calculation for each case above correct?

Question2) I have read that the purpose of maxpooling is to reduce the dimensionality of the feature map. In my case each maxpooling is of size 3 and stride 2. What does it mean by reducing the dimensionality and then what will be the size of the output from each layer upon considering that there is a maxpooling operation after each convolutional layer?

Question 1) Is my calculation for each case above correct?

No, it is not correct. The formula to calculate the spatial dimensions (height and width) of a (square shaped) convolutional layer is

$$O = \frac{I - K + 2P}S + 1$$ with $I$ being the spatial input size, $K$ being the kernel size, $P$ the padding and $S$ the stride. For your two cases that is (assuming padding of $1$ since otherwise numbers won't fit later with pooling):

Case 1:

Layer 1: The spatial dimensions are $O_1 = \frac{28 - 3 + 2\cdot1}1 + 1 = 28$. So the total layer has size $height \cdot width \cdot depth = 28 \cdot 28 \cdot 20$.
Layer 2: The second layer has $O_2 = \frac{28 - 3 + 2\cdot1}1 + 1 = 28$ with a total layer size of $height \cdot width \cdot depth = 28 \cdot 28 \cdot 40$.

With regards to parameters please note that does not refer to the size of a layer. Parameters are the variables which a neural net learns, i.e. weights and biases. There are $1 \cdot 20  \cdot 3  \cdot 3$ weights and $20$ biases for the first layer. And $20\cdot 40  \cdot 3  \cdot 3$ weights and $40$ biases for the second layer.

Case 2:

The sizes of the convolutional layer do not depend on the number of input channels. However, the number of parameters for the first layer increases to $2 \cdot 20  \cdot 3 \cdot 3$ while the number of biases remains unchanged. The second convolutional layer is not affected at all.

  Question2) I have read that the purpose of maxpooling is to reduce the dimensionality of the feature map. In my case each maxpooling is of size 3 and stride 2. What does it mean by reducing the dimensionality and then what will be the size of the output from each layer upon considering that there is a maxpooling operation after each convolutional layer?

Typically convolutional layers do not change the spatial dimensions of the input. Instead pooling layers are used for that. Almost always pooling layers use a stride of 2 and have size 2x2 (i.e. the pooling does not overlap). So your example is quite uncommon since you use size 3x3.

You can apply the same formula as above (assuming padding again - see footnote (1) for an explanation)

$O_{pool\text{ }1} = \frac{28 - 3 + 2\cdot0.5}2 + 1 = 14$

Since now the input to the second conv. layer has different spatial dimensions you would get:

$O_{CNN\text{ }2} = \frac{14 - 3 + 2\cdot1}1 + 1 = 14$

And finally:

$O_{pool\text{ }2} = \frac{14 - 3 + 2\cdot0.5}2 + 1 = 7$

Footnotes:

(1) The padding of $0.5$ is the average padding required in each spatial dimension. As an example, have a look at how the second maxpooling kernel scans an input (I have chosen the second one since the input is smaller and, therefore, easier to visualize). The input has width and height of $14$:

As you can see from the drawing the actual padding is only required on one side for height and width respectively. Therefore, the actually performed padding is
$$padding_{width} = \frac{padding_{left} + padding_{right}}{2}=\frac{0 + 1}{2}=0.5$$

And the same numbers apply to the height. Accordingly, the actual padding is $(0.5, 0.5)$.

Moreover, the drawing shows 7x7 dark blue squares (kernel centers). Which visualizes the calculations leading to an output of $pool_2$ with spatial dimensions of 7x7 ($O_{pool\text{ }2}=7$).


Beautifulsoup find_all returns TypeError: list indices must be integers or slices, not str This should have an easy solution but I can't understand how to avoid it.
    content1 = soup.find('div', class_='fusion-text fusion-text-6')
    content2 = soup.find('div', class_='fusion-text fusion-text-7')

    for para in content1:
        comp_name = para.find_all('a')['href']
        print(comp_name)

The error comes because I have a list at ['href'] together with find_all.
        comp_name = para.find('a')['href']

doesn't return an error and gives the right output (an URL), but just the first one. Since I want to scrape all of them inside my content1 I wanted to use find_all but inevitably I get the error.
How can I avoid this?
Thank you.

Use list comprehensions:
comp_name = [link['href'] for link in para.find_all('a')]


How to describe the process of using existing weights for training a deep learning model? I have weights which I use for initializing the training of a convolution neural net model as follows (in Keras):

model = load_model("my_weights.hdf5")

Now I want to rerun the entire model with further data augmentation and training using these loaded weights. I would like to know if this is a feasible approach to apply. And also, would this be considered as fine-tuning? or transfer learning?

Yes it is definitely doable, and further more, its is recommended in many scenarios. For instance, if you wanted to train some generic image classifier it would be naive idea to start from scratch and, for instance, retrain ImageNets' dataset for 2-3 weeks simply trying to retrieve optimal weights that have already been previously computed.

if you simply take pre-trained weights and use them on a different domain, then that would be referred to as transfer learning. However, since you want to retrain the existing weights with additional training data then, yes, you would be "fine-tuning" the model. When you do this however, there are considerations that need to be taken. Do you want to fine tune every layer? Or only deeper layers that detect very specific features of the dataset? Considerations like these should be taken into account.

Also check out http://cs231n.github.io/transfer-learning/


Does save_best_only in Keras prevents overfitting? I'm training a CNN and using:

model_checkpoint = ModelCheckpoint(os.path.join(output_artifacts,'weights.h5'), monitor='val_acc', save_best_only=True)

I trained the network for 70 epochs, but the validation accuracy flattened on good value (90%) after 20 epochs remaining pretty constant.

I would say that after epoch 20 (more or less 20:00 in the graph) the network is overfitting. My question is, does save_best_only takes this into account or just save the weights at epochs 1:00 am which is the overall best (but overfitted)?
Should I use earlystopping to stop the training or is not necessary? I don't want to stop the training too early..

It really only tracks the value of the metric you selected, there is no tolerance option. In the relevant documentation, the definition is given:

  save_best_only: if save_best_only=True, the latest best model
  according to the quantity monitored will not be overwritten.

In your case, combing the callback with the EarlyStopping callback would be the best option. This callback does offer some kind of tolerance, i.e. it requires the improvement of the metric to be at least as good as min_delta to be considered an improvement.

You could combine both callbacks with something like this:

from keras.callbacks.callbacks import EarlyStopping

early_stop = EarlyStopping(monitor='val_loss', min_delta=0.05, patience=3, verbose=1, mode='auto')
checkpoints = ModelCheckpoint('model_weights.h5', monitor='val_acc', save_best_only=True)

my_callbacks = [early_stop, checkpoints]
...
model.fit(..., callbacks=my_callbacks)


sklearn serialize label encoder for multiple categorical columns I have a model with several categorical features that need to be converted to numeric format. I am using a combination of LabelEncoder and OneHotEncoder to achieve this.
Once in production, I need to apply the same encoding to new incoming data before the model can be used. I've saved on disk the model and the encoders using pickle. The problem here is that the LabelEncoder keeps only the last set of classes (for the last feature it has encoded), thus it can't be used to encode all the categorical features for the new data. To face this issue I am saving on disk a different LabelEncoder for each one of the categorical features, but this does not seem to scale very well to me, especially when you have a large number of categorical features.

What is the common practice for this situation? Is it possible to serialize and save just one encoder for all the categorical features to be used in production?

LabelEncoder is meant for the labels (target, dependent variable), not for the features.  OrdinalEncoder can be used for features, and so can take a 2d array rather than the 1d array LabelEncoder requires, and so you can use a single transformer for all your categorical columns.  (You can use a ColumnTransformer to select those categorical columns, if you have continuous ones too.)

But, you should not use OrdinalEncoder before OneHotEncoder anymore:  OneHotEncoder has for some time now applied directly to string categorical columns.


Reading a CSV using R I have a CSV file that, by all appearances, is totally normal except that each line ends with ,^M.  Not sure if that has anything to do with my issue or not, but I try reading in the file in R with the usual command

df <- read.csv('file.csv')

and then when I try to inspect it by typing df it prints out this result that I don't know how to interpret:

function (x, df1, df2, ncp, log = FALSE)
{
    if (missing(ncp))
        .External(C_df, x, df1, df2, log)
    else .External(C_dnf, x, df1, df2, ncp, log)
}
<bytecode: 0x000000000b083720>
<environment: namespace:stats>

Anybody know what's going on here?  I've read in CSV files before and had no problem.

The ^M looks like the file came from a computer with a different OS from yours - maybe Unix and you're on Windows? Look on Google for a Unix2DOS utility, or maybe you can open it in Excel and then re-export to fix it.


How to determine if a neural-network has a static computation graph? I'm looking at implementing some neural networks from research papers. However, I'm concerned about which framework I should use, because I'm uncertain if the networks form static computation graphs (making them suitable for TensorFlow/Keras) or if they are dynamic (making them more suitable for PyTorch).

What are some rules of thumb to determine if a network requires a dynamic computation graph?

I believe current implementations of the best known libraries will allow you to implement any network you come across. So it boils down to the aspects of taste and practicality.

As far as deciding on a framework based on the network you want to create; if the network has branches or separate control flows (data transfer or logical) that are used conditionally during training, PyTorch is likely more useful. Otherwise, the best starting point is likely to use Keras - a high-level API over Tensorflow.

The (very) general rule of thumb is:

  Use Pytorch for experimentation, and Tensorflow for production

Now the caveats to the above sweeping statement:

Both libraries are making massive efforts to cover "the other" library's strengths:

The latest release of Pytorch (v1.0) offers several methods of making production ready code - so ways of compiling it to improve performance as well as the use of 16-bit floating point operations (as opposed to the standard 32-bit). They have also started rolling out a C++ API. These are all steps to make full-scale products for deployment. Tensorflow has introduced its eager API, for eager execution, meaning you can essentially build dynamic graphs, just as in PyTorch. The enables faster and more interactive prototyping.

The experience you have while developing might be more important:

If you are a pure Python developer, PyTorch will feel more natural. If you have been a general software engineer in other statically types languages (such as C++ and Java), Tensorflow may feel more logical.

Summary

Overall, Tensorflow along with (the now built-in) Keras wrapper, is a more mature library with a larger community and longer history, meaning more support online. PyTorch has a higher level of excitemnet around it, as far as I can tell. It is used very actively in research and newer open-source projects/courses, which indicates it may become a dominant force.


What is the best way to normalize histogram vectors to get distribution? l have the following sample of 4 vectors of dimension 5 . They are sparse vectors, in a way that each value in a vector represent the frequency (number of occurrence of values). For instance

v_1=[0,4,0,0,1]

4 at index 1 means we have four values of 1 and 1 at index 4 it means we have one value of 4.

The purpose of normalizing these histograms is to get a distribution.

Here is a sample of my data.

Vectors=[[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]]

What l tried ?

vectors=(vectors-np.mean(vectors,axis=0)) / np.std(vectors,axis=0)

and l tried :

vectors=sklearn.preprocessing.scale(vectors)

A)Is it correct what l have tried ?

B)l have question to normalization by subtracting means and dividing by standard deviation:
through all the vectors, through each dimension, means( of a dimension) is computed. Than each data point of this dimension is subtracted from its mean. Is it correct what l'm saying ?
in other way for each dimension, we have it's own mean.

For instance, if we have dim(vectors)=(400,2000) such that 400 is the number of examples.
we compute 2000 means and 2000 std since we have 2000 dimensions. And then each data point is subtracted from its means and divided by its std (mean and std of that dimension)

Thank you for correcting me.

It's okay, what you are saying, except the

  each data point is subtracted from its means

part because it's in the opposite way, you have to subtract the means from their data points.

I guess your problem is coming from that you put your vectors into the matrix as row vectors, but np.mean(..., axis = 0) calculates the column means.

If you want to use the above formula for normalizing then you should transpose your matrix.

vectors = np.array([[0,4,0,0,1],[5,0,0,3,1],[1,0,0,0,0],[0,0,6,0,0]])
vectors.T
array([[0, 5, 1, 0],
       [4, 0, 0, 0],
       [0, 0, 0, 6],
       [0, 3, 0, 0],
       [1, 1, 0, 0]])

And now the np.mean(vectors.T, axis=0) gives the correct means for your vectors:

array([ 1. ,  1.8,  0.2,  1.2])

Substract the means from the transposed matrix like vectors.T - np.mean(vectors.T, axis=0)

array([[-1. ,  3.2,  0.8, -1.2],
       [ 3. , -1.8, -0.2, -1.2],
       [-1. , -1.8, -0.2,  4.8],
       [-1. ,  1.2, -0.2, -1.2],
       [ 0. , -0.8, -0.2, -1.2]])

And finally divide it by the standard deviation of the transposed matrix

t_vectors = (vectors.T-np.mean(vectors.T,axis=0)) / np.std(vectors.T,axis=0)
t_vectors
array([[-0.64549722,  1.65027399,  2.        , -0.5       ],
       [ 1.93649167, -0.92827912, -0.5       , -0.5       ],
       [-0.64549722, -0.92827912, -0.5       ,  2.        ],
       [-0.64549722,  0.61885275, -0.5       , -0.5       ],
       [ 0.        , -0.4125685 , -0.5       , -0.5       ]])

Sklearn gives the same sklearn.preprocessing.scale(vectors.T).


How do I handle with my Keras CNN overfitting In my CNN, I have 700 images of class 0, 700 images of class 1, and 72 validation images.

My code:

visible = Input(shape=(256,256,3))
conv1 = Conv2D(16, kernel_size=(3,3), activation='relu', strides=(1, 1))(visible)
conv2 = Conv2D(32, kernel_size=(3,3), activation='relu', strides=(1, 1))(conv1)
bat1 = BatchNormalization()(conv2)
conv3 = ZeroPadding2D(padding=(1, 1))(bat1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv3)
drop1 = Dropout(0.30)(pool1)

conv4 = Conv2D(32, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(drop1)
conv5 = Conv2D(64, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(conv4)
bat2 = BatchNormalization()(conv5)
pool2 = MaxPooling2D(pool_size=(1, 1))(bat2)
drop1 = Dropout(0.30)(pool2)

conv6 = Conv2D(128, kernel_size=(3,3), activation='relu', padding='valid', kernel_regularizer=regularizers.l2(0.01))(pool2)
conv7 = Conv2D(128, kernel_size=(2,2), activation='relu', strides=(1, 1), padding='valid')(conv6)
bat3 = BatchNormalization()(conv7)
pool3 = MaxPooling2D(pool_size=(1, 1))(bat3)
drop1 = Dropout(0.30)(pool3)

flat = Flatten()(pool3)
drop4 = Dropout(0.50)(flat)

output = Dense(1, activation='sigmoid')(drop4)
model = Model(inputs=visible, outputs=output)

opt = optimizers.adam(lr=0.001, decay=0.0)

model.compile(optimizer= opt, loss='binary_crossentropy', metrics=['accuracy'])

data, labels = ReadImages(TRAIN_DIR)
test, lt = ReadImages(TEST_DIR)

data = np.array(data)
labels = np.array(labels)

perm = np.random.permutation(len(data))
data = data[perm]
labels = labels[perm]
#model.fit(data, labels, epochs=8, validation_data = (np.array(test), np.array(lt)))

aug = ImageDataGenerator(rotation_range=20, width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,
    horizontal_flip=True)

# train the network
model.fit_generator(aug.flow(data, labels, batch_size=32),
    validation_data=(np.array(test), np.array(lt)), steps_per_epoch=len(data) // 32,
    epochs=7)

model.save('model.h5')

It returns these numbers:

Epoch 1/7
43/43 [==============================] - 1004s 23s/step - loss: 1.8090 - acc: 0.9724 - val_loss: 1.7871 - val_acc: 0.9861
Epoch 2/7
43/43 [==============================] - 1003s 23s/step - loss: 1.8449 - acc: 0.9801 - val_loss: 1.4828 - val_acc: 1.0000
Epoch 3/7
43/43 [==============================] - 1092s 25s/step - loss: 1.5704 - acc: 0.9920 - val_loss: 1.3985 - val_acc: 1.0000
Epoch 4/7
43/43 [==============================] - 1062s 25s/step - loss: 1.5219 - acc: 0.9898 - val_loss: 1.3167 - val_acc: 1.0000
Epoch 5/7
43/43 [==============================] - 990s 23s/step - loss: 2.5744 - acc: 0.9222 - val_loss: 2.9347 - val_acc: 0.9028
Epoch 6/7
43/43 [==============================] - 983s 23s/step - loss: 1.6053 - acc: 0.9840 - val_loss: 1.3299 - val_acc: 1.0000
Epoch 7/7
43/43 [==============================] - 974s 23s/step - loss: 1.6180 - acc: 0.9801 - val_loss: 1.5181 - val_acc: 0.9861

When I predict some test images, the result is always 0.

I already tried various things like adding more dropouts (or making the dropout rate bigger), data augmentation, batch normalization etc. and none of these have made it work properly.

What should I do?

as your data is very less, you should go for transfer learning as @muneeb already suggested, because that will already come with most learned parameters and then you can train that model using your custom dataset.

you can try out pre-trained models from here

If you want to go for your existing custom configured model only, try adding another Dense layer before the output.

like a Dense(128, activation='relu').

Share your results here, so that we can explore more.


Dealing with new outliers after capping I'm trying to cap outliers in a column of my pandas DataFrame.
Here's the boxplot for a column of my original data.

So, using code from this stackoverflow answer, I tried capping outliers.

Here's how capped column looks like, with new outliers after upper bound.

My problem now is, I can't keep capping outliers, yes! What do I do now?

Once is all it takes; you're done. The point of dealing with outliers is that they are rare. Forcing your model to fit to those outliers can make its performance against inliers (which it will encounter far more often) worse, and that's A Bad Thing. Although you now have values that look like outliers when you plot them, this is an artifact of your earlier transformation - those values should be within the "normal" expected range of values for your data.


Sklearn: unsupervised knn vs k-means Sklearn has an unsupervised version of knn and also it provides an implementation of k-means.

If I am right, kmeans is done exactly by identifying "neighbors" (at least to a centroid which may be or may not be an actual data) for each cluster. But in a very rough way this looks very similar to what the unsupervised version of knn does.

Then what is the difference between k-means and unsupervised knn?

Unsupervised k-NN

Unlike k-means, the unsupervised k-nn does not associate a label to instances. All it can do is tell you what instances in your training data is k-nearest to the point you are polling for.

For example:

import numpy as np
from sklearn.neighbors import NearestNeighbors
samples = [[0, 0, 2], [1, 0, 0], [0, 0, 1]]

neigh = NearestNeighbors(2, 0.4)
neigh.fit(samples)

neigh.kneighbors([[0, 0, 1.3]], 2, return_distance=False)

  array([[2, 0]]...)

You can see that this returned the index of the k-nearest points, and not the label.

k-means

This algorithm is completely different. The k here denotes the number of assumed classes that exist in your dataset. For example if you have unlabeled pictures of red and green apples, you know that $k=2$. The algorithm will then move the centroids (the average of the cluster distributions) to a stable solution.

Here is an example:

Let's first make some artificial Gaussian distributed data.

import numpy as np
import matplotlib.pyplot as plt

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

print(y.shape)
print(data.shape)

  (1500,) (1500, 2)

plt.scatter(data[:,0], data[:,1])
plt.show()

The k-means algorithm from scratch

class Kmeans(object):

    def __init__(self, k=1):
        self.k = k

    def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

        count = 0
        while count < 100:
            count += 1
            if verbose == 1:
                print('-----------------------------------------------')
                print('Iteration: ', count)

            distances = np.zeros((shape[0],self.k))
            for ix, i in enumerate(data):
                for ic, c in enumerate(centroids):
                    distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

            labels = np.argmin(distances, axis = 1)

            new_centroids = np.zeros((self.k, shape[1]))
            for centroid in range(self.k):
                temp = data[labels == centroid]
                if len(temp) == 0:
                    return 0
                for dim in range(shape[1]):
                    new_centroids[centroid, dim] = np.mean(temp[:,dim])

            if verbose == 1:
                plt.scatter(data[:,0], data[:,1], c = labels)
                plt.scatter(new_centroids[:,0], new_centroids[:,1], c = 'r')
                plt.show()

            if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
                print("DONE!")
                break

            centroids = new_centroids
        self.centroids = centroids
        self.labels = labels
        if verbose == 1:
            print(labels)
            print(centroids)
        return 1

    def getAverageDistance(self, data):

        dists = np.zeros((len(self.centroids),))
        for ix, centroid in enumerate(self.centroids):
            temp = data[self.labels == ix]
            dist = 0
            for i in temp:
                dist += np.linalg.norm(i - centroid)
            dists[ix] = dist/len(temp)
        return dists

    def getLabels(self):
        return self.labels

And the results

kmeans = Kmeans(5)
kmeans.train(data)

  Ranges:  [[-15.42553872  14.88894099]  [-13.33192554  16.15415347]]
  Centroids:  [[-11.39200726 -10.71208054]  [  3.73634888  -8.9230959 ]
  [  6.17589734 -10.66376228]  [  0.78973744  -0.44245535]  [
  9.29524253   9.59127574]]

Initialize, the red points are the random centroids

Iteration 1

and after a few iterations


Number of capsules in the Primary Capsule Layer of Capsule networks What is the Number of capsules in the Primary Capsule Layer of Capsule networks?

In many articles, it is written that the number of Capsules is 32 but in the paper, by Hinton - Dynamic Routing between capsules it is written that
"In total PrimaryCapsules has [32 × 6 × 6]
capsule outputs (each output is an 8D vector) and each capsule in the [6 × 6] grid is sharing their
weights with each other."

which implies that the layer has 36 capsules.

It could be that each capsule is of 2D form (thus 6x6). So each component capsules outputs an 8D vector. The 8D vector is derived by analyzing 8 feature maps propagated from the convolutional layer. So its like placing the feature maps in groups of 8. This means each component capsule represents one capsule therefore in the "dynamic paper", there are actually 32 capsules of form 8D.
NB: The Primary Capsule Layer is actually made up of convolutional layers.
So you can look at the 6x6 as the size of the feature maps after convolution in the primary Capsule layer.
take a look at this site.
Anyway I am still reading wide on capsules, because most of the stuffs in the paper are consolidated. I am open to corrections as well. Thanks


Is it possible to use a pretrained scikit learn model to make predictions on a dataset with different features (than those used during training) Say we have a model trained on dataset A, which has a number of features, as usual. We then persist that model to disk and use it when we need to run inference (make predictions). Usually we run inference against unseen data with the same features. Is it possible to run inference against unseen data with different features. Note that these new features are very similar to the ones used in training. So in a sense it would be a type of transfer learning (actually more like “transfer inference”). Can this be done in scikit learn, given that loaded models (from say Pickle) expect the column names to be the same?

Technically, the only constraint would be the number of features that has to be the same than during the training phase (in most scikit learn models).
Will it perform well ?
I would say it really depends on the type of model/approach you are using and what is behind:

Note that these new features are very similar to the ones used in training

Generally speaking it will probably not perform well unless the training features and the new features are related in a way that is acceptable by your model.


Using Machine Learning to create marketing segments I'm working on a project to try and predict which users would be most likely to subscribe to our services. I'm using R and am working with a data frame that has a user's email address, information about their past purchases (order count and revenue generated), how they have interacted with our website (different events triggered), and whether or not they have subscribed. I'm using this data to train and test different algorithms. I've used caret with knn, lda, cart, and nnet.

I've been able to get how accurate my models are but that doesn't help me know who to market to. When I plug new data into my model I would like to know which users it thinks are the most likely to subscribe and also give me their associated email addresses. I've looked all over but haven't been able to find anything that helps me pair a prediction to an email address. Any help would be much appreciated.

PS.

I'm not using the email addresses to classify the data. I simply have them to associate a row of data to a user. I had to get rid of them to run it through the training and testing algorithms and that is where I am having the issue of knowing what prediction data is associated with which email address and in turn which user.

Wow, what a great question! This is a critical task for anyone who works with data in a marketing function.

First, I'll address your question about pairing the prediction to email. Your email is serving as a unique identifier. When you split your data set, you'll have to keep the email and suppress it from the prediction, and use it to match/compare to the validation data.

If these are new predictions, you could create a new column in your data that contains the "predicted" value. Specifics on this process would be better suited for another community, probably StackOverflow.

Second, I'd like to talk about what you're trying to achieve.

I think there are two questions ways you could approach this with the work you've already done:

Identify who is most likely to subscribe, and focus your efforts on creating nurturing campaigns to "warm" these prospects. The final goal is to get a subscription.

Here you're going to be using those "events" that you mentioned. What exactly do you have? in the past. I've just dumped web activity data and started working with that. You'll have to engineer a lot of features to capture information that your web activity won't report on (these could be # of logins, # of checkouts, time since last visit, this is where you need to get creative).

Take this data and try to make some predicte who is likely to subscribe. In some sales industries, this is refereed to as a lead score.

Since you're using R, you could try using XGBoost -https://cran.r-project.org/web/packages/xgboost/vignettes/xgboostPresentation.html. Since you're trying to predict a binary variable, you could use their decision tree.

This is going to take a while, so don't get frustrated if you don't this to production in the first week (or month). Here's some reading:

https://datastories.com/gallery/predictive-user-scoring-example

https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/

https://rdatascientist.wordpress.com/2015/01/26/predictive-lead-scoring-using-r-first-of-a-two-part-blog-series/
Identify which types of customers you want to acquire, and focus on building campaigns to engage these types of customers. The final goal here is to acquire quality customers.

This is more marketing and strategizing than it is machine learning. Don't get me wrong, there is maching learning in identifying the clusters. Here's a great example I used: http://www.kimberlycoffey.com/blog/2016/8/k-means-clustering-for-customer-segmentation.

Once you identify what the different segments and what behavior they exhibit, you have to team up with the other stakeholders and try to decide what kind of customers do you want? Whats the benefit is getting a customer who makes one large purchase a year vs. a customer who makes five smaller purchases a year? You'll have to take your time identifying customers/prospects who are the most valuable, and plan a strategy around that.


Identifying if the sentence if it comprise information about education Given a sentence I am trying to classify if the sentence contain information about education. For example:

sentence1 = "Require minimum four years of professional degree."
sentence2 = "no degree required for this job."

For identifying as a first step I have built a vocab which have the set of keywords for identifying education based sentences. I was partly successful until I have problems with sentences like this.

sentence3 = "BE or BTech or any degree equivalent to it"

In my vocab 'BE' is also a keyword as 'BE' represents bachelors of engineering degree (in case of country India). As the parsing of algorithm is done in lower case, the issue is coming with 'BE' will be 'be' in sentence. Because of that every sentence that contains 'be' is recognized as educational sentence.

To built a strong ML model I dont have much data. If I want to use vocab based recognition, for this I have to understand the words beside them in the sentence.

Are there any built models so as to import and identify such scenarios or labelled dataset available for it ?
Are there any methods for accomplishing such task ?

I think what you are looking is to differentiate between 'be' and 'BE' based on context. Word2Vec is a good place to start but to determine the difference between words based on contexts is 'Sense2Vec.' Word Sense Disambiguation paper is worth to look at. I am not sure if this is what you're looking for because if you have 'be' in every sentence which is most likely, then this won't be an ideal solution. You might have to do some extra data preprocessing like changing 'be' to 'Bachelor of Engineering' based on your sentence.


Chi-squared for continuous variables I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into the scipy.stats.chi2_contingency module. This module returns the chi-squared value and the p-value.

I have acheived reasonable results with boolean variables, but I am suspicious of the results for categorical variables with more than 2 categories.

Specifically, I am fairly sure that one continuous feature, age, is correlated with the target, to some level of significance. From plotting histograms and KDEs, I know that the probability distribution of the feature for (target = 0) is quite different from the probability distribution for (target = 1). However, when I bin the age feature into 2-7 bins, the chi-squared test yields a p-value of ~1e-39.

Is there anything that I am missing with regards to the chi-squared test and categorical variables? Does this test only work for monotonic relationships?

It sounds like the chi-squared test confirms your suspicions about age being correlated with the response.

As far as I am aware, the null-hypothesis for the chi-squared test is "there is no relationship" between the two variables. The test statistic is calculated based on the assumption that all observations are evenly distributed amongst the cells in the contingency table, so the test should work for most kinds of relationships.

A word of warning - the test is sensitive to data imbalances.


Does the SVM require lots of features most of the time? So I know about the curse of dimensionality (too many features too less data).

Say I have a 3000 sample dataset, would 3 features be too less?

I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.

Also, it seems easier to guess to the "positive" side, i.e. "might it work?" then "might it fail?".
I witnessed both small datasets (<1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (<5) features.

The stage in the processing pipeline to tackle this, is the feature selection one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)

You didn't mention a specific research environment, but I'd like to suggest scikit-learn sklearn.feature_selection.SelectKBest running inside a GridSearchCV pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:

from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

pipe = []
pipe.append(SelectKBest())
pipe.append(LinearSVC())

est_pipe = make_pipeline(*pipe)
print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())
param_grid = {
  # some svc grid params..,
  'selectkbest__k': [5, 10, 50, 100]
}

grid = GridSearchCV(est_pipe, param_grid=param_grid)
grid.fit(X, y)


Compound Annual Growth Rate Function in R Statistics I need to calculate the compound annual growth rate for two periods of time: 2014 & 2009. I'm currently working with Excel, I have two columns with data for each year (One column with only data of 2009 and the other only with data of 2014), where each row contains info for 2009 & 2014 for certain category.

I've been trying to calculate the CAGR between the data for each row, but I can't seem to find a function to do precisely that in "R".

I found a package with a growth.rate function for time series, but it doesn't take my data in the order I need (it uses bayes method for velocity of growth, which I don´t need).

Is there any package in "R" where I can get especially that rate? Any tips in how I can accommodate the data so that R reads row per row and calculates like that?

Instead of depending on a library, you can create your own vectorized function:

Code:

library(tibble); library(dplyr)

example_data <- tribble(
  ~Values_2009, ~Values_2014,
  10000, 19500,
  10500, 15000,
  25000, 35500
)

CAGR_formula <- function(FV, PV, yrs = 4) {
values <- ((FV/PV)^(1/yrs)-1)
return(values)
}

formula is (FV/PV)^1/n -1

Results:

example_data %>%
  mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)
#> # A tibble: 3 x 3
#>   Values_2009 Values_2014      CAGR
#>         <dbl>       <dbl>     <dbl>
#> 1       10000       19500 18.170386
#> 2       10500       15000  9.326511
#> 3       25000       35500  9.162151

# If Percentage is needed use the below mutate instead
#mutate(CAGR = CAGR_formula(Values_2009, Values_2014, 4)*100)


Real time noise removal using Savitzky-Golay Method I would like to ask if Savitzky-Golay can be implemented on real-time data.

I have used it on a fixed array size, but would like to extend it to output values for real-time sensor data. Can anyone refer me to appropriate implementation or hint online implementation. Thanks.

The original Savitzky-Golay paper addressed smoothing, meaning that you estimate a value for some point in the past, using values from its past and future. It sounds like what you're interested in is filtering:  at time step k, you want to reduce noise and come up with a better estimate of the value at the same time step k, without knowing the future.  That's important in many fields such as process control, and some fault diagnosis where delays can't be tolerated.

The SG approach of calculating the constants for least squares fitting for smoothing has also been done for filtering.  A curve is fit through n recent points, and instead of estimating the middle point, the end point is estimated by a least squares fit, reduced to simple multiplications by constants. There are tables of the necessary coefficients and an example at
https://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/LeastSquares-Filter/leastsquares-filter.htm

There is no time delay in the calculations, unlike with smoothing.  Also, if the input signal is a ramp, the filter will converge to the correct value once all of the points in the time window reflect that ramp.  On the other hand, there are some side effects that may or may not matter to you.  Even in the linear case, for a step change in the input, there is overshoot in the filter output.  Higher order least-squares filtering can be quirkier, so it is rarely used. But the linear version overall is quite effective, assuming the little bit of overshoot isn't a real problem  (for instance, if you don't see step inputs in the data anyway).

The link above is part of a more general "Guide to Filtering", that covers some other common filters as well, such as the exponential filter and moving average filters in particular.  (Filters don't have time delays, but most do lag the inputs - meaning smear out changes over time.  That's a byproduct of filtering out the noise, essentially cancelling out some new noise by a certain amount of averaging against old noise).  It also covers general issues for filtering, such as aliasing.

There are two other filters that are not covered in the Guide that might be considered close competitors to the SG-style filtering:  the alpha-beta filter and double exponential smoothing.  These are covered in Wikipedia:  alpha-beta filtering has its own page, and double exponential smoothing is part of the page on exponential smoothing.  The goal of both is to filter/estimate both the values and the derivative, resulting in a filter that will give correct outputs to a ramp input signal.  I've long suspected that these two filters are essentially the same, but never taken the time to look into it.


Machine Learning in real time I am a newbie in ML world, but very curious and enthusiastic about it. Have gone through articles and some hands-on too. Still got a silly doubt.

In sample datasets (like Iris or diabetes or breast cancer etc.) and exercises, I find the data to be well-formed and ready to be digested by ML model (algos). But, in reality, it is much more than that.

For example, Amazon's feature of suggesting (cross-selling) products. Based on my previous searches and orders, it probably would be looking into several features and predicting further items. I'm curious -

How does it work in real time?
How my product searches (they are not numbers) are fed into the ML model under-the-hood?

Please share your thoughts.

Regards,
RB
(using Python 3.6 & latest-1 versions of NumPy, Scikit and Pandas)

OK, first regarding the real time part.

Machine learning (particularly supervised learning) has two parts

Training
Predicting

Training is a complex and slooooow process, you need to figure out a good model, then train it with the correct data... not simple and computationally expensive.

Predicting however is MUCH simpler, as normally consist (in the world of neural networks) of a simple forward pass in the neural network, and a forward pass is, at its core, a bunch of matrix multiplications. So once a model has been built and train, predicting something using the existing model is quite quick, also one can retrain the model with new data too.

Now, regarding with the text searches and how to deal with them. There is a very cool idea called word embeddings, it simply consist on transforming words into vectors. Imagine a 2d grid (like the x,y coordinates we studied in school), now imagine that the vertical axis (y) denotes how powerful something is, the hifher the y value, the more powerful it is. And now imagine that the x axis denotes the gender of something. (lets assume negative x means male and positive x means female).

And now thing of the word : King. King is male (x=Negative value) and powerful (y=Positive value). With those two coordinates, and starting from position 0,0 you could have a vector.

Now think of Queen, in this case the coordinates will be x=Positive value and y=Positive value, now you have another vector.

And now think of the word Spartacus (he was a slave in ancient rome), so that word will have x=Negative and y=Negative.

Think of this, suddenly not only you can represent the words as numbers, you can also represent them with vectors, which allows you to possibly figure out that King and Emperor are both SIMILAR words.


How should I read the following heatmap? I've been playing around with the linear regression, and I was thought that before commencing it's always good to plot an heatmap to see whether there are features that somehow is worth testing for their significance/relationship.
Would you agree with the above?

With that being said, after having refined a tiny bit my data, I obtained the following heatmap:

As you can see I now have two features completely blank. I though in the beginning to some sort of rendering glitch, which is not the case.

In reviewing the dataset, I found out that the features have always the same value across the dataset.

I'd assume this is not being printed out due to the fact this feature has became statistically significant for every feature.

Would you say this is a correct way of referring to this? Or is there any other way to express this concept?

This is however not the case for the other feature, which contains a mix of numerical value, hence I'm not able to explain.

Can you please advice?

Since the variables isPWKinText and H1_2_Len have the same value for all examples in your dataset they have zero variance and contain no information. There is simply just no inference you can make based on them. That is why they are not relevant and shown blank.

Please note, that this depends on your dataset, of course. The two variable might just be constant for the examples you are currently looking at. For example, if this is an analysis of your training data maybe your datasplit is just really bad and the validation or test set contains examples for which these variable do show other values. That is something I suggest to double check.


Working with audio data with different sample rates in Tensorflow I am trying to implement (as a toy project) some aspects of speech recognition in Tensorflow. The audio files I want to use as training and test data have different sample rates (16, 20, 44 and 44.1 kHz).

The following list is not exhaustive, just listing a few options I could think of.

I can resample the whole signal (which is quite un-periodic) using scipy.signal.resample, probably after padding with zeros to achieve a signal length which is a power of 2, chopping of the equivalent amount afterwards.
I can try to chop the signal into short, non-overlapping windows, which I modify using scipy.signal.resample before glueing them back together and applying the rest of the chain.
I can chop the signal up into overlapping windows, scipy.signal.resample them and stick them together in a weighted fashion.
I can take the spectrogram tensor obtained using tf.contrib.signal.stft and try to stretch it, for example using rational function interpolation (reflecting tube models of the vocal tract)
I can assume that the model will learn to abstract away from sample rates, maybe even helping it (or hindering?) by adding convolution layers before the actual model I am interested in.

In which step of the model should I normalize sampling rates, and how should I do it?

If you have ffmpeg installed, you can just use tf.contrib.ffmpeg.decode_audio. That function takes a samples_per_second argument and does the resampling for you. The following calculates a log magnitude short-time spectrogram of your data.

waveform = tf.squeeze(
    tf.contrib.ffmpeg.decode_audio(
        tf.read_file(tf.placeholder(tf.string, name="filename")),
        file_format=tf.placeholder(tf.string, name="format"),
        samples_per_second=p["sample_rate"],
        channel_count=1))

log_mag_spectrogram = tf.log(tf.abs(tf.contrib.signal.stft(
    waveform, win_length, hop_length,
    n_fft, pad_end=False) + 1e-8))


How can I have an "undefined" category in multi-class classification I'm trying to classify several websites by category (finance, health-care, IT, etc...). I have at my disposal the content of the pages of the websites, and I use the words to classify. For now, I have manually classified some website to train a naive bayes model.

As I'm mostly interested in a high precision (e.g a classified website must be in the right category), I would like to add an "undefined" category in which a website would end up if it is not close enough to the other categories. To be clear, it's not a problem for me if a website is not classified, it's a problem if it is misclassified.

Are there algorithms that would allow this, or a way to train an "undefined" category ?

My best guess for now is to train something like a random forest and define a "minimum score" below which a website is "undefined".

Absolutely you can create an approach that forces high-precision class tagging algorithm (at the natural cost of recall). What's more- you can do this with (at least) any method that provides a percentage calue for predictions, which is the vast majority of classifiers. The key is, as you mention, to find the minimum acceptable value of precision and cut the predictions at that value.

If a minimum precision is your only constraint and your solution is not sensitive to the recall (getting all or the highest possible proportion of the websites correctly classified), this is a very simple matter. Some lower percentage of your observations will be classified, but those that are will be more likely to be correctly classified.

For example- if your Precision floor is 70%, your cut could look something like this:

Obeservation predictions falling into the green segment could be classified as positive examples, and predictions falling into the red segment would be unclassified.

This approach would be sufficient for Naive Bayes. Some other approaches (SVM, gradient boosting machines etc) may benefit from a custom loss function definition, in which you define a function that disproportionately punishes false positive predictions.

Something like:

$(y_i = 0) \rightarrow (d = s)\wedge (y_i = 1) \rightarrow (d=1)$

$L(p) = \frac {\sum_i\frac{(p_i-y_i)^2}{d}}{n_i}$

$L(p)$ = loss function

$p_i$ = predicted class likelihood

$y_i$ = actual class (0 for negative example, 1 for positive)

$n_i$ = number of observations

$s$ = Penalty parameter for false positives.

Would heavily punish false positives relative to false negatives. It can also be adjusted to your needs, to more or less heavily punish false positives. For $s = .5$, the function looks something like:

Please note that this function does not necessarily define the approach you should take, but only one possible approach. To create a custom loss function perfectly tailored to your use case, you'd want to know the relative cost of a false positive and a false negative, and customize the function accordingly.


How far can one go with excel? in my business we handle all analytics through Excel. This includes mostly scheduling, production planning and accounting operations. We currently are looking into adding a bit of predictive modelling and Excel does suffice to a point, but doesn't have support for complex models.

As I see it, the main benefits of Excel are its ease of use and the ease with which you can find and train people to get accustomed to it. On the other hand more sophisticated environments (e.g. R, python) can handle a wider variety of analytics tasks, but require better trained individuals.

I have also read this question on if Excel is sufficient for Data Science and while it is a bit out of scope for my purpose, the conclusion is that tools like R and Python are much better than Excel.

My question is (in the context of data analytics): "How much far can we get with Excel, without needing to change to a more sophisticated tool?" or "At what point do we need to migrate from Excel to -let's say- R?"

Thank you very much!

TL;DR

If you have unlimited time and use a 64-bit version of Excel, you can get as far with Excel as any other data analysis tool.

Time

I mention time as my first factor, because Excel only has basic funcitonality built in, such as summing, random number generation, lookups etc. These correspond to a kind of standard library, which Python and R also have. Using these basic functions, with enough time, you can build up pretty much any analysis tool out there. Don't expect good runtime performance. In Python and R, however, there are many many packages that people have already created, which perform well and have been tested by lots of people and so are trusted.

Memory

My second point about 64-bit Excel is because that allows a lot more memory to be used by a single instance of Excel. It allows many more cells to be filled. Using 32-bit excel will limit you to projects of around 2Gb. That is a fair amount of data, but it is a hard limit.

Then steps in 64-bit Excel, which basically means no more memory limits - only those that come from your hardware, and that means Python and R will also be stopped in their tracks.

To provide some numbers, we can simply compute the number of bits able to be stored in each version. Here in Python's interactive prompt:

In [1]: (2**32) / 10**9          # 10^9 means the result is 4.3 Gb
Out[1]: 4.294967296

In [9]: (2**64) / 10**18         # 10^15 means the result is 18.4 Eb
Out[9]: 18.446744073709553

Eb means exa-bytes. This means 18.4 million million million gigabytes.

I notice the computation shows 4Gb for 32-bit, while I said 2 Gb above. I read there is a kind of hard limit on the 32-bit version. I don't know or care why that is... I use Python and R  ;-)

In any case, I hope that is enough to convince you that memory is not an issue, if you are a brave person willing to invest all your time building tools from the ground up!

Summary

If you have complicated business logic, where the actual analysis is mathematically simple, stick to Excel. Business people will love you for it.

If you want to do more than linear regression, use Python or R.

Caveats

As far as I know you cannot run remote or distributed tasks using Excel, whereas that is relatively easy using Python and (a little less so in my opinion) R. So at that point, I would give up on Excel. You'd likely have to implement your own tools in C# or C++ using the .Net framework.


Higher variance in PCA can mean, that data structure is less informative? I have the possibility to describe data with two different data structures, both data structures are some sort of approximation to the true data. I would like to compare the two data structures with respect, how much one explains variance in one data structure to the other.

Therefore I decided to use PCA, nevertheless, I don't know if it safe to say such statement: If the variance explained by the same number components N (e.g. 2) is higher in a data structure than in the other, the data structure loses some part of the information. For example as on the image below, if Data_structure_A gives more variance, then it loses more information than Data_structure_B?

It is not the best use of Principal Component Analysis (PCA) to compare information loss for different data structures.

It would be more appropriate to use Information Theory. Information Theory is the study of data compression. It provides many tools to make quantitative comparisons across different encoding schemes.


Does Python have R's tidytext equivalent? I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.

Scikit-learn has a great implementation of latent dirichlet allocation, which I would argue is as straightforward to use as the implementation in tidytext. There’s a tutorial here.

Also, Python has SpaCy, which is slicker than anything R has so far in terms of tooling for NLP pipelines,

I do love R, and I feel it’s still a better language for tidying and processing data than Python. Tidytext is currently nicer than anything in Python in terms of getting data in and out of topic models. However, Python is a lot better resources than R for text mining overall.


How to measure the performance of an imputation technique I would like to know how I can measure the performance of an imputation technique. I have read a lot about this. Most literature on the web are applying a classifier after the data has been completed. So this classifier will be used in order to make predictions. However, I am not interested to use such classifier to make predictions.

I would like to know how good an imputation method performs on a dataset. We can measure this for example my variance, mean, mean squared error and so on. I would like to know if there are similar techniques to measure the quality of the data. I will use the dataset for descriptive analysis(not for predictive analysis, e.g. training classifier)

Please correct me if I am thinking in the wrong context/corner. Thanks in advance.

Best Regards

I think there is no answer to your question since there is no absolute universal "good". Everything depends on the question you ask and the tools you use. This is why there are a lot of imputation techniques. There is no replacement for a missing value. However, in the constrains given by your question and used tools, you can think of imputation which does not alter your answer or at least measure some effects of the missing values. I will give some simple examples.

One widespread technique is to replace missing values with a marginal central estimator for each variable. This might work for a classifier like decision tree, if there are no missing patterns in your data. However using this imputation would alter your confidence intervals if you study linear relations between variables using regression. This is simply because it will alter sample variance.

Another much sophisticated imputation method is to use EM algorithm to fit the maximum likelihood estimator of the variance-covariance matrix. This estimator is unbiased and using this variance covariance matrix you can recover the linear model in an unbiased way. You then can go to analyze linear relations. But this works only for linear and log linear models and requires a lot of data and also requires to have missing data at random, which is not always the case.

Another one is multiple imputation. What you actually do is to draw data at random for missing values according with a supposed distribution. You do that multiple times, let's say at least 30 times. You analyze each data set with traditional tools and later you aggregate those results into a single set of results. This works well mostly when data is missing at random, but is tedious, sometimes inconsistent and duo to randomness it can produce different results. Ultimately, sometimes is very hard to find proper ways to aggregate the results, it depends on the analysis you use.


PyTorch CNN network outputs homogeneous results I am a beginner at data science and I got a project where I want to do nlp via a convolutional neural network in PyTorch. The problem is that regardless of what comes out of the convolutional layers, the output from the perceptron is always repeated. It is as if it wasn't seeing anything.

I already tried to change the dataset, number of variables, loss function, optimization method and add or remove layers and nothing seems to happen. It's been a whole week that I'm working on this and I can't figure out what's wrong.

The loss function in this case is torch.nn.MultiLabelSoftMarginLoss, which is why I removed the Sigmoid activation on the very last layer. The optimization method is SGD. It converges within 15 runs or so.

Here's a picture of what the data looks like after it goes through the convolutional layer (I put the tensor layers side-by-side):

The height is the number of channels and the width is the number of features times the batch size.

Data for testing

Here are the y,X pairs I'm using for the training. I used torch.save to create the files.

y:  https://file.io/JiLnim

x:  https://file.io/NKecI1

Reproducible Example

import torch
from torch import nn

T = torch.load("X_ex")
y = torch.load("y_ex")

class model(nn.Module):
    def __init__(self,Test,n_cat=1):
        super(model, self).__init__()
        self.last_kernel = 3
        self.convolutions = nn.Sequential(
            nn.Conv1d(Test.shape[1], 500, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(500, 400, kernel_size=7, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=3),
            nn.Conv1d(400, 300, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(300, 200, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv1d(200, 100, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=self.last_kernel),
            )
        self.test_conv = self.convolutions(Test) #I know it's dumb but I just wanted to get it working
        self.p_channels = self.test_conv.shape[1]*self.test_conv.shape[2]
        self.perceptron = nn.Sequential(
            nn.Linear(self.p_channels,100),
            nn.Sigmoid(),
            nn.Linear(100,16),
            nn.Sigmoid(),
            nn.Linear(16,n_cat),
            #nn.Sigmoid()
        )
    def forward(self, x):
        x = self.convolutions(x)
        x = x.view(x.size(0), -1)
        x = self.perceptron(x)
        return x

m=model(T,y.shape[1])
loss_fn = torch.nn.MultiLabelSoftMarginLoss(reduction="sum")

r = 0.001 #learning rate

optimizer = torch.optim.SGD(m.parameters(), lr=r, momentum=0.9)

for t in range(15):
    y_pred = m.forward(T)
    loss = loss_fn(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

#Final y:
y_final = torch.sigmoid(y_pred)

Output (first few)

In [1]: y_actual
Out[1]:
tensor([[0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        ...,
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698],
        [0.0794, 0.0734, 0.0880,  ..., 0.0817, 0.0864, 0.0698]],
       grad_fn=<SigmoidBackward>)
In [2]: y_actual.max(axis=1).indices
Out[2]:
tensor([24, 24, 24, 24, <keeps going>, 24, 24])

So, I think I figured out what the problem was. It had nothing to do with the algorithm in itself but the data. The data which was being fed to the network was a sparse tensor; the thing is that the network wasn't being able to learn from its mistakes because every time it got it wrong it received a zero correction to the weight.

Because of that, I simply made a transformation $X\mapsto 2X-1$ on the tensor, switching the zeros by -1's and the networks started giving heterogeneous results.


How to use class_weight parameter for validation set? I am using Keras' class_weight parameter to deal with an imbalanced class problem. I am doing this to define the weights :
weights = class_weight.compute_class_weight('balanced',np.unique(trainY),trainY)

then, in my network:
model.add(LSTM(..., class_weight=weights,...,callbacks=callbacks_list))

However, I also use a callbacks list to prevent overfitting, by imposing EarlyStopping based on validation_accuracy ...:
earlystop = EarlyStopping(monitor='val_acc', min_delta=0.001, patience=5, verbose=1, mode='auto')
callbacks_list = [earlystop]

However, my weights are only defined based on my training set... But my validation set also contains imbalanced data, in different proportions than the training set, and I would also like to give appropriate weights to have a fair evaluation of val_loss parameter...
So my question is

With which weights is the val_loss computed ? With the weights given in class_weightparameter ? Or is this parameter only used for training loss ?

How could I define weights that could be used for the validation set, to have a more accurate value of val_loss ?

Thanks

You are not supposed to apply weights to your validation set since it is supposed to measure your model's performance. If you'll do that you'll probably get better results for validation but once your model is deployed it will perform worse on new data.
Weighting, resampling techniques etc. - they all should be done on the training set only!


Intuition: why are ReLu activations boundary lines linear? What is the reasoning behind ReLu boundary lines appearing linear when plotting in 2D? Does this generalize to higher dimensions in that boundary lines in large dimensions are linear hyperplanes as well?

For instance, in tensorflow's playground any time ReLu is chosen as the activation function, the resulting boundary line is made of lines.

I'm failing to see the connection between the underlying neurons outputting only z or 0 and how that connects to the output plot.

Thanks very much for any thoughts/help!

As far as i understand, relu gives linear boundary because it is linear for X>0. If you expand it to write it in the form of an equation like Z=f1(x1)+f2(x2)....fn(xn), we simply get a linear equation even if we consider the dead neurons. The part where x<0 makes sure that no -ve signals are carried forward but dosent add any non linearity to the final equation. The effect of dead neurons probaly the the final coefficients of the equaion will have coefficients with slightly less value than had the activation been just a normal fuction since there is no negative component which is added to a neuron.


which script can be used to finetune BERT fro squad question answering in hugging face library I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used now for fine tuning?

A recent PR changed the location of the scripts you are looking for to examples/legacy/question-answering


Corelation between overtime and sick leave I have a scenario where I have to identify employees, who when take sick or any other paid leave his/her colleague (any other employee) gets overtime.

My data set is as follows:

employee_code----period---Leave_hours--- Overtime hours

How to identify the relation between overtime hours and leave_hours between employees?

Which machine learning model can be used to get employee wise relation...?

I don't see why you need a model at all.  This seems like a simple counting problem.  Here is SQL that finds pairs where one employee takes leave and another gets overtime:

select leave.employee_code, overtime.employee_code, count(*) as occurrences
from time_off leave
join time_off overtime
on leave.employee_id = overtime.employee_id
   and leave.period = overtime.period
where leave.leave_hours > 0
  and overtime.overtime_hours > 0
group by leave.employee_code, overtime.employee_code
order by occurrences desc

In dplyr:

time_off %>%
  filter(leave_hours > 0) %>%
  select(employee_code) %>%
  inner_join(time_off %>%
               filter(overtime_hours) %>%
               select(employee_code),
             by=c("employee_code", "period"),
             suffix=c("_leave","_overtime")) %>%
  group_by(employee_code_leave, employee_code_overtime) %>%
  summarize(n=length(employee_code_leave)) %>%
  arrange(desc(n))

Note that I haven't run this code, so it might have typos or bugs, but it should be close to correct.

To explore the relationship between leave hours and overtime hours, you don't even need the  employee pairs, you can just aggregate by period and make a scatter plot.


Would there be any reason to pretrain BERT on specific texts? So the official BERT English model is trained on Wikipedia and BookCurpos (source).
Now, for example, let's say I want to use BERT for Movies tag recommendation. Is there any reason for me to pretrain a new BERT model from scratch on movie-related dataset?
Can my model become more accurate since I trained it on movie-related texts rather than general texts? Is there an example of such usage?
To be clear, the question is on the importance of context (not size) of the dataset.

Sure, if you have a large and good quality in-domain dataset, the results may certainly  be better than with the generic pretrained BERT.
This has already been done before: BioBERT is a BERT model pretrained on biomedical texts:

[...] a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement).

Of course, other factors may be taken into account in the decision to pretrain such a model, e.g. computational budget.


do autoencoders work well for non images? I have a classification problem for which a feedforward, fully connected neural net works reasonably well (two classes, true positive and true negative rate close to 80%).

I want to get these rates to 90%, and more features is one of the catalysts for improvements I can think of.

Do autoencoders to learn additional, interesting features work well for problems that do not involve images?

Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are lots of parameters, and not every configuration will work well. Even the optimization algorithm is a hyperparameter.


Monthly data versus yearly prediction How would you see the relation of a monthly variable categorical with a yearly target value float for differents years in order to predict.

In the farming industry the relation is not so simple for categorical values as the drought level , like pre-harvest period has more weight for some months on the yearly result because some months are more relevant.

What kind of null hypothesis test would you do for with both variables to conclude
I was thinking of anova
Any other suggestions.

You could consider each month as an explanatory variable. And the year as response variable. Because, for each year, you'll have 12 monthly variables.

With a model, you could find the relationship that has each month value with the final output. For that, you can perform a linear model (your categorical variables will or must be recoded as dummy variables for that estimation).

If you want to check the effect of the variable by itself. Yearly Target = a * Monthly Variable. I agree with @Leevo.


How to form and minimise custom features for classification in supervised learning I am having an issue in understanding how to form the features based on particular math formula, and how to adjust the weights with.
The aim is to draw ellipses for each unique category of points. Ellipse keeps inside as much of possible selected type of dots, and as little as possible other types (in the picture attached selected yellow ones).
Here is the a formula, with 5 variables, based on which we can draw any ellipse.  https://www.desmos.com/calculator/65hlgysuy7

I can make a model and calculate based on the data set whether the particular point is within the ellipse (score less than 0) or outside, and create accuracy based loss function, or just to use cross entropy for the loss function. Then i could find derivative for cross entropy, but then what's next? How basically i could adjust the weights in such case?

The the big question how i could manipulate those variables (in the link a,b,c,l and h) to form different models, and optimize them with gradient descent (if thats the appropriate technique for such task)?

Can someone please explain me the logic of how custom features in such case can be formed and weights found with gradient descent?

https://en.wikipedia.org/wiki/Support-vector_machine

polynomial kernel of degree 2, with C coefficient >.1 but you would have to figure it out based on your data

I can provide more info on request


Predicting Bitcoin price with LSTM (and other RNNs) In order to try what i learned about LSTM, i downloaded a simple bitcoin price dataset, and tried to make a network which predicts the bitcoin price by feeding a sequence of price history.
Looking for this topic on Google, it seems a lot of people have tried it, and they get the same results as I do : the LSTM seems to learn the identity function. Indeed i remarked that when predicting the prices with LSTM, the predicition seems to be the original price curve, but shifted to the right. (basically the LSTM just repeat the last known price)
And this is kind of logical, if you were to try to predict the current price of bitcoin given the previous prices, just by repeating the last known price you would be close to truth a lot of times.
I have tried changing my loss function in order to avoid the network to learn the identity function, but it didn't work. Has any work been done on that before ? Is it a known issue ? All the blog article which were using LSTM didn't mention it, tho it was obvious from their graphs... (e.g. http://www.quantatrisk.com/2020/04/01/how-to-predict-bitcoin-price-deep-learning-lstm-network-python/ prediction at the end)

I'm not particularly expert in this but I'm quite sure that the variations in the price of anything depend mostly on external factors: news of the day, economic indicators, stockmarket movements, etc.
As a rule of thumb, if a human with a lot of time can't do it, usually a ML model can't do it either. In this case if an expert in finance is given the history of the price but no access to any other information I doubt they can predict anything other than a conservative guess that the price will stay the same.
I don't see how the model can do anything better than this without some good indicators as features, and probably it would need a large number of complex indicators. For example I would assume that news like these affect the Bitcoin price, but gathering this kind of information and integrating it into a model certainly requires a lot of work.


Random Forest, Type - Regression, Calculation of Importance Example I am trying to use a Random Forest Model (Regression Type) as a substitute of logistic regression model. I am using R - randomForest Package. I want to understand the meaning of Importance of Variables (%IncMSE and IncNodePurity) by example.

Suppose I have a population of 100 employees out of which 30 left the company.
Suppose in a particular decision tree, population is split by an attribute (say location) into two nodes. One node contains 50 employees out of which 10 left the company and other contains 50 employees from which 20 left the company. Can someone demonstrate me a calculation of %IncMSE and IncNodePurity. (if Required for averages etc., please consider another decision tree)

This may look like a repeated question but I could not find a worked out example.

MSE is measure of error of the overall regression model, $\frac{1}{n}\sum\|y_i-\hat y_i\|^2$.

For an important variable, if it is replaced with random noise, you would imagine MSE with the faulty data to increase. IncMSE (Incremental MSE) for a particular variable is how much the MSE will increase if the variable is completely randomized.

This is usually computed on the out-of-bag data.

Node purity is a measure of how homogeneous a node is. An example of node purity is information entropy, i.e. $-p_1\log p_1-p_0\log p_0$ if there are two classes.
For regression models, node impurity is usually taken as the variance in a node.

Everytime you split a node, you do it to make the new nodes homogeneous, hence the purity increases.

IncPurity of a variable is weighted average of incremental purity because of each split by this variable was used to split, with the node population as the weight.


Logistic Regression vs SVM Following Andrew Ng's machine learning course, he explains how we can modify logistic regression to obtain SVM algorithm. First he replaces (sort of approximating) cross entropy loss with hinge loss as shown in the image below:

Then he removes the $\frac{1}{m}$ coefficient and divides the whole cost function by the regularization parameter $\lambda$, which leads to parameter $C$ behind the sigma ($C=\frac{1}{\lambda}$). Resulting cost function is shown in image below which is the minimization objective in SVM:

Later he shows that by choosing a very large value for parameter $C$, SVM will be a large margin classifier. Large value for parameter $C$ is the same as choosing a small value for parameter $\lambda$ in logistic regression. So, is logistic regression with a small value for $\lambda$ also a large margin classifier?

Logistic Regression is a large margin loss not classifier. (in literal sense, meaning there is a margin with the loss function not the classification boundary

There are two terms in the softmax loss, true values minus the exp or log of the observations (roughly and depending on the variant of loss) this just implies that distance of observed example from its true decision boundary needs to beat the log sum of the distances from all of the decision boundaries.

Because the softmax function is a probability distribution, the largest the log softmax can be is 0, so the log softmax returns a negative value (i.e. a penalty) that approaches 0 as the probability of the true class under the softmax function approaches 1.


Orange Web Services - Problems installing imblearn module for Python Script I am trying to implement SMOTE in Orange Web Services, but having troubles with the imblearn module.

Following screenshot is from Orange Python Script widget:

from imblearn import under_sampling, over_sampling
from imblearn.over_sampling import SMOTE

I can import it without problems with Python:

The one in Orange's Python Script widget and the one in cmd are obviously not the same Python environments. Confirm by inspecting:

import sys
print(sys.executable)

Fix the issue by installing imblearn into the Orange's Python environment.


Which classification model to use on large, high-dimensional dataset? I face a classification task: with several features a target features is to be predicted. I'm working with python.

My dataset includes 60 features from which I picked 16 which I think could be relevant (many others are time stamps, for example). The problem is that most of these 16 features are categorical - encoding them with get_dummies generates 886 features.

The data also includes about 17 million observations.

I am now wondering about how to tackle this problem and what to research and try next. I summarized this in these two questions and I'd love to hear some opinions!

First. If possible, I'd like to reduce the number of features. I tried using SelectFromModel with a RandomForestClassifier which worked okay, I think, as features were reduced drastically without much loss of prediction power. However, as my categorial features are split into several new features, only parts of the original features are selected (it's not einther all or none of the features that originated from one). Is this a problem? If so, can this be avoided?

Second. As one model can be tuned a lot by playing with parameters or input representation, I would like to focus on a few promising models. For faster results, I used only 200,000 observations to train KNeighborsClassifier, LogisticRegression, LinearSVC, DecisionTreeClassifier, RandomForestClassifier, GradientBoostingClassifier and MLPClassifier (neural network); all from sklearn.

I would not continue to pursue KNeighborsClassifier and GradientBoostingClassifier as they already took a lot of time to train for this small subset.
As RandomForestClassifier is supposed to perform nearly always better than DecisionTreeClassifier, I would drop the latter, too, but stick to RandomForestClassifier.
The two linear models LogisticRegression and LinearSVC worked well with penalty = l1, but very badly with penalty = l2, so that I would continue to pursue both with penalty = l1 (altough I expect similar results).
My first neural network performed very badly, but I guess there a plety of things to try for improving; however, I expect a very long training time with the full dataset (as this small subset took quite a while already).
I did not try a naive bayes classifier (as it is supposed to perform worse than linear models anyway) or a support vector machine (as they are supposed to perform badly with many observations).

Summary: I would continue my work by looking at RandomForestClassifier, LogisticRegression/LinearSVC and (if you think this is a good idea) neural networks. Is this reasonable?

Thanks a lot!

I don't really have experience with such a massive dataset, but my first thought would be to explore the instances in order to see if so many are needed. I would start with an ablation experiment, trying various sizes of training data with a simple method (random forest seems a good idea) in order to observe the evolution of the performance w.r.t size of the training data. It's likely that the performance reaches a plateau at some point, and it would be useful to know this point.

It might also make sense to study if the data contains duplicates or near duplicates. You can't remove duplicates directly because the distribution matters, but it might be possible to replace them by assigning weights to instances. I'm not expert in this but there are methods for instance selection.


AUC higher than accuracy in multi-class problem I stumbled upon a 3-class classification problem where all compared classifiers yield a higher AUC than accuracy (usually around 10% higher). This happens both when the dataset is balanced or slightly imbalanced.
Now, after looking at this answer: Why is AUC higher for a classifier that is less accurate than for one that is more accurate? I understand that, for binary classification, this might happen because the accuracy is typically computed at a threshold of 0.5 whereas AUC is based on all threshold values.
But what happens with multi-class classification? Specifically, those scenarios where accuracy is defined as the frequency with which the predicted labels match the true labels (tf.keras.metrics.CategoricalAccuracy) and AUC is defined as the weighted average of the AUC for each class vs the rest (One-vs-rest) (sklearn.roc_auc_score). Why might AUC be higher there?
In other words, I'm trying to understand what this result means. Does it mean that my classifiers can discern well when each class is measured against the others (AUC), but not as well when the prediction probabilities are an output of the softmax function, and therefore spread out for the three classes?

I think the question can se splitted to three parts. The first part is about comparing accuracy and AUC of the same model, the second is about comparing models and the third is about multi-class problems.
First part - I think that accuracy and AUC are not comparable metrics, I don't think that there is special meaning to the case where one is higher than the other.
Second part - ROC plot and AUC are good for understanding model and choosing model types, but in the end of the day you usually want to evaluate your final model with specific threshold so accuracy, precision and recall are the relevant metrics.
Third part - I think the answers to the previous parts above are valid both for binary classification and multi-class classification. Why do you think it have different meaning here?


What exactly does the model generation mean in this diagram? I've been trying to grasp a research paper on image colorization using neural networks here I am stuck at this diagram. What I need help on, is the Model Generation step  after Feature extraction. What exactly do we do in this step?

A model is a simplified representation of a complex real world object/phenomena. As a simple example, I see a mountain and because I cannot study the exact shape of the mountain I represent it by a model that it is a triangle. Representing the mountain by a triangle is a model. Someone else comes and makes a better model by saying it is a cone. Every model captures some features of the real world phenomena. A model is better if it is able to capture the true features of the object/phenomena.

From a machine learning perspective, this corresponds to exploring the dataset and then thinking about the dataset and creating a model of the dataset. For instance, if I have the famous Titanic dataset, I would be thinking how different features affect the probability of survival. One "model" will be to think that different attributes independently affect the chances of a person's survival. I would be wanting to use a naive bayes classifier to learn the parameters of this model and see how it performs. A different model would be to assume that different attributes probably follow a hierarchy. Maybe, first comes gender. Then among those of the same gender, the deciding factor is probably class and so on. To learn the parameters of this model I would want to train the data with a decision tree classifier or a random forest classifier. This is precisely the process of model generation. It involves thinking about the data by combining your insights from the exploratory data analysis and the domain knowledge you have. You then make certain assumptions. And then test your model.


How to interpret the value of categorical cross entropy? Categorical cross-entropy loss is usually used in settings where the target in one-hot encoded. Suppose I have a problem where there are 300 possible outcomes, and thus my final fully connected layer will have 300 neurons and the output (after softmax) is expected to be probabilities for each class such that their sum is equal to one.
My question is: how to interpret the value of the categorical cross-entropy loss? What does it mean to have a categorical cross-entropy loss of 5 or 0.9 or 0.1?
Does the interpretation of the loss get affected with the number of possible classes?

There is usually no straightforward interpretation of what cross-entropy means in the context of the given task. In practice, it is more important to follow the trend of how the cross-entropy develops during the training.
The measure comes from the information theory. It says how surprised the model is when it has some belief about the output (a distribution) and you provide it with the ground truth output. A nice (but not very useful) interpretation is that the entropy of $x$ means the same surprisal as observing an output of rolling and $2^x$-sided dice.
The value of cross-entropy indeed depends on the number of classes, just because with more classes you have the output distribution must cover more options and it is more difficult to assign more probability to the ground truth class.


How can I feed BERT to neural machine translation? I am trying to feed the input and target sentences to an NMT model, I am trying to use BERT here, But I don't have any idea how to give it to my model.
before that, I was using one-hot encoding and I got issues there and I want to use BERT.

Also, I have to note that, I am new in TensorFlow and deep learning. So please share your opinion with me about the use of BERT in NMT.

my goal is only to use BERT in my model for translation purpose

my model definition:

def build_model(in_vocab, out_vocab, in_timesteps, out_timesteps, units):
    model = Sequential()
    model.add(Embedding(in_vocab, units, input_length=in_timesteps, mask_zero=True))
    model.add(LSTM(units))
    model.add(RepeatVector(out_timesteps))
    model.add(LSTM(units, return_sequences=True))
    model.add(Dense(out_vocab, activation='softmax'))
    return model

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model.
I got a memory error when I want to use more data like 100k for my task. and I came up that my tokenizer is a kind of problem here because it takes a lot of memory to make my tokenizer for such a huge volume of data so pre-trained models like BERT are the solution to this problem to feed more data like 200k or more to your model without worrying about memory error too much.

Also, in my task, I was worry about words that do not exist in my training phase but exist in a test phase so bert solved this problem for me too, because it was trained on a large corpus.
let's dive in and find out how I used BERT to fix my problem here.

Here I am going to make English to English translation system.

Loading pre-trained BERT for English ( if your source and the target language differs from each other, you have to load them separately you can look at tfhub.dev for them )

max_seq_length =  50 # i need to test the bert so I will keep this small for now
input_word_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="input_word_ids")
input_mask = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="input_mask")
segment_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype=tf.int32,name="segment_ids")
#this is the path to pre-trained bert model
bert_layer = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/1",trainable=True)

then I call my tokenizer

FullTokenizer = bert.bert_tokenization.FullTokenizer
vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = FullTokenizer(vocab_file, do_lower_case)

here is an examples to use the tokenizer:

tokenizer.convert_tokens_to_ids(['is'])
tokenizer.convert_ids_to_tokens([2003])
tokenizer.convert_ids_to_tokens([30521])

the output is: ([2003], ['is'], ['##～'])

then I used my train data to get my sequence

s = "This is a nice sentence."
stokens = tokenizer.tokenize(s)
stokens = ["[CLS]"] + stokens + ["[SEP]"]
input_ids = get_ids(stokens, tokenizer, max_seq_length)

the output for me was:

input Ids: [101, 2023, 2003, 1037, 3835, 6251, 1012, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

this method is also needed for getting token ids and you should add it to your codes

def get_ids(tokens, tokenizer, max_seq_length):
    """Token ids from Tokenizer vocab"""
    token_ids = tokenizer.convert_tokens_to_ids(tokens)
    input_ids = token_ids + [0] * (max_seq_length-len(token_ids))
    return input_ids

for your model vocab size you can use these values,

len(tokenizer.vocab)

which in my case is: 30522

and after these steps, you can get your sequence for input and outputs and then feed them to your model.

I hope it helps. and please share your opinion if I am wrong in this case.

thanks


What is the use of user data collection besides serving ads? Well this looks like the most suited place for this question.

Every website collect data of the user, some just for usability and personalization, but the majority like social networks track every move on the web, some free apps on your phone scan text messages, call history and so on.

All this data siphoning is just for selling your profile for advertisers?

A couple of days ago developers from one product company asked me how they can understand why new users were leaving their website. My first question to them was what these users' profiles looked like and how they were different from those who stayed.

Advertising is only top of an iceberg. User profiles (either filled by users themselves or computed from users' behaviour) hold information about:

user categories, i.e. what kind of people tend to use your website/product
paying client portraits, i.e. who is more likely to use your paid services
UX component performance, e.g. how long it takes people to find the button they need
action performance comparison, e.g. what was more efficient - lower price for a weekend or propose gifts with each buy, etc.

So it's more about improving product and making better user experience rather than selling this data to advertisers.


LSTM unit: cell state dimension When we fit an LSTM model, each LSTM has a cell state which contains the information we want. I am wondering what's the dimension in the cell state (i.e. cell state itself should be a vector, then how large is the vector?)?

(from http://colah.github.io/posts/2015-08-Understanding-LSTMs/ )

i.e. In the above architecture, how do we determine the dimension of the matrix Wc?

If I am using Keras package, how do I find the Cell state dimensions of each LSTM unit? Is it something that one can adjust?

Thank you!

In Keras, the first argument in LSTM gives the dimensionality of the cell state.


How to quantify ‘compute cost’ of training of xgboost model? I want to quantify compute cost of hyper-parameter search for xgboost model. One way can be to measure training time with one particular hyper parameter configuration chosen for training and use it as proxy for compute cost. Can we quantify compute cost based on hyper parameters of this model depending upon value of hyper parameters chosen e.g., analytical expression based on max depth, num of estimators, min child weight, gamma etc or can you suggest some other way to quantify this compute cost?
I want to measure for each particular hyper parameters chosen for training on same set of data what will be the model performance and compute cost!

If you have the time on hand, you could simply measure the time taken for all combinations of hyper parameter values in a Grid Search, preferably with repetition. It's unlikely that any theoretical analytical expression will provide adequate accuracy for predicting the compute cost, as there as so many factors that contribute noise to the compute time.
You could even build a regression model to predict the computing cost of new hyper parameter combinations if you wanted to.


Why don't we use Manhattan distance instead of euclidean distance in linear regression? When I am explaining concept of linear regression to one of my peers, I got stuck in answer this question. Why don`t we use Manhattan distance instead of euclidean distance in linear regression? Can anyone give intuition behind this?

The main reason for that may be typically use Euclidean metric; Manhattan may be appropriate if different dimensions are not comparable.. While using regression-based methods you may have noticed that you usually have features of real values. You usually normalise the features and feed them to your model. The act of normalising features somehow means your features are comparable. In cases where you have categorical features, you may want to use decision trees, but I've never seen people have interest in Manhattan distance but based on answers [2, 3] there are some use cases for Manhattan too. You can also consider that they are comparable.