Getting same result for all predictions in cnn This is my first time training a model in cnn and predicting results but I am getting same value for images I have input. Here is my code

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.optimizers import Adam
from keras.layers.normalization import BatchNormalization
from keras.utils import np_utils
from keras.layers import Conv2D, MaxPooling2D, ZeroPadding2D,
GlobalAveragePooling2D
from keras.layers.advanced_activations import LeakyReLU
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Convolution2D
from keras.layers import MaxPooling2D
from keras.layers import Flatten
from keras.layers import Dense
classifier=Sequential()
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train/=255
X_test/=255
number_of_classes = 10
Y_train = np_utils.to_categorical(y_train, number_of_classes)
Y_test = np_utils.to_categorical(y_test, number_of_classes)
classifier.add(Convolution2D(32,3,3,input_shape=
(28,28,1),activation='relu'))
classifier.add(BatchNormalization(axis=-1))
classifier.add(MaxPooling2D(pool_size=(2,2)))

classifier.add(Convolution2D(32,3,3,activation='relu'))
classifier.add(BatchNormalization(axis=-1))
classifier.add(MaxPooling2D(pool_size=(2,2)))

classifier.add(Flatten())

classifier.add(Dense(output_dim=256,activation='relu'))
classifier.add(BatchNormalization())
classifier.add(Dense(output_dim=10,activation='softmax'))

classifier.compile(optimizer='adam',loss='categorical_crossentropy',metrics=
['accuracy'])
gen = ImageDataGenerator(rotation_range=8, width_shift_range=0.08,
shear_range=0.3, height_shift_range=0.08, zoom_range=0.08)

test_gen = ImageDataGenerator()
train_generator = gen.flow(X_train, Y_train, batch_size=64)
test_generator = test_gen.flow(X_test, Y_test, batch_size=64)
classifier.fit_generator(train_generator, steps_per_epoch=60000, epochs=1,
                validation_data=test_generator, validation_steps=10000)
import cv2
image = cv2.imread("pitrain.png")
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) # grayscale
ret,thresh = cv2.threshold(gray,150,255,cv2.THRESH_BINARY_INV)
#threshold
kernel = cv2.getStructuringElement(cv2.MORPH_CROSS,(3,3))
dilated = cv2.dilate(thresh,kernel,iterations = 13) # dilate
im2,contours, hierarchy =
cv2.findContours(dilated,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_NONE)
# get contours
# for each contour found, draw a rectangle around it on original image
for contour in contours:

# get rectangle bounding contour

[x,y,w,h] = cv2.boundingRect(contour)
# discard areas that are too large

if h>300 and w>300:

    continue
# discard areas that are too small

if h<40 or w<40:

    continue
# draw rectangle around contour on original image

cv2.rectangle(image,(x,y),(x+w,y+h),(255,0,255),2)
image.shape
image=image[:,:,:1]
newimg = cv2.resize(image,(28,28))
img.shape
img = np.reshape(newimg,[1,28,28,1])
cv2.imshow("screen",img)
classifier.predict(img)

The output I am getting is array of zeros with 1 at third position.
This is where I copied the contour part from https://www.quora.com/How-do-I-extract-a-particular-object-from-images-using-OpenCV

Epoch is equal to 1 because I only wanted to test my model and still I got accuracy of above 99%

Ok so I changed your model to simplify the training for example's sake. I will go through the example in detail below. When you feed a value from a different distribution to your model you are always at risk of misclassification. For example, your model was trained using handwritten numbers, thus it is not surprising for the model to missclassify numbers that are typewritten. Of course a deeper more complex model could do this.

Getting the data

The MNIST data is of size 28 by 28. We will convert these values to floating point values and then normalize them to the range 0 to 1. We will also determine that we have 10 output classes.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.

# The known number of output classes.
num_classes = 10
# Input image dimensions
img_rows, img_cols = 28, 28

Next we need to reshape the data such that it matches with the Tensorflow framework which we will be using under the hood of Keras. This requires that the instances be the first dimension and it also requires a channels dimension as the last one. thus for the MNIST data we need to have $(6000, 28, 28, 1)$.

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

Then we will bin the outputs into one-hot-encoded vectors

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

Let's build our simple model, you can add more layers to this to make it more robust

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Let's train our model

epochs = 4
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

This should yield about 98.75% on the validation set. This is pretty good and should be enough to test some new data using this set.

Validating results using MNIST

Let's pass through a few random values from the MNIST dataset to see if they get rightfully classified by our model. Note how I set the dimensionality of the data. The instance dimension must exist even if we only have a single instance to predict as well as the channel dimension

ix = 0

plt.imshow(X_test[ix,:,:,0], 'gray')
plt.show()

temp = np.zeros((1,28,28,1))
temp[0,:,:,0] = X_test[ix,:,:,0]

model.predict_classes(temp)

  array([7], dtype=int64)

Ok so that worked! Let's try another one ix = 100!

  array([6], dtype=int64)

That worked as well!

Validating the model on novel data

The image you sue has to match stylistically with the ones in the MNIST data set. Ideally they should be of the same distribution. The image you provided luckily gets correctly classified but it should be noted that this may not be the case for other numbers that are typewritten. If you want your algorithm to detect these then you should really train with typewritten numbers as well as handwritten numbers.

from skimage.transform import resize

def rgb2gray(rgb):
    return np.dot(rgb[...,:3], [0.299, 0.587, 0.114])

im = plt.imread("7 type.jpg")
im = rgb2gray(im)
im = resize(im, (28, 28))

plt.imshow(im[:,:], 'gray')
plt.show()

temp = np.zeros((1,28,28,1))
temp[0,:,:,0] = im

model.predict_classes(temp)

  array([7], dtype=int64)

Validating on hand drawn numbers

More realistically would be to use hand drawn numbers since that is what we used to train. I just drew this one using Paint.

  array([8], dtype=int64)

  array(2, dtype=int64)


context-based model and interaction terms I have implemented a prediction model.
Now I am checking if I should split the model to two context-based models (men and women).
I have created a new binary feature that is relevant to only part of the women's group and it can affect the prediction accuracy for this group.
I don’t have a lot of data so I am not eager to split to context-based models.
I know that in this cases it is best to use an interaction terms features (womenxnewfeature) but this feature is only part of the women's’ group and I have heard that adding the feature as it is it’s not good.

Any suggestion on what I can do?

If you have gender data, use it. You're right that it will probably be a binary dummy feature (i.e., male = 0 or 1, and so on). You can use gender directly as a predictive feature, and/or interact it with other variables. You don't need to split your dataset by gender. Adding arbitrary information increases the noise in your set and decreases model quality. But you can learn whether gender is arbitrary or not! Add gender and compare to a model without gender to see if there's any change.


Multi-label classification model in python? Assume you have the following artificial dataset

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD'))
df['sex']  = [np.random.choice(['male', 'female']) for x in range(len(df))]
df['weight'] = [np.random.choice(['underweight',
        'normal', 'overweight', 'obese']) for x in range(len(df)) ]

This produce the following artificial dataset

df.head()
Out[50]:
          A         B         C         D     sex       weight
0  0.955136  0.802256  0.317182 -0.708615  female       normal
1  0.463615 -0.860053 -0.136408 -0.892888    male        obese
2 -0.855532 -0.181905 -1.175605  1.396793  female   overweight
3 -1.236216 -1.329982  0.531241  2.064822    male  underweight
4 -0.970420 -0.481791 -0.995313  0.672131    male        obese

I am trying to know both the sex and the weigh based on the value of the features A,B,C,D. I learned that this a multi-label classification problem and there is a nice python library that should help (e.g. scikit-multilearn ). However I do not know how this is achieved. Can someone show me how I could train a model and test its accuracy on this artificial dataset? Specifically:
1. Assuming that A,B,C,D are the feature and sex and weight are the label, how to create a test and training sets? The following will not work

X=df[list('ABCD')]
y=[['sex','weight']]
X_train, X_test, y_train, y_test = train_test_split(X, y)

How to you train and test the model?

First of all you would need to encode your target columns.We can use sklearn.preprocessing.MultiLabelBinarizer here:

from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()

X=df[list('ABCD')]
Y=pd.DataFrame(mlb.fit_transform(df[['sex','weight']].values), columns=mlb.classes_, index=df.index)

yields:

In [75]: Y
Out[75]:
    female  male  normal  obese  overweight  underweight
0        1     0       0      1           0            0
1        1     0       0      0           0            1
2        1     0       1      0           0            0
3        1     0       1      0           0            0
4        1     0       0      0           0            1
5        1     0       1      0           0            0
6        0     1       0      0           0            1
7        1     0       1      0           0            0
8        0     1       0      0           0            1
9        0     1       0      0           0            1
..     ...   ...     ...    ...         ...          ...
90       1     0       0      1           0            0
91       0     1       0      0           0            1
92       1     0       0      0           0            1
93       1     0       0      0           1            0
94       1     0       1      0           0            0
95       1     0       0      0           0            1
96       0     1       0      0           0            1
97       0     1       0      0           1            0
98       1     0       0      0           1            0
99       1     0       1      0           0            0

[100 rows x 6 columns]

now we can use one of the classifiers that support multi-label classification (see Support multilabel:)

Example:

from sklearn.neighbors import KNeighborsClassifier

knc = KNeighborsClassifier()

X_train, X_test, Y_train, Y_test = train_test_split(X, Y)

knc.fit(X_train, Y_train)
Y_pred = knc.predict(X_test)


How to evaluate the K-Modes Clusters? K-modes algorithm is available here

I want to do clustering of my binary dataset. I need to specify the number of clusters that I need as an output:

KModes (n_clusters, init, n_init, verbose)

My dataset contains 1000 lines and 1000 rows, I want to calculate the distance between my clusters in order to know the exact number of cluster that I need to choose. I don't know how to compare between them. Probably I want to use the hamming distance because it is the most suitable distance to compare between binary data.

Any recommendations that you can give are welcomed.

The are some techniques to choose the number of clusters K.

The most common ones are The Elbow Method and The Silhouette Method.

Elbow Method

In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion.

Then, you plot them and where the function creates "an elbow" you choose the value for K.

Silhouette Method

This method measure the distance from points in one cluster to the other clusters. Then visually you have silhouette plots that let you choose K.

Observe:

K=2, silhouette of similar heights but with different sizes. So, potential candidate.

K=3, silhouettes of different heights. So, bad candidate.

K=4, silhouette of similar heights and sizes. Best candidate.


How does the validation set get used in the training phase? I am confused about how the validation set is used during the training phase (neural network like CNN)?
In a platform like Matlab or python(Keras), I split my dataset into train set, validation set and test set.
I knew that validation set is used to tune hyperparameters(like the number of neurons and learning rate), suppose SDG optimizer is used, how the tuning hyperparameters happened based on validation set?
Does the validation set just give an indicator of how the neural network performs on unseen data(validation set) then based on that I manually set hyperparameters? or something automatically(the optimizer) tune hyperparameters?

The validation set is used just to give an indication about the network's performance. The hyperparameters are tuned based on the training data itself.
Based on the optimizer, the neural network calculates the loss for a training sample. Based on the computed loss, the weights of the network are updated. This is repeated for every sample in the training set.
The validation set just uses the latest weights of the network on unseen data to check the model's performance.


Anomaly detection on sparse categorical data I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ones) and being able in the future to identify new strange combinations of future clients immediately.

clientid
choice

cl1
a

cl2
b

cl2
c

cl3
d

cl4
b

cl4
c

If I transpose the table by clientID I have a row for each client and different columns based on the choices, it will became a sparse dataset with categorical variables (choices). Some clients have only one choice and some have multiple ones and I want to find outlier records (clientid)
Which type of algorithm could help me in this type of problem?
It is unsupervised, so I dont know what are the normal combinations and it is sparse data on categorical variables.

No need for machine learning here.
After you've transposed the dataframe, just count the number of unique combinations in the new column, and then rank them by frequency. Set a suitable threshold of "rareness" (like freq=2 below) and you will have your list of strange combinations.
There's a tool in Pandas for this called df.values_count()
e.g.

combination
freq

a,b
1

a,c
1

a,d
1

a,b,c
2

a,b,c,d
10

b,d
10

Then just compare you new combinations with your "bank of rare combinations", and update them if they are no longer rare.


Dueling DQN - can't understand its mechanism I am trying to understand the purpose of Dueling DQN.
According to this blogpost:

our reinforcement learning agent may not need to care about both value and advantage at any given time - this seems to be what I can't understand.

Let's assume we are in state $S_t$ and we select an action which has the highest score. This is the promised total reward that we will get in the future if we take the action.

Notice, we don't yet know the V or A of the future state $S_{t+1}$ (where we will end up after taking this best action), so decoupling V and A in any state, including S_{t+1} seems unnecessary. Additionally, once we do get to work with them we still seem to recombine them during $S_{t+1}$ into a single Q-value, just as was noted in the blog post.

So, to complete my thought: V and A seem to be a "hidden intermediate step", that still gets combined into Q, so we never know it's even there. Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q?

A slightly unrenated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Can someone provide a different example to the sunset?

Edit after answer was accepted:

Found a friendly explanation about this architecture here.

Also, if someone struggles to understand what V, Q and A is, read this answer, and my comment under it.

A slightly unrelated thought, 'V' is the score of the current state only. 'A' is the total future expected Advantage, for a particular action, right?

Not quite. $V$ is the total (discounted) future expected reward, assuming starting in state $s$, following current policy (in a control problem, usually best guess so far at the optimal policy) into the future. That includes selecting current action, $a$, according to the policy being assessed. The advantage function $A(s,a)$ (blog post has the arguments wrong for $A$) is the difference in value between selecting $a$ according to the current policy and selecting a specific action $a$. The value of $A$ also sums all future rewards assuming that the current policy is then followed into the future after this, maybe different, selection. Note that when the policy is the optimal one, and $V(s)$ is accurate, then $A(s,a)$ should always be zero or negative; optimal actions score zero, non-optimal ones will be negative.

  Even if network somehow benefits from one or the other, how does it help if both streams still end up as Q?
  ... Can someone provide a different example to the sunset?

The more prosaic explanation is that the function decomposition is always technically correct (for an MDP). Coding the network like this incorporates known structure of the problem into the network, which otherwise it may have to spend resources on learning. So it's a way of injecting the designer's knowledge of reinforcement learning problems into the architecture of the network.

Conceptually this is similar to designing CNNs for computer vision with local receptive fields because we know edges and textures can be detected this way in images. Although CNNs have more than just that benefit, one of the positive aspects of the design for vision tasks, is that they structurally match known traits of the problem being solved.

Value-based RL control methods (as opposed to policy-gradient methods) work due to "generalised policy iteration", where the agent is constantly assessing the current values of a policy, then using those value estimates in order to make improvements. The split between $V$ and $A$ functions fits very well with that conceptually. The $V$ function is generally being adjusted to assess the current policy as accurately as possible, whilst positive values in $A$ function identify likely changes to the policy.


Changing colors on Orange Scatterplot widget? This is probably has an obvious solution, but I cannot seem to figure it out.
I have this scatterplot below:

I know that there is a drop down menu on the left that allows you to decide how to color the data. However, I cannot seem to find a way to change which color the data points appear as. As you can see from the image above, the colors are not exactly the best for visually displaying the this data.

I just realised I'd missed the 'colour' widget. By connecting the data table to the 'colour' widget, I could set what colour I wanted different data sets to be. Then I could attach the colour widget to my scatter plot, and BAM, the colours I wanted. I needed to make sure the scatter plot widget was connected to both the colour widget and the data table. There may be a more streamlined way to do it, but that worked.

(Posting my own answer rather than deleting, in case anyone else has similar issues).


Classification method when idea conditions are known? Dataset: Concrete measured on 8 sets of properties. ~4000 data points.

Known: under ideal condition, value of 8 properties for 10 different types of concrete.

The objective is to find: in 8 dimension space, what is the 'type of concrete' to which the given data point is nearest to.

I think image well explains the question if my words are confusing. Black = idea condition. Red =  points who's category need to be identified.

Clarification:

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=8)
knn.fit(X_train, y_train)
pred = knn.predict(X_test)

If i understand correctly, it would be acceptable to: X_train will be array of 8 by 10 and Y_train will be array of 1 by 10. Is this correct.

Welcome to the community Martan!

If I understood your question well, you have a set of patterns (you call them idea condition) and a set of query points (samples) and you want to determine (predict) their labels according to their affinity/similarity/closeness to the pattern.

If it's right, then K-Nearest Neighbor algorithm is what you are looking for. Please note that in high dimensions Euclidean distance is distorted however 8 dimensions is fine.

Hope it helps and in case I did not understand well, please comment here so I can update my answer. Good Luck!

Update

What you mentioned in your comment about training in KNN is right. Let me clarify the thing.

Classification as A Supervised Learning Process: This means that you already had some data and their classes. So you can partition your space according to all those labels. Image below shows partition of different classes (i am not the best painter specially in MS paint :D). Having this, a new point falls into one of these partitions so you can determine its label. Building such a partitioning is done in the training process as you said.

Nearest Neighbor Search: But isn't this method pretty natural? You really do not need to know Machine Learning to perform such an algorithm and indeed you use KNN in daily life (today you see your friend with some of his colleagues and a new person you dont know and you instantly think he is probably a colleague. The other day you see your friend with his family and a guy you dont know and you guess he is most probably a relative. That is conceptually KNN!). In your example you dont need to learn how to partition the space as you already have your predefined labels with fixed position (we do training to find this out. you already have it, so just go on!). Now you can just make a nearest neighbor search and say to which class a point belongs.


What algorithmic approach for selecting similar, relevancy based documents I have an application that tracks people making mentions of various topics. We've used a Bayes algorithm to do some simple classification (users give a thumbs up/thumbs down) to pick the people that they believe are the best fit for their project.

Our intention was to use this data to help us sort and order the influencers based on "fit" to the customer's needs.

However, we've got some trained data, and now all we can do is say are they similar to the "thumbs up" group, or the "thumbs down" group.

What algorithm should we have used for this instead?

Basically, the ideal is to have a score.. and the biggest, smallest based on the trained data is the one that gets shown first.

Thoughts?

Even better if its in Ruby.

Check out term frequency-inverse document frequency (TF-IDF), if you're parsing large chunks of text for mentioned topics. The metric measures importance of topics bounded by how often they appear across a corpus of documents, which helps to weed out topics that may be very common, but are very common to all discussions, and therefore not very useful.

It's a simple calculation, but there is a ruby gem that calculates it directly: tf-idf-similarity.

Hope this helps.


understanding batch normalization In the paper Batch Normalization: Accelerating Deep Network Training b
y Reducing Internal Covariate Shift (here) Before explaining the process of batch normalization the paper tries to explain the issues related with (I am not getting what the exact issue addressed here is).

excerpt from section 2, para 2:

  We could consider whitening activations at every training
  step or at some interval, either by modifying the
  network directly or by changing the parameters of the
  optimization algorithm to depend on the network activation
  values (Wiesler et al., 2014; Raiko et al., 2012;
  Povey et al., 2014; Desjardins & Kavukcuoglu). However,
  if these modifications are interspersed with the optimization
  steps, then the gradient descent step may attempt
  to update the parameters in a way that requires
  the normalization to be updated, which reduces the effect
  of the gradient step. For example, consider a layer
  with the input u that adds the learned bias $b$, and normalizes
  the result by subtracting the mean of the activation
  computed over the training data: $\hat x= x − E[x]$ where
  $x = u + b, X = {x_{1...N}}$ is the set of values of $x$ over
  the training set, and $E[x] = \frac 1 N(\sum_{i=1}^nx_i)$
  .

  If a gradient
  descent step ignores the dependence of E[x] on b, then it
  will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then
  $$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$.

  Thus, the combination of the update to b and subsequent
  change in normalization led to no change in the output
  of the layer nor, consequently, the loss. As the training
  continues, b will grow indefinitely while the loss remains
  fixed. This problem can get worse if the normalization not
  only centers but also scales the activations.

here is my understanding of the literature:

We have a batch of size N (One training batch)
Let there be two arbitrary hidden layer connected to each other (L1 and L2) connected by parameters $W$ and $b$
output coming out of L1 is x1
$u = x1W$ (this is where the literature above starts. dimension of u is MxN) (M is the number of units in L2)
$x = u+b$ (dimension b = dimension x = dimension u = MxN)
Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x= x − E[x]$)
We compute the loss and backpropogate the gradient and update just this layer's $b$ in order to give it a sanity test. New $b$ = $b + \Delta b$
We run it again on the same batch with updated $b$
repeat 3 and 4
$x_{new} = u+b + \Delta b$ (dimension b, $\Delta b$ = dimension x = dimension u = MxN)
Now before feeding x into L2 we centre it by subtracting the mean of $x$ from each entry in $x$ ($\hat x = x + \Delta b − E[x + \Delta b] = x - E[x]$).
which is the same as what was calculated before updating b and hence updating b had to effect on the training

My question is with this part of the excerpt:

"If a gradient
descent step ignores the dependence of E[x] on b, then it
will update $b ← b + ∆b$, where $∆b ∝ −\partial l/\partial\hat x$. Then
$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$."

Why is

"
$$u + (b + ∆b) − E[u + (b + ∆b)] = u + b − E[u + b] \tag 1$$." dependent upon what comes before it? What is even the point of that bit? Please also note the useage of word "Then" (made bold) implying the statement necessarily draws causality from what comes before

Let's assume that you are trying to minimize the following loss for a given task, $$\ell(y, \hat{y}) = \frac{1}{2}\| y - \hat{y}\|^2, $$ where $\|
\cdot\|$ is the Eucledian distance, and the predicted output is $\hat{y} =\hat{x}$ for simplicity. The gradients then can be computed as follow:

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial b}, \hspace{20pt} \Delta \omega = - \frac{\partial\ell}{\partial\hat{x}} \cdot \frac{\partial\hat{x}}{\partial \omega}$$

Now, the gradient of $\hat{x}$ with respect to the bias $b$ is $$\frac{\partial\hat{x}}{\partial b} = \frac{\partial}{\partial b}(x-E[x]) = \frac{\partial}{\partial b}\left((u+b)-E[(u+b)]\right) = 1 - \frac{\partial}{\partial b}E[(u+b)]$$

Ignoring the fact $E(x)$ is dependant on $b$ makes the above gradient equals to 1, and hence keeps updating the bias as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[x] = 1 -0
 = 1$$
and then

$$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (1), \hspace{20pt} b \leftarrow b + \Delta b $$

Otherwise, if you wanna consider this dependency, the gradient becomes 0, and hence no update as follow:

$$\frac{\partial\hat{x}}{\partial b} = 1 -  \frac{\partial}{\partial b}E[u+b] =  1 -  \left(\frac{\partial}{\partial b}E[u]+\frac{\partial}{\partial b}E[b]\right)
 = 1 - (0+1) = 0$$
and then
 $$\Delta b = - \frac{\partial\ell}{\partial\hat{x}} \cdot (0), \hspace{20pt} b \leftarrow b + 0 $$

In both cases, regardless bias update, the loss function will remain fixed,

$$u+(b+\Delta b)−E[u+(b+∆b)]=u+b−E[u+b], $$

however, in the first case, the bias will grow indefinitely.


Train new data to pre-trained model Let's say I've trained my model and made my predictions.

My question is... How can I append some new data to my pre-trained model without retrain the model from the beginning.

It all depends on the specific algorithm you're using. Some of them support incremental learning, while others don't.

For example, in the case of scikit-learn, using fit() more than once on the same model will simply overwrite the model's weights each time (see here for more details).

What you can do however, is look for algorithms that implement the partial_fit api, and use this to retrain your existing models - see the documentation for a list of algorithms that support incremental learning and thus implement this api.

An alternative solution is to look for algorithms that support the warm_start parameter, e.g. LogisticRegression. Note that warm_start might also be influenced by other parameters, so you need to pay attention to their values, too - e.g. in the case of LogisticRegression, warm_start won't work if you use the liblinear solver (which is the default).


How do you find the standard deviation of a dataset over a certain period? I'm new to R.
I could get the std. dev. of the whole data set,
data <- read.csv("Berkshire.csv")
sd1 <-sd(data$BRKRet, na.rm = TRUE) #0.0675
mean<- mean(data$BRKRet, na.rm = TRUE) #0.0189

but I'm not sure how to find the std. dev. of a dataset over a certain time period. For example, from Nov. 17, 2010 ~ Nov.16, 2020.
Thanks for the help.

I think you can only do that if your dataset contains a column encoding the date (see Dates and related info) that you can filter by.
See below a simulated dataset of such type.
df <- data.frame("x" = rnorm(4000, mean = 0, sd = 1+(1:4000)/1000), "d" = seq.Date(from = as.Date("2010-01-01"), by = "day", length.out = 4000))

Then it should be pretty standard:
# sd of x over the whole dataset
sd(df$x)
#[1] 3.121845

# define a logical vector identifying the dates of interest
interval <- (df$d >= as.Date("2010-11-17")) & (df$d <= as.Date("2020-11-16"))

# calculate the sd over the dates of interest only
sd(df[interval,]$x)
#[1] 3.227157

# display it as a plot
plot(x ~ d, df, col = ifelse(interval,2,1))

Only, make sure you check what your date format is, as that might depend on your system. Like:
Sys.Date()
[1] "2021-03-21"

This shows that my system expresses the date as "year-month-day". That's the format I used for the filtering in as.Date conversions.
[Note: there might be clever ways to use other formats].


Get feature importance for each observation with XGBoost I have trained an XGBoost binary classifier and I would like to extract features importance for each observation I give to the model (I already have global features importance).

More specifically, I am looking for a way to determine, for each instance given to the model, which features have the most impact and make the input belong to one class or another. I would like to know something like the top 5 features which make the observation belong to some class and indications on how I should modify these 5 features so that the probability of belonging to this class decreases or increases.

For example, let’s say my model predicts whether a house costs more than 100,000 dollars (this is the positive class) based on its location, surface and number of bedrooms. I give it the following input: London, 400 square foots, 4 bedrooms and my model predicts a probability of 56% for the house to be in the positive class. I am looking for a Python module or a function that would show the most influential features for each observation.

I would suggest you're probably looking for something like one of these two packages:

https://github.com/slundberg/shap
https://github.com/marcotcr/lime

See this notebook of lime's for example, which shows how you can use it to see why a specific sample in your data resulted in the prediction from the model:

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html


Similarity coloring of a self organizing map I have implemented the algorithm to train self organizing maps in Python and it seems to be working well. I checked with some labeled data and the maps are learning the topology well. Here are some images:

I am calculating a Unified Distance Matrix for visualization which does not seem to be working well as seen in the 2nd image.

I believe that similarity coloring as seen in the world poverty map developed by Prof. Kaski and Prof. Kohonen would be apt for my application but I am unable to find a clear explanation of how it is to be calculated. The procedure has been outlined in the paper "Tips for Processing and Color-coding of Self-Organizing Maps ", however, it omits some technical details which are necessary to implement the similarity coloring. Here is the image that I am referring to:

Could someone help me with the exact steps to calculate the similarity coloring map? I have searched everywhere for a month and I am unable to find an explanation.

I don't know exactly how the above was created, but I did something similar in the late '90s and reproduced it recently using somoclu in Python. While I've not applied the colours back to the SOM cells, this would be pretty trivial.

I've put the notebook on kyso, I hope it's useful.


Can I use model which is trained using Keras to Scala? I train the NLP classification model using Keras by Python.
But to deploy the trained model to a platform I need to use Scala.

Can I use Scala with Keras model what I trained using Python?

Thanks

Yep, check this blog: https://towardsdatascience.com/deploying-keras-deep-learning-models-with-java-62d80464f34a

Dump your model as .h5 like you normally would and load it in java using those libs from blog.

Another idea would be to create separate AI microservice in python(django/flask). That's what I usually do.


What is sensitivity in confusion matrix? A confusion matrix is a table that is often used to describe the performance of a classification model (or "classifier") on a set of test data for which the true values are known. The confusion matrix itself is relatively simple to understand, but the related terminology can be confusing.

What is sensitivity in confusion matrix?

Sensitivity or true recall rate:

Sensitivity is calculated as the number of correct positive predictions (TP) divided by the total number of positives (P)


Tips & Tricks on training DCGAN on small dataset I have made a DCGAN which I am trying to train on custom dataset of only 1200 images. I have tried to gather more, but even gathering these 1200 was hard enough. If you are wondering I used Google Chromes extension "Fakun Batch Download Image" to gather my dataset.

TRAINING DETAILS:
In training procedure I am simultaneously updating parameters of both, Generator, and Discriminator network. I've read that it works much better then training only one player ( Discriminator ) for K steps and then other ( Generator ) for one.

QUESTION: Should I perform maybe some kind of transformation on all of those images and then merge transformed images with the initial ones, or something similar?

Extending a small dataset comprised of images

A deep learning algorithm will learn a mapping function from your input space to your outputs. The variations in your input images will be learned within this function. Thus, you will want to consider this fact when you augment your dataset. The distribution of your input features should be concise as to what you plan to model.

You are thus left balancing between adding data and adding variability to your input space. For example, it might not be worth it to rotate by 180 degrees when trying to generate Dragon Ball characters. Their heads should not be where their feet are. You would want your network to understand that the bottom of the image should contain some strange space boots, and the top exotic haircuts or bald heads.

Augmenting images

Here are some useful transformations that you can use to get more data

Apply transformations (rotations, translations)
Mirror the image
Add distortions
Change zoom factor
Add blurring thus to better generalize the input data
Invert colors (skew, add brightness, etc.)

In Keras

In Keras you can use the ImageDataGenerator functions.

import numpy as np
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
%matplotlib inline

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')

# set up your data generator
datagen = ImageDataGenerator(
   featurewise_center=True,
   featurewise_std_normalization=True,
   rotation_range=60,
   width_shift_range=0.2,
   height_shift_range=0.2,
   shear_range=0.2,
   zoom_range=0.2,
   horizontal_flip=True,
   vertical_flip = True)

# Fit the generator using your data
datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))

# let's look at some generated images
image = X_train[5]

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')
for j in range(15):
   augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()


What is the problem with: the condition has length > 1 and only the first element will be used? newbie to r. i have:

> > age <- function(Age) {
> +     answer<-NULL
> +     if(Age<15) answer<-"under 15"
> +     else if(15<=Age&&Age<=50) answer<-"15 to 50"
> +     else answer<-"over 50"
> +     answer
> + }
> > titanic_3 <- titanic %>%
> +     select(Survived, Pclass, Age, Sex) %>%
> +     filter(!is.na(Age)) %>%
> +     mutate(agecat=age(Age)) Warning message: In if (Age < 15) answer <- "under 15" else if (15 <= Age && Age <=  :   the condition has
> length > 1 and only the first element will be used
> >

if i use one & instead of &&, i get two warnings. i know that one of these is because of the shortcut thing.

how do i write this function so as to get rid of the warning message?

thanks

edit: i tried:

age <- function(Age) {
    answer<-NULL
    if(Age<15) answer<-"under 15"
    else if(Age<=50) answer<-"15 to 50"
    else answer<-"over 50"
    answer
}

and got even more warnings:

Warning messages:
1: In if (Age < 15) answer <- "under 15" else if (Age <= 50) answer <- "15 to 50" else answer <- "over 50" :
  the condition has length > 1 and only the first element will be used
2: In if (Age <= 50) answer <- "15 to 50" else answer <- "over 50" :
  the condition has length > 1 and only the first element will be used

this one seems to work, but they want us to use cut.

age2 <- function(Age) {
    ifelse(Age<15,"under 15",ifelse(Age<=50,"15 to 50","over 50"))
}

The reason you receive a warning is that Age is a vector, but you are comparing it with a scalar. R automatically takes the first element of Age. You can use "ifelse" and convert the class if needed.

If you want to limit yourself to base R, you can use the following:

age <- function(Age){
    sapply(Age, function(x) if(x < 15) "under 15" else if (x <= 50) "15 to 50" else "over 50")
}


Does it make sense to use KNN to target ticket buyers for arts events? I work for a medium-sized non-profit. We have a database with approximately 40,000 profiles of folks (mostly patrons). For each show a patron attends, we add an attribute that indicated that they're into that kind of show. For example, if they saw a folk music show, they would get a Folk Music tag.

Does it make sense to use some form of the K-nearest neighbors algorithm to figure out who we could target to purchase tickets? Maybe along with some location data?

For instance, we have about a 100 seat theater, so if we have sold out about half a show, would it be effective to use KNN to try to find other patrons similar to those 50, or is that too small a sample-size?

This is a classification problem that tries to answer the question: who is likely to buy a ticket to a particular type of show. If you have information about "the previous shows" that each patron attended, you can absolutely use KNN to find which patrons have similar aggregate viewership to a particular type of show.

This might not work if you just have demographic information because there will be a lot of people that share the same demographics. However, as you mentioned, you can use the lat-long or zip code level information to get US census demographic data.

Evaluating the accuracy of this model is a little bit tricky, since you don't know who didn't attend a show due to external factors, but maybe you just want a quick and fast heuristic (which in my opinion sometimes works best).


How to chose training sample for neural network in fraud detection? I am working on a personal project sponsored by a data scientists, he offered some data sets to me.

And I have built a neural network system based on my own theory.

But when I tried to train it, I always can't get a ideal result. I think carefully chose the training data set samples may help.

So I would like to know if there are some principles or theories things which can guide researchers choosing samples.

Any books, papers or blogs would help.

by the way, I am working on a fraud detection project.

UPDATE

Ok, I think there are some misunderstandings in the question.

What I am looking for is a systematical theory which explains how to organize the training set.

For a naive example:

I have example data set from US. There are tons of data, so it is not possible to train the machine with all the example set. Here comes the question, how should I pick carefully the training data set from the example data set? how to do that is a reasonable way? and a logical way?

For a further example:

Choose the training set according to states evenly? or choose training set according to their population as weight?

Thanks, and I will provide more information if it is needed.

I think you are facing an unbalanced dataset problem (fraud), and here are some Q&A associated with it.

Poor performance shown on Rare event modeling

https://stats.stackexchange.com/questions/134599/how-to-classify-a-unbalanced-dataset-by-convolutional-neural-networks-cnn


k-means and LDA for text classification: how to test accuracy? I have many tweets that I would like classify based on their similarity.
Unfortunately I am not quite familiar with text-classification and nlp, so I had to read a lot of documents before having an idea on the topic.
My tweets have no labels so I cannot classify them: only manually, but it would be time consuming.
I would like to group them by topic, so i have first considered LDA for topic classification, then k-means clustering.
Is it a good approach to proceed?
What are the differences and how I can test the accuracy of the classification?

First of all, you use two terms Clustering and Classification interchangably and I would like to draw your attention to this. Your problem is purely Clustering.
Secondly, you asked for testing accuracy. As your problem is pure Clustering, there is no evaluation for that.
The last but not least is the problem of "Short Text Understanding". In short texts, LDA of TF-IDF based approaches (like LSA) do not work well as they rely on co-occurrences of words in texts.
Considering these two facts let's discuss the solution. I would recommend that you use a pre-trained model (I recommend S-BERT which is implemented in Sentence-Transformer Python package). Simply follow this semantic similarity search piece of code and you can implement it easily in a few lines.
Probably fine-tuning the model will be tricky according to the nature of your task, so just use a pre-trained model and see how it works.
For evaluation, I recommend to capture some similar tweets manually (more the better) and check the performance on them.
For Topic Modeling approach you can use the pytho implementation of the paper I mentioned above here.
Two above mentioned can also be combined creatively (search similarity with S-BERT and compare to bi-term topic model for example)
Hope it helped.
Good Luck!


How can I get a forecasting model to improve its forecasts over time instead of fitting to training data? Say I have a model that I'm using to forecast demand for some product.
I can train it on some training data in order to get good predictions.

But, the predictions it outputs will only really match past demand. This seems to be a problem related to the data that we give the model because if the past demand data is only how much we sold, but that was our supply as well, then this wouldn't be able to help improve future predictions.

For example, if I had in supply 10 units of product x, and I always sell 10 units of that product, then based on this training data, my model will always say that I should always order 10 units for every day. But, really, it might be better to order 25 units because if I'm always selling everything, then there might be more demand than supply. The problem is that I don't know what the true demand is.

So my question is, how would I get my model to account for that and actually improve predictions rather than trying to fit to historical data.

Like you stated, you get good predictions based on the information in the model.

The model conflates demand and supply in the case of under-supply, since it has no information about unfulfilled demand: how many products were requested but not sold due to lack of supply. Instead it tracks $sold = min(demand,supply)$.

Extend the Model

The obvious solution is to have a separate feature for the true demand rather than the conflating it with unit sold. However, this data may not be available.

Learn from Correlations

Another option is to use richer model: if you have multiple products where most of them don't sell out (i.e. where $sold$ reflects the true demand) and their sales are correlated, then it can predict that a particular product needs to be resupplied at a higher rate than the observed sales. This prediction would be based on the correlations with products that don't hit the $min(demand,supply)$-cap and .

Reinforcement Learning

Finally, another approach is to look to Reinforcement Learning where the agent learns the best resupplying strategy by experimenting.

In a simple setup, your agent will select between two actions: one that orders according to its prediction in the normal case and another action that orders more than predicted. In the case where the prediction matches actual demand the probability of the over-ordering action will be quite low, but it will still be selected with some small probability $\epsilon$.

Reinforcement Learning allows the agent to learn the relative probability of the two, and based on the cases where the actual demand was higher than what you had in stock, it will slowly learn to order more with a higher probability until the prediction based on past sales increases to a point where there is no extra sales to gain from ordering more and the probability of taking the over-ordering action starts to fall again.


How does keras train without disrupting the data set order I want to train a neural network using keras.

model.fit(trainX, trainY, batch_size=400, epochs=1000,shuffle=False)

I want the model training using the sequence of original data set order, not shuffle.

By default, Keras will shuffle training data before each epoch (shuffle=True).

If you would like to retain the ordering of your dataset, then set shuffle=False (docs here).


Machine learning mode retraining after getting new data I have trained a model for predicting the sale of items daily, such as daily car sales, with a machine learning model in Python. Now I get the new time real data (time series data). I want to retrain my mode. There are some issues:

How much data I should use for retraining model, namely what's the time
  range of data or when I should retrain my model. For example use the new one day or one week et al., data to retrain model.
What's condition of new data if I want to use the new data for retraining model.
How to evaluate the stability of new training model, and How to know abnormal prediction for new model.

Once you trained your model, you have a measure of accuracy.

You should retrain your model when the accuracy drops below a certain threshold.

Another approach is to use a Bayesian model, in which you update your model with each new observation.

I would suggest looking into pymc3's examples


Information Extraction/Semantic Search for long, unstructured documents I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store them in a structured database.

The ultimate goal is to extract and store information in a way that we can query those and any new incoming documents for fast and reliable information. For instance, I want to query a combination of entities from the knowledge base and then return the n-most relevant paragraphs/sentences from the documents. Since some entities like “World Bank” are extracted dozens of times for some documents, I need a way to query the entity in context. Otherwise I just end up with a database that contains the names of specific entities without any way to map them back.

NER usually seems like a good solution for this, however, the documents all have very unique structures which also change from document to document. For instance, a lot of relevant information is stored in tables, but also in long paragraphs.

As far as I understand, NER uses the surrounding words to identify entities, hence loading in the whole documents as raw text and manually tagging terms in the tables will probably not serve as good training data.

For now I built a function that extracts the raw text from pdf, doc and docx documents and extracted entities with spaCys NER model, however, I need to define my own entities and the domain is to scientifc for spaCy to deliever good results.  I also obtained a prodigy license for annotation.

Also, it seems I have to find a way to distinguish between "relevant" text and table entries and "junk" - aka footnotes, annexes, titles, subtitles, etc.

Any input is highly appreciated!

Thanks!

If you think filtering certain POS-tag words or "bad" topics from topic modelling doesn't suffice to distinguish between relevant text and junk, you can implement your custom filter. Try regex rules and use your prodigy licence to annotate junk to fit a classifier based on tf-idf or other text feature extractor.

But don't forget, there isn't a preprocessing framework that fits all needs.


Which performs better in time series forecasting, LSTM or SVR? I have run LSTM and SVR models on various datasets having sample values in the range of 1-4000 and the MAPE obtained in SVR was consistently lesser than that obtained through LSTM. I was told the reverse is true (that LSTM should perform better) but haven't found much information on this online. I would appreciate any feedback about this and any links to articles or papers (so far, I found grossly varied opinions).

So far I know, Recurrent Neural Network(RNN) is best for time series problem. LSTM(type of RNN), always consider long term dependencies and evaluate new value after understanding the whole series pattern.
Whereas SVR, consider each row as a sample for training data and predict the outcome and will not consider the previous patterns.

  For example: 10->20, 20->40, 30->60, and so on ...

  LSTM will try understand the whole series and then predict the next
  value, whereas for SVM each row is a individual training sample divided into
  Features/Target and will predict value based upon the different values
  it learnt so far.

Thats the reason RNN is better for timeseries.

Please refer:
https://www.researchgate.net/post/how_I_can_do_sequence_to_sequence_prediction_using_SVM

For SVR explanation
https://machinelearningmastery.com/time-series-forecasting-supervised-learning/


How to estimate machine learning training speed on different hardware I need to put together a proposal for buying a better computer for machine learning. Is there any good way to estimate the general training speeds of computer hardware? Basicly, I want to be able to say that if we purchase a certain computer it in trains x times faster than our current computers. Currently I have a i5 63000U 2.4ghz 2 core cpu, with no gpu. I know switching to a computer with a gpu will speed things up greatly. I just need a way to quantify the improvement to justify the investment. Any ideas will be greatly appreciated. Thanks in advance.

Is there any good way to estimate the general training speeds of computer hardware?

Probably not in general, because "training speed" means something completely different if you train  a kNN model (virtually no training) or a complex DL model, what kind of data and what kind of task.

However estimations exist for fixed settings, for instance I found this MLPerf Benhchmark or this paper.

Btw desktop computers are not ideal for very computer-intensive ML, cluster or cloud computing offer better options.


Multivariate Linear Regression with exponential trend-line The dataset that I'm working with has 2 independent variables (qty, volume) and 1 dependent variable (cost).  When I plot individual X with Y, it turns out qty vs cost gives an exponential decay trend line while volume vs cost gives a linear relationship.
I'm trying to come up with a linear model (since it's beginner-friendly) to predict the cost when a new input, volume set is given.  I tried to excel and while the statistic output looks reasonable, the predicted values are off quite a lot.  I read up on some articles and there is something regarding log transformation so that both qty and volume have a linear relationship with cost.  I'm not sure if log transformation applicable in this situation.  What would be the best way to approach this problem?
I plan to use R instead of excel next.
Here are some outputs of what I've done

Note: Thickness, width, and height essentially become volume so it doesn't affect the data much

I would suggest transforming "qty" into the log space. Even, you can do this using Excel. You can make a new column (for example qty_log), which is equal to the log of "qty". Then, you can fit a linear regression as follows:
$cost = a1 * log(qty) + a_2 * volume + a_0$
You will get much better results in this case.
This equation simply means that try to fit a model using volume and log of qty for predicting cost. You did so before but you used qty instead of log(qty)


Based on transformer, how to improve the text generation results? If I do not pretrain the text generation model like BART, how to improve the result based on transformer like tensor2tensor?
What are the improvement ideas for transformer in text generation task?

If you have a lot of data available to train, you should apply the techniques used in large transformer models, like GPT-2: very deep models (48 layers for the 1.5B parameters), modified initialization, pre-normalization,
and reversible tokenization. You could also apply GPT-3's locally banded sparse
attention patterns.
If you have very small training data, you can apply the "unwritten" aggressive techniques described in this tweet, namely data augmentation, discrete embedding dropout, normal dropout and weight decay, and lots of patient training time.
Update: I feel like the tweet thread I referred to is important, so here are the most relevant tweets:

How can you successfully train transformers on small datasets like PTB and WikiText-2? Are LSTMs better on small datasets? I ran 339 experiments worth 568 GPU hours and came up with some answers. I do not have time to write a blog post, so here a twitter thread instead.

To give a bit background: All this came about by my past frustration with replicating Transformer-XL results on PTB and having very poor results on WikiText-2 (WT2). On WT2, my best model after 200+ experiments was 90ish ppl which is far from standard LSTM baselines (65.8 ppl).

...

The key insight is the following: In the small dataset regime, it is all about dataset augmentation. The analog in computer vision is that you get much better results, particularly on small datasets, if you do certain dataset augmentations. This also regularizes the model.

The most dramatic performance gain comes from discrete embedding dropout: You embed as usual, but now with a probability p you zero the entire word vector. This is akin to masked language modeling but the goal is not to predict the mask — just regular LM with uncertain context.

The second most important factor is regular input dropout: You take the embeddings and dropout elements with probability p. This also has a data augmentation effect very similar to dropping out random pixels for images. What is a good way to think about this? 1/2

Remember that we can do King-man+woman=Queen? Now imagine input dropout removes the "man" component of "King". This forces the model to distribute specific information (gender in this case) into multiple dimensions to improve generalization making it more robust. 2/2

Otherwise, it is a game of further regularization (more dropout + weight decay) and of patience. I can train a good model without these tricks in 15 minutes and get 97 ppl. If I apply all these dropouts the model underfits after 7h of training to 63.4 ppl (better than LSTM).

You can also apply these data augmentation recipes to large datasets, but nobody would like to train for months on WT-103 for a couple of ppl points. In my opinion, techniques that require so much extra compute are more harmful to the community than useful. 1/2

Here the code changes to the public Transformer-XL repo that my results are based on: https://github.com/TimDettmers/transformer-xl/tree/wikitext2

With my changes to the public Transformer-XL repo, you can run this script to get down to 63.4 ppl on WT2: https://github.com/TimDettmers/transformer-xl/blob/wikitext2/pytorch/replicate_wt2.sh


predicted value is more conentrated than actual with DNN I have created an Deep Neural Network model with keras.
def nopitid_model(optimizer,loss='mean_squared_error',epochs=50):
model = Sequential()
model.add(Dense(24, input_dim=24, kernel_initializer='normal', activation='relu'))
model.add(Dense(15, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(5, activation='relu'))
model.add(Dense(1,))
# # Compile model
model.compile(loss=loss, optimizer=optimizer)
prediction = model.fit(x=X_norm,y=Y,batch_size=int(X_norm.shape[0]/5),
          verbose=0,
          epochs=epochs,
          validation_split=0.3)
return model,prediction

The MAE is about 5%, which is acceptable. However, after plot the predicted value vs actual value, I have noticed that the predicted value scattered between 1260 - 1290, whereas actual data range from 800-1600. More interesting, some predicted values also can be observed at 1230. I can't figure out what has caused this.
Looking for all suggestions.

It seems like your model hasn't learnt much and may almost always be predicting the mean of the target distribution. Please check if the mean of the actual target variable resides in the 1260-90 range.
This may happen if your features are not correlated to the target.
Also, by looking at your graph, the MAE is certainly larger than 5%.
You want to collect more informative features for tour model.


Designing a pretrained DNN for image similarity I am pretty new to deep learning and really hope that you can help me.

I want to write a python program that lets me choose an area in a reference image. This subimage of variable size should then be used to search in a database of images. Then the parts of the images with the highest similarity to the reference sub image should be given.
However I have big problems with the sizing of the reference and database images. I tried to read into pretrained DNNs (like VGG19) and use the features of a last layer for similarity computation. But these DNNs seem to accept input arrays only in certain resolutions.
Should I then rescale the reference image? The database images will most likely be much larger than the reference. Should I then partition all the database images into smaller subsets? Or use a single shot algorithm like YOLO?

Since there are so many different algorithms I would be very thankful for every comment or idea.

Lasse

That is not currently possible.

There is no such thing as a database that supports arbitrary content-based image indexing or image queries. Image Databases (IDBs) are indexed by R-tree family or members of the quadtree family. Those only support limited SQL (text) queries.

You would have search image-by-image for each query image which would defeat the purpose of having a database.


How to represent a dataset as a linked list-like graph? I would like to visualize this data set using Python and probably a visualization tool like Matplotlib. The data set contains three columns: a user id with a question, a user id with an answer, and time. I would like to visualize this data set as a linked list style graph. That is a tree-like structure which would show relationships based on the connections between questions and answers. I've tried different sets of keywords in an attempt to find out how to visualize a data set in this way, but to no avail. [I'm new to data science or analysis]. So I would appreciate some help with this.

The graph resulting from this kind of dataset is also known as a Network Graph and the kind of analysis you are trying to do is known as Social Network Analysis.

There are many prominent Python libraries for visualization and subsequent analysis of network graphs. The most widely used is NetworkX. It is easy to add nodes and directed edges in a NetworkX graph and visualize them with Matplotlib.

Installing NetworkX is a prerequisite.

pip install networkx

The following code creates a NetworkX DiGraph object, adds edges(and thereby nodes) to it and plots the graph.

import networkx as nx
from matplotlib import pyplot as plt
G = nx.DiGraph()
G.add_edges_from([(1, 3), (2, 3), (2, 4), (4, 1)])
nx.draw(G)

Hopefully, this should get you started :)


xgboost or lightgbm to handle Binomial problems I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like xgboost or lightgbm) to model the success probability. Which parameter shall I set to handle this in lightgbm or xgboost?

I do not think this is possible with the main (as I can tell) xgboost library, because this essentially sounds like a regression on rates/proportions situation (a dependent variable bounded in [0,1]).

An easy way to have implemented this would have been to allow the Beta distribution (see how it is bounded) as an objective function in xgboost, but it is not an option at this point.


Classification when one class is other I am working on a litigation support application using the Enron corpus, which contains about 600,000 unique text documents.

In litigation, one is often concerned with whether a document is responsive or non-responsive. One produces responsive documents to the opposing side, unless they are privileged (e.g., attorney client communication).

Here, I have sample sets of over 200 responsive and non-responsive documents.

The challenge here is that the topic of the responsive documents is about one thing, whereas the non-responsive documents could be about any number of topics,  ranging from spam, to soccer practice, to business documents, etc.  The non-responsive is what I would call diluted.

I don't know what those non-responsive classes are up front, and there is no purpose or value to breaking them out up front. If a document is non-responsive, then it needs to be quickly (and cheaply) dismissed.

If the samples are random when the classification is applied to the corpus my customers expect the split between corpus responsive and non-responsive to be close to the split of the sample.

What is the best approach to classify responsive versus non-responsive in this situation?

Below I briefly describe what I have tried. If there is a better approach, please share.

Using tf-idf, create an average vector for each document class (responsive and non-responsive).
Take the first 1000 terms (sorted by weight), such that each vector is the same length.
Normalized the vectors.
Note: these two vectors are only a 0.03 cosine similarity to each other.
For each document in the responsive sample set, calculate the cosine similarity to the single average vector.
Note: the average is 0.06.  It is a small number, as documents in the sample set have around 40 terms, as compared to an average vector of 1000.
In the non-responsive perform the same analysis (compare each non-responsive sample document to the non-responsive average vector).
In the non-responsive that same comparison is 0.03. Basically, the non-responsive average vector is diluted.

Based on this approach, I cannot really conclude if document has a higher cosine similarity to responsive compared to non-responsive then it is responsive.  By that measure 1/3 of the documents in the non-responsive sample set would be classified as responsive.

The non-responsive needs to be handicapped.  In this approach, how do I handicap it?  Is there another approach that would accomplish the same thing?

As a first step you should try logistic regression on your tf-idf vectors.  It is simple to implement and would provide a good baseline for comparison.  You can find an implementation in whatever language you're using.  You could also try some kind of (perhaps supervised) topic modeling to create a better feature space, but that would be more involved.


Distance Based Classification in Python I have large 3 dimensional dataset that I need to classify in a particular way. I'm hoping for something like a k-means clustering method, but instead of inputting the number of clusters I would like to input the maximum radius of a cluster.

Said another way, given a sphere of a defined size, I would like to find the minimum number of non-empty spheres that will cover all the data and classify the points accordingly.

I would like to output a 2D array of the same height and width, with labels that represent the sphere each point in the 3D array was classified in. No two points with the same label should have a euclidian distance greater than the maximum radius, no sphere should be empty, no spheres should overlap and the algorithm should use the minimum number of spheres possible to accomplish this.

Though I have little knowledge of the field, I have been looking into clustering algorithms for a little while and have not found one that accomplishes this.

Finding the minimum number of spheres likely makes this problem NP-hard, and a variant of the set cover problem. Good luck. It gets particularly difficult if you want to consider balls tha can be inbetween of data points.

There are two obvious clustering algorithms that will find an approximation: Leader and Complete Linkage Agglomerative Hierarchical Clustering. But they will not find the maximum, only an approximation, and probably with no guarantees.

Also, your assumptions are wrong: if you cover the data with balls of radius r, the maximum distance of two points will be 2*r.


Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding) Pandas has a method called get_dummies() that creates a dummy encoding of a categorical variable. Scikit-learn also has a OneHotEncoder that needs to be used along with a LabelEncoder. What are the pros/cons of using each of them? Also both yield dummy encoding (k dummy variables for k levels of a categorical variable) and not one-hot encoding (k-1 dummy variables), how can one get rid of the extra category? How much of a problem does this dummy encoding create in regression models (collinearity issues - a.k.a. dummy variable trap)?

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True.

One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a transformer which you can apply to your training and test set separately if you specify the total number of categories. This doesn't work with get_dummies ,for example, if the training set misses categories present in the test set.

You can still delete categories by simply deleting columns from the resulting numpy array (e.g. using n_values_ or feature_indices_ to see which columns correspond to the same feature). Some models work regardless, for example tree-based models. Also, L1 regularization can often set redundant features to zero (see Lasso regression).


Unbalanced data set - how to optimize hyperparams via grid search? I would like to optimize the hyperparameters C and Gamma of an SVC by using grid search for an unbalanced data set. So far I have used class_weights='balanced' and selected the best hyperparameters based on the average of the f1-scores.
However, the data set is very unbalanced, i.e. if I chose GridSearchCV with cv=10, then some minority classes are not represented in the validation data.
I'm thinking of using SMOTE, but I see the problem here that I would have to set k_neighbors=1 because in some minority classes there are often only 1-2 samples.
Does anyone have a tip how to optimized the hyperparameters in this case? Are there any alternatives?

Many thanks for every hint

Scikit-learn's GridSearchCV uses StratifiedKFold so all classes will be proportional represented in the splits. GridSearchCV can be used for hyperparameter search.
Imbalanced-learn's SMOTE can also be used. If there are fewer samples than k, it will only use available samples.


Access keys of pandas dataframe when using groupby I have the following database:

And I would like to know how many times a combination of BirthDate and Zipcode is repeated throughout the data table:

Now, my question is: How can I access the keys of this output? For instance, how can I get Birthdate=2000101 ZipCode=8002, for i = 0?
The problem is that this is a 'Series' object, so I'm not able to use .columns or .loc here.

You may use
df.groupby(['BirthDate', 'ZipCode']).size().reset_index().rename(columns={0: 'n'})

and now you have a data frame that you can easily manipulate.


Replace human judgements with mathematical approach/theories I would like to give a context of what I did.

1) Let's say there are two dictionaries (dict A and dict B) each containing a list of words/terms as shown below.

2) Now my task is to find matching words for dict A in dict B

3) I use an automated tool (fuzzy-matching/similarity) to do the above task and the output looks like below

4) Once I get the output as above, you can see that there are some records with match % less than 100. It is totally possible that dict B didn't have the exact matching term. It's fine.

5) So, what I do is review terms that have match % less than 50. Meaning I take those terms (that are less that are 50% match) and check for a related term in dict B again. Doing this, I am able to update the output like below. Because we know through human experience that sore throat lozenge and strepsils are related (matching is better now when compared to earlier where it was mapped to orange (totally irrelevant)). So this problem is more of a semi-automated task rather than full-blown ML task

So, now my question is (not on NLP or ML but below)

1) But how can I prove that choosing 50% as the threshold for manual review is the right one? Because this is a subjective thing/ based on individual judgment. Meaning I could have chosen 30% or 40% as well, it could have saved my time in manually reviewing terms

2) Meaning, this 50% isn't written in stone but what I am looking for is some theory/mathematical/statistical approach through which I can arrive at this threshold value rather than based on my judgment/subjective which I cannot defend/justify?

Can you people share some views/techniques on how can this be done in a systematic approach?

When it comes to choosing a threshold, I can see 3 approaches:

Make an educated guess

This is what you are currently doing. You pick a value and would need to argue why this is a reasonable threshold. Obviously, the argument is as strong as the assumptions you make.

Unsupervised way

If you compute the matching score for all pairs between A and B, you can analyze the histogram of these scores. Most likely, you will have quite a lot of scores near 0 and a few scores near 100. Then, you can decide to pick a threshold. This itself can be done in different ways. You can pick the median/mean matching score, this becomes close to option 1 but at least, the number comes from a specific mathematical concept. You can use the Jenks-Fisher algorithm (here is a Python implementation). In summary, the algorithm will find a threshold that splits your data into clusters which minimize the intra-cluster deviation. This would be better than the median/mean since it's expected that the data will be quite skewed.

Supervised way

If you somehow have access to which matches are confirmed/overturned by humans, you could use entropy/information gain to find the best threshold. This is similar to building a decision tree of depth 1 (a decision stump), where your input is the matching score and the target is a binary variable (whether or not a human says this is a match).

So your data would be something like this:

score, is_match
0.0, 0
0.1, 0
0.2, 1
0.3, 0
0.4, 1
0.5, 0
0.6, 0
0.7, 0
0.8, 1
0.9, 1
1.0, 1

My personal recommendation would be to use the Jenks-Fisher algorithm (option 2)


reason why there is two dense layer? I have been using the following model for image classification. I wonder why there is 512 Dense layer before the Dense3. Should I just use only Dense3? instead of Dense512-Dense3?
model = tf.keras.models.Sequential([
    # Note the input shape is the desired size of the image 150x150 with 3 bytes color
    # This is the first convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(150, 150, 3)),
    tf.keras.layers.MaxPooling2D(2, 2),
    # The second convolution
    tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The third convolution
    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # The fourth convolution
    tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    # Flatten the results to feed into a DNN
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dropout(0.5),
    # 512 neuron hidden layer
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dense(3, activation='softmax')
])

The Convolution layers do the job of feature extracting. Using those feature the DNN layers try to do the classification task. The 512 layer does the job of a feature selector like which feature is relevant for a class or not while last layer is just calculate sigmoid probability.


Consolidating multivariate time-series information from many data sets I am having trouble setting up a problem with regards to time series analysis. I have 30 data sets, where each set corresponds to a certain project. Each project has 7 features, and each feature has time series information sampled every week from 2018 to today.
One of the features is how much the project is under/over budget and I wish to use this as the label.
If I was learning on a single project it could be a straight forward multivariate time-series. As an example I could transform the data from this:
Week   X1   Y1
1      0.5  3
2      1    5
3      1.5  8

to this:
X1   X2   X3   Y
-    -    0.5  3
0.5  3    1    5
1    5    1.5  8

Then I would use the second table as my input data. However, with 30 different projects all with the same time-steps I'm not sure how to combine this information so a single model could learn it. One solution I thought I could do was a bagging approach. I would train 30 models and I could do a voting/weighted average for predictions, but I feel like this isn't the best approach. If anyone has dealt with a problem like this before, please let me know. Thanks in advance!

You could put all Information into 1 table with 30*7 features, since your timeline is the same for all tables.
Then, in order to predict the under/over budget, you can build a model with 30 output-nodes, where one node indicates whether a certain project was under/over budgeted.


Is there a fundamental difference from creating a model for each value in a category? I am creating a few models based on service requests. The services being requested are not distributed equally, some services being used sparingly, whereas others are quite common.
I had these services as categorical variables and built pipelines to incorporate them through one-hot encoding. I got to thinking that it may make more sense to train a model per service(at least for the common ones). Or does it make more sense to lump in the less common ones in a special category?
I am struggling with the regression model, coming in at 0.41 for my R2 value.

Is there a fundamental difference from creating a model for each value in a category?

Yes there is.
If a model is trained for each specific value of a variable (a category), then only the subset of data for this category can be used to train and test the model. As a consequence each model has a smaller number of instances to be trained from. Consequences:

In the case of a small category, there might not be enough instances to obtain a reliable model.
Every model is independent. This can be good or bad depending on whether this independence is also true in the data or not, or to what extent:

If the features behave in a completely different way depending on the category, then it's better to create individual models since each can really exploit the specific patterns for this category.
If the features have a very similar behavior across the categories, then independent models by category would potentially lose a lot of information.

In conclusion the choice often depends on:

How much data is available for each category.
How independent are the other features with respect to the category.


Keras: How to normalize dataframe with continuous and categorical data? I have a dataframe with about 50 columns. The columns are either categorical or continuous data. The continuous data can be between 0.000001-1.00000 or they can be between 500,000-5,000,000. The categorical data is usually a name, for example a store name.

How can I normalize this data so that I can feed it into a dense layer of a Sequential model?

The Y values are either 0 or 1, so it is a binary classification problem. I am currently normalizing all of the continuous data to be 0-1 and one-hot encoding all of the categorical data, so that if I have a column with 5 names it in, I will get a matrix with 5 columns filled with 0's and 1's. Then I join all of the continuous and categorical data and feed it into a Dense layer with init='uniform' and activation='relu'.

Is this the standard way of doing things?

Yes it does, you're doing well!

In most cases, categorical features(columns) should be one-hot encoded. However, continuous features might be a little complicated.

There are two common ways to preprocess continuous feature:

scaling features to range [0, 1] (as you have done)
removing the mean and scaling to unit variance(make the feature has zero mean and 1 standard variance)

In my practice, I take these two ways depending on my dataset.


How to deal with memory insufficient read by pandas in python I use pandas.read_csv to read a huge file for machine learning, but I got memory error.

Someone recommend me to set arg chunksize but I need sort, random access...etc. So I need to load whole data into memory or use another way.

Some ways I think it's possible is Hadoop. Another one is incremental training, but it's like reading chunksize in read_csv

Or other software/library/ways can I use?

I would suggest you to use Dask. I used it successfully when I had to read large data with my 4GB RAM. You can get more details here.

To read a CSV, you can do the following:

import dask.dataframe as dd

csv_file = 'data.csv'
df = dd.read_csv(csv_file)


Differentiating between normal and attack clusters In the below figure, we plotted some data of sensors in normal condition and under attack(outlier):
1.  Green points are normal samples in the training dataset.
2.  Cyan and red points are normal samples in the test dataset.
3.  Black and blue points are attack samples in the test dataset.
We wish to classify attack from normal samples in the test phase by using only the information we extract from the training dataset (the green points). I highly appreciate any suggestions for this purpose.

Thanks.

First of all please ignore my previous answer. I did not understand you properly.

The difficulty of the your problem goes back to the distribution of normal points in train and test. The distribution of new normal points are so different than what your model can learn from only green points (training set) so I assume any model will have a relatively poor performance on it. Anyways, having a look at scikit-learn documentation is good if you did not.

About Clustering as you asked I have an idea. An active are in Graph Analysis and Network Science is Community Detection (which is simply the clustering of graph nodes). The idea is that, assume all your points in this figure are nodes of a graph and based on their Euclidean distance in PC space, you put an edge between them (either binary or weighted). Then you can use Modularity-based algorithms to find dense subgraphs. I assume if you apply it to your data it will work. Also a similar idea is Spectral Clustering which internally uses the idea of graph partitioning.

According to the density of clusters, I would also propose to use density-based clustering approaches. They might be able to distinguish between those overlapping clusters "a little bit" (e.g. this paper). This paper has an approach to find the peaks of densities. Sounds compatible with your data.
Please note that those overlapping clusters are anyways difficult to distinguish.

My apologies for the first misunderstanding and hope it helps.

Good Luck!


One-class classifier for time series data classification My problem is different from the common time series data problem. What I need to do is check if future time series data is in accord with previous time series data I already consider to be correct.

In short, I need a one-class classifier applied to time series data, which have variable length (going from 110 to 125 points).

Does anyone know any lib that might help me with this task? I usually use Python+scikit-learn, but I'm open to use any other configuration.

Apart from the approach @Rolf Schorpion mentioned, there are others.
For example, you could use a deep neural network, specifically, an auto-encoder (see here for a tutorial).

But there's an important catch to all purely "data-driven" approaches: if the figure of 30 time series you mention in the comments is a typical order of magnitude for your training set, the results will be more or less arbitrary. If you don't define "accordance" in any way, this is a classification problem with only positive training data which consists of 30 data points with more than 100 features each.

Unless your data is very, very special (e.g., all time series are identical), there is just too much freedom in the solution of this problem. Different algorithms (or different parameter sets for the same algorithm) will use this freedom in very different ways. So you will probably see very different solutions when you experiment with different methods.

So if you don't want to do more or less arbitrary experiments and choose from the solutions afterwards, you have to use a method in which you somehow define the meaning of "in accordance" in advance. You may try and use traditional time series techniques to find a common model that fits each of the time series well. You could then postulate that "accordance" is a good fit to this model.

Or you might just do some exploratory analysis of your data to decide upon a simple rule which could be used to detect accordance. There's lots of possibilities, and without further details on the application it's difficult to decide which one is appropriate. Delegating the decision of what's "in accordance" to some algorithm doesn't seem to be a good choice in this case, though.


Appending dataframe with .values Is it necessary to append train X & test X with .values? But I checked and found the model works fine without appending .values, then why append?

In other words.. which among the following is better, and why?

X = df.iloc[:,4].values

OR

X = df.iloc[:,4]

.values coverts the dataframe into an array and remove the underlying structure introduced by pandas. The pandas DataFrame is a higher version of numpy array which gives the data a tabular look and feel which helps in better visualising the data (with the column names). However when we use .values we remove the higher structure and represent the data in a lower numpy array format. The advantage here is that the array format is computationally less expensive than the DataFrame format. In other words it increases the speed and allows faster computations. I hope this clears your question.


How can I split hourly time series? I'm newbie in R and time series analysis. I'm trying to build an Arima model.
My dataset has this structure:
DATA         Ora    VALORE
<chr>       <dbl>    <dbl>

2018-09-01    1      3646742
2018-09-01    2      3273110
2018-09-01    3      3069245
2018-09-01    4      2969621

I've converted this dataset in a hourly time series and split it in a training set  and a test set with this code:
y <- ts(data$VALORE,start=c(2018,09, 00:00), frequency=24*365)

y_train <- window(y, c(2018, 09), c(2020, 06))

y_test <- window(y,c(2020, 07), c(2020, 08))

Problems:

test set contains two observations (instead 1488 = all July and all August 2020);

training set contains 17518 observations (instead 16032).

Can I solve these problems?

The Solution that I have found:
y <- ts(data$VALORE,
               frequency = 24,
               start = 1)

train <- ts(y[1:12264], frequency=24) #70%
val <- ts(y[12265:17520], frequency=24) #30%


Finding cause and effect correlation in a dataset What methods might one use in order to find and analyze cause-and-effect in a dataset? An example might be something like plotting this:
import matplotlib.pyplot as plt
import numpy as np

n1 = np.array([100, 100, 150, 150, 150, 100, 100, 100, 100])
n2 = np.array([100, 100, 100, 100, 120, 120, 120, 100, 100])

plt.plot(n1, label="feature1")
plt.plot(n2, label="feature2")
plt.legend()
plt.show()

Where we see that some causal happens at feature 1, and a number of iterations later, feature2 experiences some sort of effect. The scales may be different, the iterations may bee slightly different, but looking at it on a graph like this, it appears that they are correlated by some sort of cause-and-effect.
Is there any way that one would be able to programmatically identify something like this? Or maybe give it a score of how correlated through cause-and-effect they might be?

Identifying correlation in data is possible with standard statistical tools, and it can be done programmatically. Correlation requires a representative sample, so in a case like the single graph that you show it wouldn't work: you'd probably need at least 20-30 such graphs in order to have a chance to confidently show correlation.
Identifying causation (cause and effect) is much harder. To the best of my knowledge it cannot be done from raw data without any additional knowledge. This is simply because with any dataset, there's no way to know if some external variable plays a causative role in the observations.
The way causation is proved is through a specific experimental setting, i.e. it's the design of the experiment which allows collecting data susceptible to prove causation. A/B testing is a method used for this kind of experiment. Example: suppose we want to test whether new drug A has an effect on outcome B. Design the experiment to observe the response variable B on a representative sample of patients. Randomly split the sample into two subsets: the first one is the control group, they are given a placebo. The second is given drug A. At the end of the experiment one can prove causation if and only if the second group shows a significantly different outcome compared to the control group.


Why do we use 2D kernel for RGB data? I have recently started kearning CNN and I coukdnt understand that why are we using a 2D kernel like of shape (3x3) for a RGB data in place of a 3D kernel like of shape (3x3x3)?
Are we sharing the same kernel among all the channels because the data would look the same in all the channels?

welcome to the community
I guess there's a confusion in your understanding of the kernel we use in case of rgb data. We normally use a kernel of equal number of channels as the input coming in (in this case as you mentioned it's RGB, so my number of channels for convolution operation would be 3). So instead of using a 3 X 3 Kernel, we use a 3 X 3 X 3 kernel. Weight matrix multiplication of kernel and image pixels happen channel-wise.
However, having said this, you can use a kernel of size 3 X 3 when input image is rgb, by specifying the stride as 1 in the third dimension. What this will do is convolute the kernel not only horizontally and vertically but also through the depth or specifically through the channels as well. I don't exactly know why one would like to do it.
Apart from this I guess the course or the video you are referring to might have specified '2D convolution on a 3D image'. That doesn't mean using a 2D kernel. And a 2D convolution on a 3D image uses a 3D kernel and after weight matrix multiplication you get a 2D image hence justifying the 2D convolution name.


Keras data structure for LSTM-networks After reading a while, I am confused now about my LSTM data structure. Assuming that I have a supervised learning problem with 1000 samples and 40 features as input. Now I want to create 10 timesteps of x. My resulting dimension of Keras data structure is

x: (1000, 10, 40)

with every two-dimensional matrix (1000, 40) shifted row-wise and ten times.

The question is now:

Which dimension has my target y to be?

My dimension is y=[1000,1] for one resulting target, but I also read y=[1000,10,1]. I suppose, the second dimension with value 10 should be the row-wise shifted target-vector. But, do I really need this? Shouldn't my solution be the right one?

Your target y can be whatever you need.

If you want to do sequence to sequence for example, y will have the same number of time steps than the inputs (you predict something for each timestep).

But you can also define an output with one timestep only, for text classification for example.
Your input data has 10 timesteps, but the output is only a prediction at the last timestep.

It depends on the problem you’re trying to solve.


LDA vs Word2Vec vs Others for predicting recipients of a message I'm investigating various NLP algorithms and tools to solve the following problem; NLP newbie here, so pardon my question if it's too basic.

Let's say, I have a messaging app where users can send text messages to one or more people. When the user types a message, I want the app to suggest to the user who the potential recipients of the message are?

If user "A" sends a lot of text messages regarding "cats" to user "B" and some messages to user "C" and sends a lot of messages regarding "politics" to user "D", then next time user types the message about "cats" then the app should suggest "B" and "C" instead of "D".

So I'm doing some research on topic modeling and word embeddings and see that LDA and Word2Vec are the 2 probable algorithms I can use.

Wanted to pick your brain on which one you think is more suitable for this scenario.

One idea I have is, extract topics using LDA from the previous messages and rank the recipients of the messages based on the # of times a topic has been discussed (ie, the message sent) in the past. If I have this mapping of the topic and a sorted list of users who you talk about it (ranked based on frequency), then when the user types a message, I can again run topic extraction on the message, predict what the message is about and then lookup the mapping to see who can be the possible recipients and show to user.

Is this a good approach? Or else, Word2Vec (or doc2vec or lda2vec) is better suited for this problem where we can predict similar messages using vector representation of words aka word embeddings? Do we really need to extract topics from the messages to predict the recipients or is that not necessary here? Any other algorithms or techniques you think will work the best? Should I just use supervised learning instead?

What are your thoughts and suggestions?

Thanks for the help.

Why the detour with the topic?

You can just learn a linear SVM to directly predict the recipient and avoid all the difficulties from topic modeling. What if a mail isn't about any of the previous topics? how many topics are there?