Incorrect output dimension? I am trying to start the learning of a cnn network which has 72 input and one output being a vector of length 24 stating the a class for each third input 72/24 = 3. There are 145 classes.

this is how i've designed the network currently passed my data:

print "After test_output/input"

print "Length:"
print len(data_train_input)
print len(data_train_output)

print len(data_test_input)
print len(data_test_output)

print "Type;"
print type(data_train_input[0])
print type(data_train_output[0])

print "Size [0]"
print data_train_input[0].shape
print data_train_output[0].shape

list_of_input = [Input(shape = (78,3)) for i in range(72)]
list_of_conv_output = []
list_of_max_out = []
for i in range(72):
    list_of_conv_output.append(Conv1D(filters = 32 , kernel_size = 6 , padding = "same", activation = 'relu')(list_of_input[i]))
    list_of_max_out.append(MaxPooling1D(pool_size=3)(list_of_conv_output[i]))

merge = keras.layers.concatenate(list_of_max_out)
reshape = Reshape((-1,))(merge)

dense1 = Dense(500, activation = 'relu')(reshape)
dense2 = Dense(250,activation = 'relu')(dense1)
dense3 = Dense(1 ,activation = 'softmax')(dense2)

model = Model(inputs = list_of_input , outputs = dense3)
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam" , metrics = [metrics.sparse_categorical_accuracy])

reduce_lr=ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1, mode='auto', epsilon=0.01, cooldown=0, min_lr=0.000000000000000000001)
stop  = EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=1, mode='auto')

print "Train!"
list_train_input = []
list_test_input = []

for i in range(len(data_train_input)):
    list_train_input.append(data_train_input[i])

for i in range(len(data_test_input)):
    list_test_input.append(data_test_input[i])

hist_current = model.fit(x = [np.array(list_train_input[i]) for i in range(72)],
                    y = np.array(data_train_output),
                    shuffle=False,
                    validation_data=([np.array(list_test_input[i]) for i in range(72)], np.array(data_test_output)),
                    validation_split=0.1,
                    epochs=150000,
                    verbose=1,
                    callbacks=[reduce_lr,stop])

Which generates this output:

After test_output/input
Length:
9436
9417
1017
1035
Type;
<type 'numpy.ndarray'>
<type 'numpy.ndarray'>
Size [0]
(72, 78, 3)
(24,)
Train!
Traceback (most recent call last):
  File "keras_cnn_phoneme_classification.py", line 382, in <module>
    model(train_input_data_interweawed_normalized, output_train, test_input_data_interweawed_normalized, output_test, test_name_interweawed_normalized)
  File "keras_cnn_phoneme_classification.py", line 361, in model
    callbacks=[reduce_lr,stop])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1405, in fit
    batch_size=batch_size)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1299, in _standardize_user_data
    exception_prefix='model target')
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 133, in _standardize_input_data
    str(array.shape))
ValueError: Error when checking model target: expected dense_3 to have shape (None, 1) but got array with shape (9417, 24)

Why am i getting this error?
I tried changing the output size of the dense3 to many  different?  but is is it expecting?

The error message is pretty clear... you feed a vector of length 24 to your model, but your model is outputting a vector of length 1.

Change :

dense3 = Dense(1 ,activation = 'softmax')(dense2)

to :

dense3 = Dense(24 ,activation = 'softmax')(dense2)


Setting up Neural Network for this problem I have a question regarding neural networks considering I am not an expert in NN.
Assume have a 5 by 5 grid that depending on me pushing any square (or combination of squares) some of those squares (not necessarily the squares I have pushed) will light up. My question is:

Can we set this problem as a NN problem if I have a set of input and outputs? Assume input layer with 25 neurons where all are zero except the ones are pushed and output layer is another 25 neuron where all are zero except the squares that light up. Assume in my training data set I have done this experiment 50 times.
Let’s say, in my training set square (3,3) has never been pushed. Can my trained model predict what would the response be if the square is pushed. So basically if a square has not been pushed in training set can NN still be used for prediction?
At the end, if all this is possible how would you set up a problem like this? Which NN method would you choose?

Yes, you can use a neural network exactly with the architecture you have described. Just define a consistent mapping from your grid to your input and output layer shapes. For example square $(x,y)$ can map to input neuron $5*y + x$ for zero-based indexing.
If square (3,3) is never on in your training input data then it will be hard to predict what will happen in an unseen example where it is on, so the short answer is no. However, if you think your data exhibits any special structure you might be able to leverage this to still make a good prediction.

If you think nearby squares behave similarly you might get good results with a convolutional neural network, which can learn from similar patterns nearby but possibly not exactly on the square you are interested in.
If you think certain patterns of squares behave similarly despite translation or rotation you could augment your training data with translations and rotations and you might then have training examples where (3,3) is on.

With all that said, 50 training examples is low so it might be hard to get good results.


Keras: Prediction performance does not match accuracy I am using Keras/CNN to identify plankton images collected with an in situ camera.  When making confusion matrices on the test sets following training I am finding that the accuracy from the predictions is quite poor.

I have a large number of files and have been using flow_from_directory and generators, I suspect that there might be something happening with the indexing of the predictions (e.g. this post), but as near as I can tell the indexing of the filenames/labels is matching up.

I worked up a quick example that is similar to what I am doing with the mnist_png dataset:

import numpy as np
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.layers.advanced_activations import LeakyReLU
from keras import backend as K
from keras.preprocessing.image import ImageDataGenerator

img_width, img_height = 28, 28

train_data_dir = 'S:/mnist_png/training'

num_epochs = 100
batch_size = 128
num_test_samples=10000

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()

model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))

model.add(Conv2D(16, (3, 3)))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.5))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))

model.add(MaxPooling2D(pool_size=(3, 3)))
model.add(Dropout(0.5))

model.add(Flatten())
model.add(Dense(512,activation='linear'))
model.add(LeakyReLU(alpha=.3))
model.add(Dropout(0.5))

model.add(Dense(512,activation='linear'))
model.add(LeakyReLU(alpha=.3))

model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='Adam',
              metrics=['accuracy'])

# augmentation for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip=True,
    rotation_range=90,
    validation_split=0.1)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='categorical')

model.fit_generator(
    train_generator,
    steps_per_epoch=batch_size,
    epochs=num_epochs)

...after 100 epochs I'm getting loss: 0.7517 - acc: 0.7482.

I then evaluate the test set thusly:

test_data_dir = 'S:/mnist_png/testing'

test_datagen = ImageDataGenerator(
    rescale=1. / 255)

test_generator = test_datagen.flow_from_directory(
    test_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    shuffle='False',
    class_mode='categorical')

#Evaluate model on test set
scores = model.evaluate_generator(test_generator,workers=12)

...the scores for that were 0.6184 and 0.8168, so in the same ballpark.

But it gets weird when I look at the predictions, e.g.:

test_generator.reset()# Necessary to force it to start from beginning
Y_pred = model.predict_generator(test_generator)
y_pred = np.argmax(Y_pred, axis=-1)
sum(y_pred==test_generator.classes)/10000

The proportion of predictions that are actually correct (calculated in the last line) is around 0.1; when I look at a confusion matrix it's all over the place and the diagonal is a lot of zeros.
I have verified that the test_generator.classes match up with the directories in test_generator.filenames, and shuffle is off.  Per this post calling test_generator.reset() should force it to take the files in order, but I'm wondering if it is not.

Does anyone have any thoughts on why this is happening or further steps to troubleshoot it?

Setting shuffle=false to evaluate_generator and predict_generator fixed the issue for me.


Reading Date Time in R (01.01.2019 00:00) How do you read a date in R that is given in the following way:

01.01.2019 00:00

While reading it with read.csv("my_file.csv", sep = ";", dec = ",") I am getting the date column as a "factor" class. I am tring to figure how to read this data time data as a date. Any thoughts?

Thank you!

You can use this method (assuming that 'time' is the column you want, and the date format is Date.Month.Year Hour:Minute):

d <- read.csv("my_file.csv", sep = ";", dec = ",")
strptime(as.character(d$time), format="%d.%m.%Y %H:%M")


Understanding why my binary classification is approaching 50% accuracy using TensorFlow and Keras I'm using the EMBER dataset (malicious and benign binaries).  I used
the built in EMBER code to vectorize the data and then filtered the
unlabeled from the training data.
https://github.com/endgameinc/ember

Now I'm building a very simply NN using TensorFlow and Keras and no
matter what parameters I play with it seems that the accuracy
approaches 50%.  For a comparison the EMBER team get's 98% when
using a Decision Tree (LGBM i think).

as an example this is one of the many things I've tried.

model = keras.Sequential()
model.add(keras.layers.Dense(100, input_dim=ndim, activation='relu'))
model.add(keras.layers.Dropout(0.5))
# model.add(keras.layers.Dense(100, activation='relu'))
# model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.fit(X_train,y_train, epochs=10000, batch_size=100)

In this instance I reduced my dataset to 1000 items from 600K to
attempt more epochs.  But either way I seem to approach .50.  Here's
an example epoch result

Epoch 2588/10000 1000/1000 [==============================] - 0s
86us/step - loss: 7.6877 - acc: 0.5180

I feel like I must be missing something fundamental as at some point
the training data should get better even if it's just due to over
fitting.  Also, when the model completes and I evaluate the test
data my accuracy is also almost exactly .50

EDIT 1: How I got the data
The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.

data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)

not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))

data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]

new_y_train.flush()
new_X_train.flush()

The above code was run once, then the below is run every time

#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)

#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]

tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))

EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine.  So the model itself seems to be fine, but there is something wrong with the data set.  I'm uncertain what that could be at this time.

You should consider analysing your data before applying NN. For instance, unlike Decision Trees Dense layers are not supposed to work well with denormalised features.


Using tensorflow for any type of dataset I have a problem that I have been working on with logistic regression which involves classifying clients based on their income and socio-demographic variables, all of which are categorical into those who have defaulted on their credit and those who haven't.

I have recently had tensorflow recommended as a package to use for neural networks, however on their website https://tensorflow.rstudio.com/tutorial_mnist_beginners.html they discuss tensorflow in terms of image classification.

My concrete question is thus:

2) Can tensorflow be used for any type of classification problem, for example, a binary classification of observations based upon categorical variables?

Cheers,
Steve

TensorFlow is a general purpose library for numerical computation using data flow graphs. It is primarily used for neural networks but can be used for any mathematical operations on multidimensional data arrays (tensors).

Thus, TensorFlow can be used to estimate binary logistic regression with explanatory categorical variables. An example can be found in the TensorFlow tutorials here.


Forecasting time series outside the training/test set I am trying to predict some time series based on precedent values using LSTM.

I have pretty good results when I compare the predicted time series with the test set (0,18% error)

I just miss how to forecast outside the interval of data ^^'

I have to admit that I used a point by point prediction method that looks like this:

def predict_point_by_point(model, data):
    predicted = model.predict(data)
    predicted = np.reshape(predicted, (predicted.size))
    return predicted

I then, I used it to override the predict function.
maybe the original function could have nailed the prediction to have a future time series? maybe the point by point isn't that bad neither?

I mean; how could I predict, some precise interval of time series (3months for example) without just reffering to the test set?

Example: the test set starts 01/01/2018 and ends 01/12/2018 and I want to predict 4 months from 02/12/2018

Thanks in advance for your help

Let's say you trained a forecasting model using the following base data:

time       | index_1 | index_2 | label
01/01/2018 | 80      | 70      | 1
01/02/2018 | 60      | 30      | 0
01/03/2018 | 75      | 90      | 1

You used time, index_1 and index_2 to predict label. Then you would simply need a dataset like this to predict 01/04/2018:

time       | index_1 | index_2
01/04/2018 | 60      | 75

Using your model on this data set should predict the label-value.

Now in a time series this can be more complicated, let's say what you actually want to predict is the label-value of time X from the indicie values of time X-4 months. In this case your data to build the model should look like this:

time       | index_1_lag_4_months | index_2_lag_4_months | label
01/04/2018 | 80                   | 70                   | 1
01/05/2018 | 60                   | 30                   | 0
01/06/2018 | 75                   | 90                   | 1

This model would predict the label-value for 01/04/2018 based on the indice values of 01/01/2018. To actually get a prediction we again give a data set like this:

time       | index_1 | index_2
01/04/2018 | 60      | 75

Only the output would not be the label-value for 01/04/2018 but instead for 01/08/2018.


Dissecting performance issues with Random Forest My task is to identify potential situation for trading and determine whether a candidate is going to succeed or not. I have a system in place to identify candidates, but there is a high rate of false positives.

To try to reduce it, I have been training a Random Forest in the hopes of pre-emptively eliminating high risk candidates. The problem is that I am using two separate sources of data: one which is a 10-year historical record, and the other which is obtained in real-time and stored in a separate database. If I split the historical data into training and testing (let's say 0.7 - 0.3 split), the results are improbably perfect: no mistakes in the classification for categories with 100's of observations. However, when I try to apply the trained model to the observations from the real-time data, the labelling accuracy is about 50%. This has presisted, despite attempts to tweak hyperparameters and introduce new features.

How could I investigate if there are substantial differences between the data from the two sources, and how could I try to go about ameliorating this situation?

There are several things you can do, even though giving a detailed descripton of your datasets would help a lot:

The first thing is that you should is to thoroughly compare the two datasets:

Are the same features present in both datasets?
Is the data collected in the exact same way? As an example, if age is a feature make sure that it is not recorded differently. Let's say one dataset could use the categories (<25, 25-50, 50+) and the other the exact age.
How do they differ in size?
Plot the distribution of each feature in both dataset: do they differ significantly, and why? You could test the hypothesis that a feature is drawn from the same distribution in both datasets.
Look for correlated features: correlated with each other and with the target distribution

The amelioration of your algorithm will entirely depend on the answer to these questions. If you observe significant differences between the two datasets then your historical data will not be a good training set for this task.

Independently from that, your 100% accuracy is very suspicious and your algorithm is probably overfitting.


Hybrid classification neural network I have product data and I need to classify products to categories (for example Lenovo laptop to Laptops category, etc.), each product has properties such as:

description
list with image URLs (typically 4 photos)
product-specific properties (watches have a mechanism type attribute, etc.)
manufacturer
category ID

Category ID is my target variable, do you know some resources (articles/books) where someone did something similar? I heard about the transfer learning (answer to this question Hybrid Convolutional and Conventional Neural Networks, is it a good approach?). My biggest problem is that I don't know how to connect CNN for images similarity and conventional neural network.
Thanks for the help.

Basically, you need to create a whole system which contains multiple ML algorithms. We can go feature-wise and see how the system could work.

Description: This will consist of text which could be easily vectorized using Doc2Vec. This vector will act as a feature for the final model.
Product images: You will need to create a model which make some sort of classification. Once, the model is trained to remove the last layer ( mostly the Softmax layer ). This will fetch us the encoded representation of the image. You may use an AutoEncoder for this.
Manufacturer and other properties: I think you may need to omit this feature as they will be product-specific and create complexities while training.

Once, we have all the above models trained, we can feed the features ( generated from these models ) to a neural network, which will produce the final result.


How to handle addresses of the restaurants to feed the data-set in the ML model? I have data from different restaurants which have also address of the restaurants now I want to predict the food delivery timing based on the given data, now the restaurant address is one of the crucial data which I need to predict the food delivery timing, now my problem is the address is in string format so how do I handle this address to feed the data in the ML model. I have a total of 35 unique addresses if I do one-hot encoding my dataset will be very large and it will take too much time to train, is there anyway except one-hot encoding to handle the addresses of restaurants.

Data Sample-

You can convert each address to a pair of geographical coordinates (latitude, longitude). In this way, you'll have a rough measure of the distance between different location. This can be done with Google Maps, for example.

I suggest you to use compute Manhattan distance between locations, in order to properly estimate travel times within a city.


Confusion regarding classification accuracy calculation and result The total number of data points for which the following result is obtained = 1500. Out of which, I have

1473 labelled as 0 and
the remaining 27 as 1 .

As can be seen from the confusion matrix, out of 27 data points belonging to class 1, I got only 1 data point misclassified as 0  . So, I calculated the accuracy for individual classes and got  Accuracy for class labelled as 0 = 98.2% and for the other as 1.7333%. Is this calculation correct? I am not sure...I did get a pretty good classification for the class labelled as 1 so why the accuracy for it is low?
The individual class accuracies should have been 100% for class0 and around 98% for class1

Does one misclassification reduce the accuracy of class 1 by so much amount? This is the how I calculated the individual class accuracies in MAtlab.

cmMatrix  =
1473    0
1       26

acc_class0  = 100*(cmMatrix(1,1))/1500;
acc_class1= 100*(cmMatrix(2,2))/1500;

You might want to take a look at the confusion matrix  wiki page.

Accuracy = (TP + TN) / (P + N) = (26 + 1473)/1500 = 99.9%

I guess, if you want the break down by the class, it would have been:
1473/1473
and
26/27, but you have 1500 as the denominator in both classes.

Even though I think conventionally most people report on the entire model rather than just on a particular class.


Does the Bayesian MAP give a probability distribution over unseen data? I'm working my way through the Bayesian world. So far I've understood that the MLE or the MPA are point estimates, therefore using such models just outputs one specific value and not a distribution.
Moreover, vanilla neural networks do in fact something like MLE, because minimizing the squared-loss or the cross-entropy is similar to finding parameters that maximize the likelihood. Furthermore, using neural networks with regularization is comparable to the MAP estimates, as the prior works like the penalty term in error functions.
However, I've found this work. It shows that the weights $W_{PLS}$ gained from a penalized least-squared are the same as the weights $W_{MAP}$ gained through Maximum a posterior:

However the paper says:

"The first two approaches result in similar predictions, although the
MAP Bayesian model does give a probability distribution for $t_*$ The
mean of this distribution is the same as that of the classical
predictor $y(x_*; W_{PLS})$, since $W_{PLS} = W_{MAP}$

What I don't get here is how can the MAP Bayesian give a probability distribution over $t_*$, when it is only a point estimate? Consider a neural network - a point estimate would mean some fixed weights so how can there be a output probability distribution? I thought that this is only achieved in the True Bayesian where we integrate out the unknown weights, therefore building something like the weight averaged of all outcomes, using all possible weights.

Bayesian MAP gives a point estimate $\mathbf{w}_{MAP}$ for model parameter $\mathbf{w}$, which implies
$p(\mathbf{w}|\mathbf{t},\cdots) = \left\{\begin{matrix}
1 & \mathbf{w}=\mathbf{w}_{MAP}\\
0 &\text{o.w.}
\end{matrix}\right.$
However, distribution $p(t_*|x_*, \mathbf{w}_{MAP}, \sigma^2)$ is the probabilistic model of target $t_*$ given input $x_*$ and parameters; after eq. (8), paper explicitly says $x$'s are omitted from the givens of target likelihood $p(t|.)$, but I think they should have been kept there to avoid confusion. It might also be helpful to remind that $(\mathbf{x}, \mathbf{t}) = \{(x_1, t_1),...,(x_n, t_n)\}$ are the observed (input, target) pairs.
Paper says when $\mathbf{w}_{MAP}$ is calculated, which would be the same as $\mathbf{w}_{PLS}$, mean of target distribution will be the same as the deterministic target $t_* = y(x_*; \mathbf{w}_{PLS})$ from Penalized Least Squares. That is,
$\mathbb{E}[t|x_*,\mathbf{w}_{MAP}, \sigma^2] = \int tp(t|x_*,\mathbf{w}_{MAP}, \sigma^2)dt = y(x_*; \mathbf{w}_{PLS})$
In summary, in (Penalized) Least Squares, both target and parameters are deterministic. In non-Bayesian and Bayesian MAP, parameter is deterministic but target is probabilistic. In (true) Bayesian, both target and parameters are probabilistic. In full Bayesian, hyper-parameters (parameters of distribution of parameters, e.g. $\alpha$) are also probabilistic.


Transfer learning between Language Model and classification Following this fast.ai lecture, I am trying to understand the mechanism of Transfer Learning in NLP from a general Language Model (LM) to a classification problem.

What is exactly taken from the Language Model training? Is it just the word embeddings? Or is it also the weights of the LSTM cell? The architecture of the neural net should be quite different - where in a LM you would output a prediction after every sequence-step, in a classification problem you would only care about the output of the final sequence step.

(I would happy to know what is the general practice, and also if anyone knows how fast.ai does it)

Ok, so it seems that the whole idea of transfer learning in NLP is to use more than just the word-embeddings which is considered "low level", but rather higher level representations. This is akin to what goes on in Computer Vision where the final (or almost final) layer embedding in nets trained on ImageNet are then used in transfer learning in other tasks, or as the authors dubbed it: "a counterpart of ImageNet for NLP". This is similar to what GPT is doing.

You can find more info also in the accompanying article one of the authors wrote.

Specifically, from what I gather after reading the paper, is that they use a 3-layer LSTM with dropout for the Language Model part. Then they fine tune it with the relevant text (using a bunch of tricks specified in the paper/lecture). Then they use the same architecture weights, but augment the model with 2 additional linear blocks. The input for the final blocks are: the final input, the final hidden state, maximum and averaged hidden state of all previous hidden states of the regular. This model is also fine-tuned on the supervised task (classification) using "gradual unfreezing".


Value Error: Operands could not be broadcast together with shapes - LSTM I am trying an LSTM model using tensorflow following this tutorial .

I am having trouble understanding why am I getting an error in my test set when I try to invert scaling for forecast (line 86 in the tutorial).

After loading the dataset and created featured, following is what I did for reframed dataset,

# split into train and test sets
values = reframed.values
split_point = len(reframed)- 168

train = values[ : split_point, :]
test = values[split_point: , :]

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

# reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))

print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
>> (2399, 1, 39) (2399,) (168, 1, 39) (168,)

# design network
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

# fit network
history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)

# make a prediction
yhat = model.predict(test_X)
test_X = test_X.reshape((test_X.shape[0], test_X.shape[2]))

inv_yhat = np.concatenate((yhat, test_X[:, 1:]), axis=1)

After this step, the following code that has scaler.invert_transform throws the error.

inv_yhat = scaler.inverse_transform(inv_yhat)

  ValueError: operands could not be broadcast together with shapes (168,39) (41,) (168,39)

Shape,

test_X.shape
>> (168, 1, 39)

yhat.shape
>>(168,1)

inv_yhat.shape
>> (168,39)

Any help or sugeestions would be appreciated!

The answer that OP provided is correct, yet I would like to elaborate a little more on it, in an attempt to shed more light.
First of all, you have to understand what is performed by the call
reframed = series_to_supervised(scaled, 1, 1)

in line 50.
Shortly, let us assume that the input has n samples of m features (variables). This means that the shape of the scaled dataframe is (n) x (m). The particular call to series_to_supervised() in line 50 implies that the scaled dataframe will be reshaped into a (n-1) x (2m) dataframe, to include the current and the 1-step look-back values, after dropping the row of NaN values due to the shift operation inside the body of series_to_supervised(). Most likely, your error is caused by line 52:
reframed.drop(reframed.columns[[9,10,11,12,13,14,15]], axis=1, inplace=True)

because you are probably dropping an arbitrary number of columns. The code is structured so that it forecasts one time series based on the past values of multiple time series (including the signal of interest). Therefore, you have to drop (m-1) columns, so that in the end all m past values persist, along with the current value of the signal of interest that will be used as ground truth. This means that the shape of reframed should be (n-1) x (m+1) after line 52. If not, you probably get that error with the dimension mismatch when reverting the scaling.
At this point I would like to emphasize on the fact that you should be very careful with logical errors when dropping columns, because you want to drop all but the signal of interest in line 52. If you keep any other column the code will still run, but you will certainly get logical artifacts.
Moreover, if you drop any column other than the first one (at index 0), you have to modify the concatenation in lines 85 & 90 accordingly, otherwise you will also get logical artifacts.


Should I use regularization every time? I have learned regularization for linear and logistic regression but when I implement that algorithm to my code generally my estimates not changing.I mean,it looks like ineffective.I know,it's for overfitting.So if I use it in my code every time ,could it be a problem? or is this a good thing?

Normally you use regularization. The exception is if you know the data generating process and can model it exactly. Then you merely estimate the model parameters. In general you will not know the process, so you will have to approximate with a flexible enough model. If the model is not flexible enough you will not need to regularize but you won't approximate well anyway. If you use a more flexible model, you will get closer on average (low bias) but you will have more variance, thus the need for increased regularization. In other words, the amount of regularization you need depends on the model. This is related to the bias-variance trade-off.

Welcome to DataScience.SE.


Using regex in redshift to find dollar values I have a field in a Redshift table that has user-generated text.  The field is where users can say how much they think something costs.

Ideally it'd just be a decimal, but it's varchar.  So users can type
"I think this is worth \$25", or "I'd pay 55" or "\$117".

So I'm trying to use regexp_substr to pull this out.  Specifically regexp_substr(f.comment_text, '\\$?[0-9]*').  But this doesn't work on a subset of entries for some reasons (eg Could do for $115).

If I remove the ? it works on that, but no longer on entries that don't use $.  Why?  And what should I use instead?

I am getting that you want extract numbers with either '$' or even not without a dollar.

Why are not using just [0-9]+ for that field? The results will be like

    some $1 something >> ['1']
    some other 115 >> ['115']
    $115 some thing >> ['115']

If you want to include '\$' sign, then '\$' can be repeated zero or once or by typo more, then regex would be

    \$*[0-9]+

which outputs

    some $1 something >> ['$1']
    some other 115 >> ['115']
    $115 some thing >> ['$115']

If this is not what you want, comment.

Hope this helps.

I would recommend using online regex matchers like here. They have short and sweet references which really help us get to the context.


Sklearn ValueError: X has 2 features per sample; expecting 11 I try to visualizing multiple logistic regression but I get the above error.

I'm practicing on red wine quality data set from kaggle.

Here is a full traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-88-230199fd3a97> in <module>
      4 X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
      5                      np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
----> 6 plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
      7              alpha = 0.75, cmap = ListedColormap(('red', 'green')))
      8 plt.xlim(X1.min(), X1.max())

/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/base.py in predict(self, X)
    287             Predicted class label per sample.
    288         """
--> 289         scores = self.decision_function(X)
    290         if len(scores.shape) == 1:
    291             indices = (scores > 0).astype(np.int)

/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/base.py in decision_function(self, X)
    268         if X.shape[1] != n_features:
    269             raise ValueError("X has %d features per sample; expecting %d"
--> 270                              % (X.shape[1], n_features))
    271
    272         scores = safe_sparse_dot(X, self.coef_.T,

ValueError: X has 2 features per sample; expecting 11

Here is the code:

#Split the variables
X = dataset.iloc[:, :11].values
y = dataset.iloc[:, -1].values

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

# Fitting Logistic Regression to the Training set
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(random_state = 0)
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)

# Making the Confusion Matrix
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred)
acc_score = accuracy_score(y_test, y_pred)
print(acc_score*100)

Below is the visualization code:

# Visualising the Training set results
from matplotlib.colors import ListedColormap
X_set, y_set = X_train, y_train
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 1, stop = X_set[:, 0].max() + 1, step = 0.01),
                     np.arange(start = X_set[:, 1].min() - 1, stop = X_set[:, 1].max() + 1, step = 0.01))
plt.contourf(X1, X2, classifier.predict(np.array([X1.ravel(), X2.ravel()]).T).reshape(X1.shape),
             alpha = 0.75, cmap = ListedColormap(('red', 'green')))
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1],
                c = ListedColormap(('red', 'green'))(i), label = j)
plt.title('Logistic Regression (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

I know that the error is that the model has been trained using 11 functions, but it is envisaged to use 2 functions, but I do not know exactly what to change.

X = dataset.iloc[:, :11].values

It is asking for 2 features per sample. But you are passing the 11. Do it like this

X = dataset.iloc[:, :2].values


Structured and unstructured pruning for deep learning models I was trying to understand structured and unstructured pruning techniques used for deep learning models: link 1 and link 2. To recap what I have understood that unstructured pruning is based on weight pruning however structured pruning is basically done through filter pruning, channel pruning, and filter shape pruning. So, my question is the unstructured pruning is done only for CNN models and cannot be done for example for MLP models as there are no filters and channels?

You can do structured pruning for MLPs. You would remove things like whole neurons or layers (not specific weights).


Why is Local Outlier Factor classified as Unsupervised if it requires training data with no outliers? In Scikit-Learn, the Local Outlier Factor (LOF) algorithm is defined as an unsupervised anomaly detection method.

So then I don't understand why this algorithm requires pre-filtered training data. Perhaps "training data" here simply means "data to start with?" But the example code provided by SciKit-Learn clearly shows training data which explicitly contains NO anomalies. Does that mean that this model would NOT work if the training data contains anomalies? And more importantly, how do I find anomalies in the training data using this algorithm?

Here is the website.
https://scikit-learn.org/stable/auto_examples/neighbors/plot_lof_novelty_detection.html#sphx-glr-auto-examples-neighbors-plot-lof-novelty-detection-py

I changed the sample code to include anomaly data in the training data set and the model still found a decision boundary that looked correct. So am I just getting confused because of the way the documentation and sample code is written? Or is does this model really need a clean training data set?

It appears that SciKit-Learn implements two "modes" for LocalOutlierFactor, where one is unsupervised and one is semi-supervised. I think I misunderstood the documentation and implementation at first.

The two modes are "outlier" and "novelty". The sample code provided on the SciKit-Learn website uses Novelty detection mode. When using the "outlier" mode the model does NOT require a training dataset--an unlabeled dataset is used as whole and a score is given to every data point.

  novelty : boolean, default False
  By default, LocalOutlierFactor is only meant to be used for outlier detection (novelty=False). Set novelty to True if you want to use LocalOutlierFactor for novelty detection. In this case be aware that that you should only use predict, decision_function and score_samples on new unseen data and not on the training set.


Does logistic regression can only solve binary classification problem? Does logistic regression only solve binary classification problems?
How can logistic regression solve multiple-class problems?

No, multiclass classification is also possible. Try reading up on 'One vs All' multiclass classification.

https://en.wikipedia.org/wiki/Multiclass_classification

Multiclass, One vs All classification in the case of logistic regression:

https://www.coursera.org/learn/machine-learning/lecture/68Pol/multiclass-classification-one-vs-all


Why does Caret featurePlot returns returns NULL? I am trying to use featurePlot of caret in R.

This is the code I am using:

trellis.par.set(theme = col.whitebg(), warn = FALSE)
caret::featurePlot(x = df[, 3:6],
            y = df$ud,
            plot = "scatter")

where all the columns of df[, 3:6]are of numeric type, while df$ud is a binary factor.

I also have installed the library ellipse as suggested elsewhere.

However, I keep getting NULL.

Any idea why?

I had this problem due to my y variable not being a factor which is required. call:

typeof(df$ud)

make sure it is not interpreting the binary as character or numeric but indeed ad a factor.


Effect of embedding dimension in performance of embedding algorithms Consider a simple Matrix Factorization or any embedding algorithm. It is obvious that a very small embedding dimension does not contain enough information so the algorithm will have low performance and enlarging the dimension will improve the performance. However, as I tried various algorithms, enlarging the dimension more than a threshold decreases the performance of the algorithm. Why does this happen?

I know using larger embedding while you can get the same result with smaller ones does not make sense, however, I am just curious to find out more about that.

You are probably just overfitting.

When you increase the number of parameters that are being trained in your model, the likelihood of them learning some aspect of the training data that doesn't generalize to unseen data increases. So there should be some sweet spot, where the model is able to express the underlying structure in the data sufficiently, without putting too much emphasis on those aspects that are specific to the training data.


How to use scikit-learn to extract features from text when I only have positive and unlabeled data? I'm looking for something similar to this
https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#sphx-glr-auto-examples-text-plot-document-classification-20newsgroups-py
But instead of positive and negative examples, I have positive examples and a bunch of unlabeled data that will contain some positive examples but is mostly negative.
I'm planning on using this in a pipeline to transform text data into a vector, then feeding it into a classifier using
https://pulearn.github.io/pulearn/doc/pulearn/
The issue is I'm not sure the best way to build the preprocessing stage where I transform the raw text data into a vector which would then be fed into the classification model.
If anyone has any different ideas on how I can transform positive and unlabeled raw text into a vector to feed into the pulearn module I would like to hear as well, thanks!

So you should be clearer with what you are asking.

What are the classes of your classifier? Positive and unlabelled?

To create numeric feature from text you can use:
a) tf-idf, which works well with small datasets/ sentences.
b) Handmade features, like extracting sentence length, occurence of certain words..
c) Word embeddings, like word2vec or sentence2vec. Strong method when a does not work. This will convert every word or sentence/ document to a numeric vector of length n, usually between 100 and 500. See the gensim library for that
d) a combination of the above.

The main criteria of choosing the feature transformer should be that it captures your text well and that is something you can decide only by looking at the data yourself and trying out some experiments.
Good luck!


dataframe.columns.difference() use I am trying to find the working of dataframe.columns.difference() but couldn't find a satisfactory explanation about it. Can anyone explain the working of this method in detail?

The function dataframe.columns.difference() gives you complement of the values that you provide as argument. It can be used to create a new dataframe from an existing dataframe with exclusion of some columns. Let us look through an example:

In [2]: import pandas as pd

In [3]: import numpy as np

In [4]: df = pd.DataFrame(np.random.randn(5, 4), columns=list('ABCD'))

In [5]: df
Out[5]:
          A         B         C         D
0 -1.023134 -0.130241 -0.675639 -0.985182
1  0.270465 -1.099458 -1.114871  3.203371
2 -0.340572  0.913594 -0.387428  0.867702
3 -0.487784  0.465429 -1.344002  1.216967
4  1.433862 -0.172795 -1.656147  0.061359

In [6]: df_new = df[df.columns.difference(['B', 'D'])]

In [7]: df_new
Out[7]:
          A         C
0 -1.023134 -0.675639
1  0.270465 -1.114871
2 -0.340572 -0.387428
3 -0.487784 -1.344002
4  1.433862 -1.656147

The function returns as output a new list of columns from the existing columns excluding the ones given as arguments. You can also check it:

In [8]: df.columns.difference(['B', 'D'])
Out[8]: Index(['A', 'C'], dtype='object')

I suggest you to take a look at the official documentation here.


Stuck on building a customer support chatbot from scratch using reddit dataset I've a trained model who can mimic day to day conversation occurring on reddit. But, here my problem is that I want it to reply to a specific use cases based on the vocabulary it had learned.

Summary:-
I am building a college chatbot, and there are many use cases. I've trained a model on reddit dataset, and now I've a model who can mimic reddit conversation. But, I want it to map it's vocabulary to one of my use cases. How should I tackle this scenario?

Any idea? Please help, I've tried searching the internet for days, but no answer.

Welcome to the site! I'm going to make some assumptions here to clarify your question and attempt to give you some guidance: it sounds like (1) you are working with a general chatbot framework and (2) that framework has been trained on a dataset that contains text from reddit.

So, what you need to do is now take that framework and train it again on your own custom dataset (relating to colleges?). I'm assuming that the training can be done based on using a chatbot framework that has this flexibility.

At this point it would beneficial if you posted (1) which chatbot framework you are using and (2) tell us about your data. Do you have a lot of it? Is it in question and answer format?


Why is the dimensionality reduction and/or feature selection techniques are considered to be not always effective? Can you please explain when (under what circumstances) the dimensional reduction and/or feature selection techniques are not effective to mitigate the curse of dimensionality?

This is helpful when the data is not understandable and you don't have any data dictionary or you have too many columns which doesn't make any sense even after through investigation, then it is wise to go for Dimensionality Reduction.

There is very good chance that you loose information at granular level, for example you gave 100 columns and you got 10 PC's(Applied PCA) which can explains most of the Data. As you can only get that much out of that technique.

Feature Selection: If you miss even a single feature which is significant WRT target variable but you overlooked that feature then your model might not be able explain the most, you are even satisfied with the result but still there is window of improvement. You should be very careful and you need check as many times(many iterations) as possible to build a good/decent model and get the best results out of it.

I hope this may help you.


Standardization on training and split data I am confused on which of the following should be used for standardization:

method 1: fit transforming training data and only transforming test data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

method 2: fit transforming both training and test data
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
# scaler_train=sc.fit(X_train)
#X_train_sd=scaler_train.transform(X_train)
X_test = sc.fit_transform (X_test)
#scaler_test=sc.fit(X_test)
#X_test_sd=scaler_train.transform(X_test)

this is a follow up question to:
StandardScaler before and after splitting data

You should only fit your scaler on training data. Your scaler is part of your model and fitting your scaler to some data can be considered as learning from this data.
Test data is used to evaluate your model on previously unseen data, so if you fit your scaler to test data, it is not "unseen" data anymore.


Connect Tableau and KoBoToolbox Is it possible to connect kobotoolbox to tableau in any way?
Any kind of method would be much appreciated.

I see that Kobol offers a REST API. Thus, one could create WebDataConnector to fetch the data in Tableau. If that's an overkill, you could write a script in e.g. Python which extracts the data from the REST API and creates a csv file.

REST API
http://support.kobotoolbox.org/en/articles/592398-api-and-rest-services

REST API -> Tableau
https://www.youtube.com/watch?v=1y5LsqQd00s&list=WL&index=2&t=0s


SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one? I am using LogisticRegressionWithLBFGS to train a multi-class classifier.

Is there a way to get the probability of all classes (not only the top candidate class) when I test the model on new unseen samples?

P.S. I am not necessarily obliged to use the LBFGS classifier but I would like to use the logistic regression in my problem. So if there is a solution by using another LR classifier type, I would go for it.

I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of predictions = model.transform(testData) as print(predictions) in PySpark you will get the probability of each labels, You can check the below code and output of the code:

from pyspark.sql import DataFrame
from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a Random Forest model.
rf = RandomForestClassifier(labelCol="label", featuresCol="features", numTrees=12,  maxDepth=10)

# Chain RF in a Pipeline
pipeline = Pipeline(stages=[rf])

# Train model.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

Now your work start from here. try printing predictions and values of predictions

print(predictions)

Output:

DataFrame[label: double, features: vector, indexed: double, rawPrediction: vector, probability: vector, prediction: double]

So, in the DataFrame you have the probability which is the probability of each indexedLabel, Further I have checked it as:

print predictions.show(3)

Output:

+-----+--------------------+-------+--------------------+--------------------+----------+
|label|            features|indexed|       rawPrediction|         probability|prediction|
+-----+--------------------+-------+--------------------+--------------------+----------+
|  5.0|(2000,[141,260,50...|    0.0|[34.8672584923246...|[0.69734516984649...|       0.0|
|  5.0|(2000,[109,126,18...|    0.0|[34.6231572522266...|[0.69246314504453...|       0.0|
|  5.0|(2000,[185,306,34...|    0.0|[34.5016453103805...|[0.69003290620761...|       0.0|
+-----+--------------------+-------+--------------------+--------------------+----------+
only showing top 3 rows

Only for probability column:

print predictions.select('probability').take(2)

Output:

[Row(probability=DenseVector([0.6973, 0.1889, 0.0532, 0.0448, 0.0157])), Row(probability=DenseVector([0.6925, 0.1825, 0.0579, 0.0497, 0.0174]))]

In my case I have 5 indexedLabels and so the probability vector length is 5, Hope this will help you to get the probability of each labels in your problem.

P.S: You will probably get the probability in Decision Tree, Logistic regression. Just try to get the summary of model.transform(testData).


ML model to transform words I build model that on input have correct word. On output there is possible word written by human (it contain some errors). My training dataset looks that:

input - output
hello - helo
hello - heelo
hello - hellou
between - betwen
between - beetween
between - beetwen
between - bettwen
between - bitween

etc.
During preprocessing I add a measure of the distortion of a word. Then I hardcoding letters for numbers.
My current model's using CNN. The number of neurons of input is the same as the longest word in training dataset and the number of neurons of output is the same as the longest word in traning dataset.
This model doesn't work as I excepted. Word on the output is not look as I except.
eg.

input - output
house - gjrtdd

Question:

How can I build/improve model for this task? Is CNN a good idea? What other methods can I use for this task?

Try a totally different approach, using Generative Adversarial Networks.

For this purpose you need:

A Generator
A Discriminator

See the scheme (credit O'Reilly):

The "Real Images" block in the scheme should be your training dataset (or ground truth). The Generator should generate the distorted words and the Discriminator should verify if the word is "adequately" distorted, based on a criterion of your choice, which can be any similarity measure between known words (database) and the generated one. Both the Generator and the Discriminator get trained on-the-go while in the training phase and in the end you will have two trained networks, of which the Generator would be very useful for your purpose.

Helpful source: https://deeplearning4j.org/generative-adversarial-network


Generating Polynomial Features in R Is there an optimized way to perform this function "PolynomialFeatures" in R? I'm interested in creating a matrix of polynomial features i.e. interactions between two columns among all columns but I can't find a base function or a package that does this optimally in R and I don't want to import data from a Python script using sklearn's PolynomialFeatures function into R.

The idea is to transform this:

array([[0, 1],
       [2, 3],
       [4, 5]])

into this:

array([[ 1.,  0.,  1.,  0.,  0.,  1.],
       [ 1.,  2.,  3.,  4.,  6.,  9.],
       [ 1.,  4.,  5., 16., 20., 25.]])

Intercept and only-interaction options are well, optional. I could expand-omit those myself.

You can use two methods: one that takes a formula argument or if you are not using formula you can use model.matrix.

You can do the following:

# Using the same data from the question:
data <- data.table(A = c(0, 2, 4),
                   B = c(1, 3, 5))

> data

   A B
1: 0 1
2: 2 3
3: 4 5

If only the raw data and the interactions are required

formula = y ~ .^2

model.matrix(formula, data=data)
  (Intercept) A B A:B
1           1 0 1   0
2           1 2 3   6
3           1 4 5  20

If raw data, squared columns, interactions and intercept are needed, just like the question

You can use the following formula for all variables:

formula = y ~ .^2 + poly(var, 2, raw=TRUE)[, 2] ... etc

In addition you can paste the variables names automatically. Formula based on this post

formula <- as.formula(paste(' ~ .^2 + ',paste('poly(',colnames(data),',2, raw=TRUE)[, 2]',collapse = ' + ')))

> formula

~.^2 + poly(A, 2, raw = TRUE)[, 2] + poly(B, 2, raw = TRUE)[, 2]

> model.matrix(formula, data=data)

  (Intercept) A B poly(A, 2, raw = TRUE)[, 2] poly(B, 2, raw = TRUE)[, 2] A:B
1           1 0 1                           0                           1   0
2           1 2 3                           4                           9   6
3           1 4 5                          16                          25  20

The resulting dataframe is identical to the array in the question.

If squared columns, interactions and intercept are needed.

> formula <- as.formula(paste(' ~ A:B + ',paste('poly(',colnames(data),',2, raw=TRUE)[, 2]',collapse = ' + ')))

> formula
~A:B + poly(A, 2, raw = TRUE)[, 2] + poly(B, 2, raw = TRUE)[, 2]

>  model.matrix(formula, data=data)
  (Intercept) poly(A, 2, raw = TRUE)[, 2] poly(B, 2, raw = TRUE)[, 2] A:B
1           1                           0                           1   0
2           1                           4                           9   6
3           1                          16                          25  20


The Why Behind Sum of Squared Errors in a Linear Regression I'm just starting to learn about linear regressions and was wondering why it is that we opt to minimize the sum of squared errors. I understand the squaring helps us balance positive and negative individual errors (so say e1 = -2 and e2 = 4, we'd consider them as both regular distances of 2 and 4 respectively before squaring them), however, I wonder why we don't deal with minimizing the absolute value rather than the squares. If you square it, e2 has a relatively higher individual contribution to minimize than e1 compared to just the absolutely values (and do we want that?). I also wonder about decimal values. For instance, say we have e1 = 0.5 and e2 = 1.05, e1 will be weighted less when squared because 0.25 is less than 0.5 and e2 will be weighted more.   Lastly, there is the case of e1 = 0.5 and e2 = 0.2. E1 is further away to start, but when you square it 0.25 is compared with 0.4. Anyway, just wondering why we do sum of squares Erie minimization rather than absolute value.

Simple google search on "stats why regression not absolute difference" would give you good answers. Try it yourself!

I can quickly summarise:

Your regression parameters are solutions to the maximum likelihood optimisation. That involves derivative, but absolute difference doesn't have a derivative at zero. There's no unique solution for least absolute regression.
Least absolute regression is an alternative to the regular sum of squares regression, commonly classified as one of the robust statistical methods.
You'd prefer least absolute regression if you care about outliers, otherwise the regular regression is generally better.

You might want to read about L1 vs L2:

  https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models


How to best use geographical information as a factor? I am trying to predict crime rates and I have naively used lat and long as two separate factors (which seem to work well!). Are there any best practices for location as a factor?

If you are predicting crime rates in a certain region, we may use clustering to deduce useful information. In clustering, basically, we will try to group similar data points together and treat them as a single class. We can understand this with an instance.

We have various points ( Latitude and Longitude ) and each of them represents a certain type of crime. Even by mere observation, we can conclude that some specific types of crime occur in a particular region only. Basically, we are going to cluster such points which are in the vicinity of each other and belong to the same class ( kind ).

  For example, an emergency call from an area ( with more cases of robbery ) arrives, the probability that the victim has also suffered from the robbery is more than any other crimes.

As we get more data, we can retrain our clustering algorithms to make more clusters and thereby increase efficiency.


Heatmap on a map in Python Mode Analytics has a nice heatmap feature, but it is not conducive to comparing maps (only one per report).

What they do allow is data to be pulled easily into a wrapped python notebook.  And then any image in python can easily be added to a report.
So my question is: how do I recreate a heatmap on an actual map in Python?  I've checked out follium and plotly, but neither seem to have similar functionality.

Looks like the gmaps package is what you're looking for.
You can do things like this with it:


Need help with method="leapSeq" Here is my code:

ctrl <- trainControl(method = "cv")
fm <- train(y=train.df$Price,x=train.df[,-1], method = "leapSeq", trControl = ctrl)
fm$results
coef(fm$finalModel)

What I get back is this:

      nvmax RMSE Rsquared  MAE RMSESD RsquaredSD MAESD
    1     2 1888    0.760 1315    266     0.0703   130
    2     3 1530    0.842 1171    221     0.0458   157
    3     4 1707    0.809 1231    244     0.0315   132
> coef(fm$finalModel)
Error in coef.regsubsets(fm$finalModel) :
  argument "id" is missing, with no default

There is no Coefficients tab under finalModel.  If I use "lm" for method instead of "leapSeq" the coefficients tab is there in finalModel.

Any guidance is appreciated.

you can print the coefficients with:
coef(fm$finalModel,model No. e.g.1)


If Deep learning or Machine learning is blackbox then why companies are still investing? I was wondering whether there is mathematical evidence or proof of functions that are happening at the backend of deep learning/machine learning. Particularly in training and testing operations. Secondly, deep learning is invented many years ago but still very minimal hardware systems to do something real. Is this a dark secret of artificial intelligence?

You make a huge assumption that because something is a black box, that it is not valid. Almost every decision made by a human is essentially a black box decision. Most could tell you the important factors in a decision, but they can rarely quantify accurately, how each component factored into the decision.

Another problem with your question is that you assume all Machine Learning techniques are black box.  For example, a Decision Tree is indisputably a Machine Learning technique, but it is definitely not a black box algorithm.

That said, in some applications, a black box technique is not desirable. For example, in the pharmaceutical field and credit approvals, you want to be able to explain the mechanics of the model.  But in other applications, black box algorithms trained on data generated by experts may perform nearly as well as experts, and probably better than the average human in a given task. For example, in the field of medical diagnosis from imaging, black box algorithms are enjoying success and they can be built right into the imaging software.  This means that the imaging machine can 'refer' the most challenging instances to a doctor for further diagnosis.  This means a doctor can spend more time with the most challenging patients and this is the real reason companies are investing in all manner of Machine Learning applications; they gain some economic benefit that outweigh the costs.

By all means, there are many failures of Machine Learning, but there are so many examples of Machine Learning that have been deployed and nothing is ever really mentioned about it.  For example, my iPhone employs facial recognition to help me search for people in my photos.  With minimal input from me, my phone has tagged, pretty accurately, all the people I typically take photographs of (mostly family).  I think the key is that we are seeing Machine Learning going mainstream to the point that it is not even really acknowledged anymore.


train_test_split : stratify can not be recognized? I'm trying to set stratify = True, but somehow jupyter notebook says 'name 'y' is not defined'

X_train, X_test, y_train, y_test = train_test_split(
    loan.drop('Loan_Status', axis=1),
    loan['Loan_Status'], test_size=0.2, random_state=0, stratify=y)

Can anyone tell me what is the proper way to do it?

I'm using Scikit-learn v0.19.1 and have tried to set stratify = True / y / 2 but none of them worked.

You need to pass an array containing the class-labels (or whatever the criterion for stratifying is) as an argument to stratify.

In your case, the answer is probably loan['Loan_Status'].values.


Algorithms for text clustering I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings.
What algorithms are suggested to do this? I don't know number of clusters in advance (and as more data is coming clusters can change as well), what features are normally used to represent each sentence?
I'm trying now the simplest features with just list of words and distance between sentences defined as:
$|A \cup B$ \ $A \cap B|$/$|A \cup B|$
(A and B are corresponding sets of words in sentence A and B)
Does it make sense at all?
I'm trying to apply Mean-Shift algorithm from scikit library to this distance, as it does not require number of clusters in advance.
If anyone will advise better methods/approaches for the problem - it will be very much appreciated as I'm still new to the topic.

Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in commercial products, you have to acquire commercial license.
Another interesting set of open source libraries, IMHO suitable for this task and much more, is parallel framework for machine learning GraphLab, which includes clustering library, implementing various clustering algorithms. It is especially suitable for very large volume of data (like you have), as it implements MapReduce model and, thus, supports multicore and multiprocessor parallel processing.
You most likely are aware of the following, but I will mention it just in case. Natural Language Toolkit (NLTK) for Python contains modules for clustering/classifying/categorizing text. Check the relevant chapter in the NLTK Book.
UPDATE:
Speaking of algorithms, it seems that you've tried most of the ones from scikit-learn, such as illustrated in this topic extraction example. However, you may find useful other libraries, which implement a wide variety of clustering algorithms, including Non-Negative Matrix Factorization (NMF). One of such libraries is Python Matrix Factorization (PyMF). Another, even more interesting, library, also Python-based, is NIMFA, which implements various NMF algorithms. Here's a research paper, describing NIMFA. Here's an example from its documentation, which presents the solution for very similar text processing problem of topic clustering.


How can I reduce the number of dimensions using a Clustering algorithm in a mixed dataset? I am working with a mixed data set, corresponding to TV consumption data, with the aim of reducing the number of features to only those relevant to detect TV consumption patterns (or consumption groups) using clustering.
The dataset is composed of about 20 dimensions and 2.000.000 samples for 1 day of consumption. I have access to the data of up to 3 years of consumption, so I can exploit up to ~1 billion data. My idea is to start working with only a few days of consumption (and therefore a few million data).
3 dimensions are of continuous/numerical type (the date-time of consumption, the duration...) and the remaining dimensions are of discrete/categorical type, with features with binary options (e.g. whether the programme is live or not) or with multiple and even hundreds of options (e.g. name of the programme, theme, type of device, etc). For this reason, I am trying to implement a clustering algorithm with Python that can deal with mixed data.
Since I suspect that there are many dimensions that might be irrelevant to my study, I would like to reduce the number of features. To do so, I have thought of applying some clustering algorithm and check that, by removing a given feature, the clustering results are not affected.
However, I don't know what metric I should calculate or use to evaluate the clustering results and the effect of removing one of the attributes.
At the moment I have applied the K-prototypes algorithm which is based on K-means but for mixed data. It is easy to implement in Python (https://antonsruberts.github.io/kproto-audience/). However, it only returns the cluster labels, the centroid coordinates and the cost (defined as the sum of the distance of all points to their respective centroids). Therefore, I do not know how to interpret the results or how to study the effect of the features.
I would like to know if my approach to the problem is correct and what metric I should use to evaluate the clustering results and reduce the number of dimensions, as well as if there are other easily implemented algorithms in Python (for clustering or unsupervised feature selection) capable of dealing with the type of data I have.

Some common techniques to reduce number of features:

Missing Values Ratio. Data columns with too many missing values are unlikely to carry much useful information. Thus data columns with number of missing values greater than a given threshold can be removed. The higher the threshold, the more aggressive the reduction.

Low Variance Filter. Similarly to the previous technique, data columns with little changes in the data carry little information. Thus all data columns with variance lower than a given threshold are removed. A word of caution: variance is range dependent; therefore normalization is required before applying this technique.

Random Forests / Ensemble Trees. Decision Tree Ensembles, also referred to as random forests, are useful for feature selection in addition to being effective classifiers. Personally I will prefer this method as it is easy to implement. Example use is given in the kernel -->
https://www.kaggle.com/prashant111/xgboost-k-fold-cv-feature-importance?scriptVersionId=48823316&cellId=74

High Correlation Filter. Data columns with very similar trends are also likely to carry very similar information. In this case, only one of them will suffice to feed the machine learning model. Here we calculate the correlation coefficient between numerical columns and between nominal columns as the Pearson’s Product Moment Coefficient and the Pearson’s chi square value respectively. Pairs of columns with correlation coefficient higher than a threshold are reduced to only one. A word of caution: correlation is scale sensitive; therefore column normalization is required for a meaningful correlation comparison.

Refer : https://www.kdnuggets.com/2015/05/7-methods-data-dimensionality-reduction.html


What is the efficiency difference between different cost functions in case of neural networks? I'm studying the theory behind neural nets and I wonder if there is any actual difference between using different lost/cost functions?

Let's say I could use either MAE or MSE for backpropagating loss; does this decision have any actual effect on the model efficiency?

At the end of the day both functions just calculate the difference between $y$ and $\hat y$ (albeit on a different scale). But for the optimizer, only the error tendency is what really matters, not the absolute difference.

Of course, this question is relevant for any other model evaluation.

Think of it in terms of navigating a landscape.  The land you move across is created by your error function (that is, the realtion between your model and your data), and the way you move across the landscape is your training function.  Especially in Neural Networks, falling in 'local minima' is a big problem.  In fact, Neural Networks have been theoretially able to approximate any function for... 20 years?  See Universal Approximation Theorem.  But the problem has always been learning how to train a given network.

More specifically, in your case comaring MAE and MSE, the difference is in the 'square' part.  The steepness of the 'fitness landscape' will be much steeper where error is greater with MSE compared to MAE.  1*1 = 1.  i.e., squaring does nothing at unit-error.  10 * 10 = 100.  Absolute error of 10 will become a squared error of 100.  That is looking at one sample, and has huge implications around falling in local minima where the error is great.

Looking across the whole dataset, another huge implication is how the NN balances the error it sees.  With MSE, samples with high error become samples with very high error.  Thus, a NN trained with MSE will 'care more' about learning samples that are very wrong at the expense of small improvements to 'easier' samples.


What is different between R2 and mean of R2 in multiclassification probelm? Which one is correct? I have a question. I have a big dataset (unfortunately confidential).
What I did?
I have trained my model with Naive-Bayes.
BRNBReg=BernoulliNB(alpha=0.01, binarize=0.0, fit_prior=True, class_prior=None)
BRNBReg.fit(x_train,y_train)

#CrossValidation
cv_BRNBReg_score=cross_val_score(BRNBReg,x_train,y_train,cv=9)
cv_BRNBReg_pred=cross_val_predict(BRNBReg,x_train,y_train,cv=9)

print("R2-Socre-Mean: ",cv_BRNBReg_score.mean())
print("Score: ",r2_score(y_train,cv_BRNBReg_pred))

## Prediction
BRNBReg_pred=BRNBReg.predict(x_test)
print("Score: ",r2_score(y_test,BRNBReg_pred))

My Score with CV is:
R2-Socre-Mean:  0.908198797087686
Score:  0.8673184920974637

My problem:
My Score with test-data is:
Score:  0.6244917483855538

the difference is huge! But I don't know why?

Is my model overfit\underfit?
Shall i tune my hyperparams?

First of all, if you are in a classification problem as you said, R2 score is not good, it should be used for regression problem. For classification problem you have to use something like accuracy, precision, recall and F1 score.
First question
Anyway, I will answer your first question about R2score:
R2 mean: you are calculating the R2 for each split you have in your CV(train the model with training data, and evaluating it in test data). And then, you calculate the mean of each value of the split. In your case, you will have 9 values of R2score since you have 9 CV. And the result is the meaning of these values.
r2_score: you are evaluating the y_true and y_pred. You already have the y_true. To obtain the y_pred you used cross_val_predict. What you do, is calculate the predicted value using only the CV where you have the y_true in you test split.
I would recommend you use R2 mean, it's more robust score in my opinion
I recommend read this post to understand it in more detail.
Second questions
First of all, take care of what I mention: r2 is not a good score for classification problem. But let's suppose that you use another score and you have similar score outputs.

Is my model overfit\underfit?

Yes, it is. Basically, if your training accuracy is much bigger than your test data, you are overfitting. Your scores must be quite similar. What you have now, is a model that is "memorizing" and not learning. Be careful.

Shall I tune my hyperparams?

Yes, you have to try tunning your hyperparameters in order to achieve similar score between training data and test data. You should check how the model works changing hyperparameters and decided which helps you to avoid overfitting. You can do it manually or using GridSearchCV for example.


TensorFlow: trainable_variables() is empty I want to retrieve the list of trainable variables/weights in my model (wrapped in a tf.Estimator). However, tf.trainable_variables always returns an empty list, what am I doing wrong?

import numpy as np
import pandas as pd
import sys
import globals
import pkg_resources
import numpy as np

import tensorflow as tf

def cnn_model_fn(features, labels, mode):

    input_layer = tf.reshape(features["x"], [-1, 51, 13])
    input_layer = tf.cast(input_layer, tf.float32)

    # Convolutional Layer #1
    conv1 = tf.layers.conv1d(inputs=input_layer, filters=32, kernel_size=5, padding="same", activation=tf.nn.relu)

    # Pooling Layer #1
    pool1 = tf.layers.max_pooling1d(inputs=conv1, pool_size=2, strides=2)

    # Convolutional Layer #2 and Pooling Layer #2
    conv2 = tf.layers.conv1d(inputs=pool1, filters=64, kernel_size=5, padding="same", activation=tf.nn.relu)

    # Pooling layer #2
    pool2 = tf.layers.max_pooling1d(inputs=conv2, pool_size=2, strides=2)

    # flatten the feature map
    pool2_flat = tf.reshape(pool2, [-1, 12 * 64])

    # Dense Layer
    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=mode == tf.estimator.ModeKeys.TRAIN)

    # Logits Layer
    logits = tf.layers.dense(inputs=dropout, units=5)

    predictions = {
        # Generate predictions (for PREDICT and EVAL mode)
        "classes": tf.argmax(input=logits, axis=1),
        # Add `softmax_tensor` to the graph. It is used for PREDICT and by the `logging_hook`.
        "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
    }

    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)

    # Calculate Loss (for both TRAIN and EVAL modes)
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)

    # Configure the Training Op (for TRAIN mode)
    if mode == tf.estimator.ModeKeys.TRAIN:
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)

    # Add evaluation metrics (for EVAL mode)
    eval_metric_ops = {"accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])}

    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)

def main(unused_argv):

    total = pd.read_feather('testfile.feather')
    labels = total['labels']
    features = total.iloc[:, 16:679]

    mnist_classifier = tf.estimator.Estimator(model_fn=cnn_model_fn, model_dir="/tmp/gait_convnet_model")

    # Log the values in the "Softmax" tensor with label "probabilities"
    tensors_to_log = {"probabilities": "softmax_tensor"}
    logging_hook = tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=50)

    # Train the model
    train_input_fn = tf.estimator.inputs.numpy_input_fn(x={"x": np.array(features)}, y=np.array(labels), batch_size=100, num_epochs=None, shuffle=True)
    mnist_classifier.train(input_fn=train_input_fn, steps=1, hooks=[logging_hook])

    temp_list = tf.trainable_variables()
    print(temp_list)

if __name__ == '__main__':
    tf.app.run()

There are functions in the Estimator class that handle this, namely: get_variable_names and get_variable_value.

I guess they want to avoid polluting the global scope. Note that you can pass a name parameter to your layers on creation to disambiguate them.


Detecting address labels using Tensorflow Object Detection API I am experimenting with the Tensorflow Object Detection API on a Windows 7 machine.  I am trying to detect US address labels (and similar blocks of text) as they appear on a piece of mail or an envelope. I am not trying to detect individual words or lines, but rather the full rectangular block of text. My address labels are typically isolated on the letter or envelope and they are surrounded by whitespace. For example:

I followed the tutorial to train a customer object detector. I used the pre-trained SSD Inception V2 COCO model along with 50 images of letters/envelopes containing address labels that were annotated with LabelImg. To annotate my objects (address labels), I drew the bounding box around the entire label with about 5-15 pixels of padding. After 200 training steps, I reached a loss of 2.5 and stopped the training.

Using the same tutorial, I obtained the trained inference graph. Finally, I adapted this tutorial for doing inference. I tested two images, once containing a dog and the other containing an address label. The dog was detected, but nothing was detected in my image with the address label was detected.

My questions are:

Is it reasonable to expect to detect the full block of text given it is surrounded by whitespace and not solid edges?
Am I annotating properly? I left about 5-15 pixels of padding between the address text and the bounding box when I annotated with LabelImg.
Do I have enough images for one-class detection in which the address labels vary little from letter to letter
Should I let the training go longer?

It is difficult to say whether the algorithm will detect full box of text or not. This is kind of difficult problem because object to detect does not have proper structure. I might be wrong here!

You are labeling correctly.

You have very few images to learn from. More images are always better for learning tasks.

Don't look at loss value which appears in the terminal, instated check total loss in TensorBoard (I am sure this loss will be more).
For how long to train? in the example config file from TensorFlow API they say on pets dataset (which has only 7349 images) you need to train model for 200000 steps to get satisfactory results. If this problem is learnable, training for more steps should work.


What is the difference between Gradient Descent and Stochastic Gradient Descent? What is the difference between Gradient Descent and Stochastic Gradient Descent?

I am not very familiar with these, can you describe the difference with a short example?

For a quick simple explanation:
In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function.
While in GD, you have to run through ALL the samples in your training set to do a single update for a parameter in a particular iteration, in SGD, on the other hand, you use ONLY ONE or SUBSET of training sample from your training set to do the update for a parameter in a particular iteration. If you use SUBSET, it is called Minibatch Stochastic gradient Descent.
Thus, if the number of training samples are large, in fact very large, then using gradient descent may take too long because in every iteration when you are updating the values of the parameters, you are running through the complete training set. On the other hand, using SGD will be faster because you use only one training sample and it starts improving itself right away from the first sample.
SGD often converges much faster compared to GD but the error function is not as well minimized as in the case of GD. Often in most cases, the close approximation that you get in SGD for the parameter values are enough because they reach the optimal values and keep oscillating there.
If you need an example of this with a practical case, check Andrew NG's notes here where he clearly shows you the steps involved in both the cases. cs229-notes
Source: Quora Thread


GPU performance is about 50% slower than benchmarks Running this benchmark I get 50% slower performance than the author on practically all deep learning sub problems (SINGLE precision and on TRAINING only):

I tested this on a GeForce 1080 GTX Ti and am using PyTorch 1.0.1 with Cuda 10.0.13 (cudnn version 7402) running on an AMD Ryzen 7 1700 Eight-Core Processor (with 16 threads). OS is arch linux. The GPU temperature remains below 80 degrees at full volatile usage.

Would anyone have an idea what causes the lower performance? Can someone else with the same gfx run this and let me know what differences they see?

My guess is that the reduction in performance is due to differences in versions PyTorch. The published benchmark uses torch==0.4.1 (even though the README states something different). You are using PyTorch 1.0.1.

PyTorch 1.0 speed is sometimes slower than lower than 0.4. PyTorch 1.0 can be speed-up by adding torch.backends.cudnn.benchmark = True and compiling from source.

Often times published benchmark are heavily tuned from the "off-the-shelf" version.


feature importance between two data distributions I have two textual datasets collected from different domains (Twitter and Reddit).

I extracted a set of features in the same way from these two datasets, one of these features as an example called X_positive.

How could I know if the feature X_positive is more representative (exists more) in one of the datasets?

My two cents:
I can think of several ways calculating specific terms repressiveness:

1) Calculating rate of documents(tweets/posts) where these terms appear more than
1(or X > 1) times. You can decide on X by looking at the distribution of frequency per document.
2) Calculating terms TF-IDF score per each document and aggregating across all documents. Aggregating across all document  depends on your need and what you mean by representative.
3) Calculating terms appearance rate - how many times these terms appear divided by total number of words in the corpus(possibly after subtracting stop words).


Is the output size of the last layer of a standard fully connected neural network the same as the input size? Let's say I have a neural net with Dense layers. The input layer has 3 neurons, the single hidden layer has 5 neurons, and the final output layer has 2 neurons.
For layer 1, 3 inputs go in and 5 inputs go out. For layer 2, the 5 inputs from layer 1 go in and 2 inputs go out. So for layer 3, would 2 inputs go in and 2 inputs come out?

In a Dense NN,

Input to a Layer depends on the output of the previous layer and its Neuron count
Previous Layer for the first layer is the Input Data Features
Output of a Layer is equal to the Layer's neuron count. A copy of each goes to all the Neuron of the Next Layer

It means for your case,
- First Layer will have *M(Input features)3 inputs going in i.e. M to each Neuron and 3 coming out.
 - The Second layer will have *3(Previous Layer)5 inputs going in and 5 coming out
 - The Last layer will have 5(Previous Layer)*2 inputs going in and 2 coming out
A Neural Network is a complex Tensor operation. Arrow and Circle are logical representations. Each arrow got a weight and each Circle got the Activation function and the Bias term. So, you may count this way too.
$\hspace{3cm}$
 Validate this with Keras Model.summary result which shows the parameter count.


What model can I build with a limited dataset? I have a dataset consisting of purchasing history from an e-commerce website. The columns consist of customer id, product id, postal code, quantity of products purchased, date of the purchase. There are thousands of different customer ids, hundreds of product ids, and some million rows.

I ran ARIMA modelling for forecasting the purchased quantities of a given product.
Now I want to try other methods for analysing the dataset, but cannot find models which could fit the dataset well. What other models could I run in order to gain more insight on this data?

Based on your dataset you can also try to create a product recommendation model so that given a new customer that just purchased a particular item you can predict/suggest or recommend what other items they could purchase next during their shopping trip or after:
- Collaborative filtering can be used to achieve this
- Frequent pattern mining can also be used.


Reducing the dimensionality of word embeddings I trained word embeddings with 300 dimensions. Now, I would like to have word embeddings with 50 dimensions: is it better to retrain the word embeddings with 50 dimensions, or can I use some dimensionality reduction method to scale the word embeddings with 300 dimensions down to 50 dimensions?

There is a paper on this subject calle

  Simple and Effective Dimensionality Reduction for Word Embeddings, Vikas Raunak

You can read it here

You can also find the implementation here

In my opinion it works quite well


Why should we use (or not) dropout on the input layer? People generally avoid using dropout at the input layer itself. But wouldn't it be better to use it?

Adding dropout (given that it's randomized it will probably end up acting like another regularizer) should make the model more robust. It will make it more independent of a given set of features, which matter always, and let the NN find other patterns too, and then the model generalizes better even though we might be missing some important features, but that's randomly decided per epoch.

Is this an incorrect interpretation? What am I missing?

Isn't this equivalent to what we generally do by removing features one by one and then rebuilding the non-NN-based model to see the importance of it?

Why not, because the risks outweigh the benefits.

It might work in images, where loss of pixels / voxels could be somewhat "reconstructed" by other layers, also pixel/voxel loss is somewhat common in image processing. But if you use it on other problems like NLP or tabular data, dropping columns of data randomly won't improve performance and you will risk losing important information randomly. It's like running a lottery to throw away data and hope other layers can reconstruct the data.

In the case of NLP you might be throwing away important key words or in the case of tabular data, you might be throwing away data that cannot be replicated anyway else, like gens in a genome, numeric or factors in a table, etc.

I guess this could work if you are using an input-dropout-hidden layer model as you described as part of a larger ensemble though, so that the model focuses on other, less evident features of the data. However, in theory, this is already achieved by dropout after hidden layers.


Improve Pandas dataframe filtering speed I have a dataset with 19 columns and about 250k rows. I have worked with bigger datasets, but this time, Pandas decided to play with my nerves.

I tried to split the original dataset into 3 sub-dataframes based on some simple rules. However, it takes a long time to execute the code. About 15-20 seconds just for the filtering.

Any alternative way that will improve the performance of the code?

import pandas as pd

#read dataset
df = pd.read_csv('myData.csv')

#create a dataframe with col1 10 and col2 <= 15
df1 = df[(df.col1 == 10) & (df.col2 <= 15)]
df = df[~df.isin(df1)].dropna()

#create a dataframe with col3 7 and col4 >= 4
df2 = df[(df.col3 == 7) & (df.col4 >= 4)]
df = df[~df.isin(df2)].dropna()

In the end, I have the df1, df2, df dataframes with the filtered data.

The concept to understand is that the conditional is actually a vector.  So, you can simply define the conditions, and then combine them logically, like:

condition1 = (df.col1 == 10) & (df.col2 <= 15)
condition2 = (df.col3 == 7) & (df.col4 >= 4)

# at this point, condition1 and condition2 are vectors of bools

df1 = df[condition1]
df2 = df[condition2 & ~condition1]
df = df[~ (condition1 | condition2)]

This will be considerable faster as it only evaluates the conditional once.  Then it uses them to perform indexed lookup to create the new smaller dataframes.


How to deal with multiple categorical data set Please tell me how with sex, smoker, region?
Should I perform one hot encoder for all?

Simply yes. Before that you may want to check how correlated those features are, so you can simply deselect redundant features, but in general you are right. Starting with one-hot encoding is a good choice.

What may need more inspection later, is the number of different regions. Then you come up with many sparse features for which you need to reduce dimensionality. If you provide more information on the whole project I can provide more insight.


Normalized 2D tensor values are not in range 0-1 Below function takes in 2D tensor and normalizes it using broadcasting .The issue is except all values to be in range 0-1 but the result has values outside this range . How to get all values in 2D tensor in range 0-1
def torch_normalize(tensor_list):
    means = tensor_list.mean(dim=1, keepdim=True)
    stds = tensor_list.std(dim=1, keepdim=True)
    normalized_data = (tensor_list - means) / stds
    return normalized_data

INPUT
tensor_list=tensor([[-5.6839, -7.5829, -7.2277, -6.5066, -8.4702, -7.9844, -5.6841,  1.8570,
          1.6170, -3.7592, -4.4140, -0.4981,  0.2501,  5.8463,  1.8897, -1.3968,
         -5.5402, -2.4561, -5.6819]])

Normalized result

tensor([[-0.5981, -1.0615, -0.9748, -0.7988, -1.2780, -1.1594, -0.5981,  1.2420,
          1.1835, -0.1284, -0.2882,  0.6673,  0.8499,  2.2155,  1.2500,  0.4480,
         -0.5630,  0.1896, -0.5976]])

The normalisation you do does not re-scale to $[0,1]$ range! It normalises to have mean $0$ and std $1$ instead.
To scale the tensor to be in $[0,1]$ range you should subtract min value and divide by absolute max-min value.


Python pandas index error Encountred an index error when trying to run the code found here.
The error occurs in section In[10]:
for n in range(len(dominantTheta)):
    shift = dominantTheta[n]
    regressionDelta += dominantAmp[n] * np.cos(n * np.array(range(len(df))) + shift)

Error message:
Traceback (most recent call last):
  File "C:/Users/xxx/Desktop/Python Workspace/DSP.py", line 62, in <module>
    shift = dominantTheta[n]
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\series.py", line 868, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\indexes\base.py", line 4375, in get_value
    tz=getattr(series.dtype, 'tz', None))
  File "pandas\_libs\index.pyx", line 81, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 89, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 987, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 993, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 0

I have the latest versions of numpy and pandas installed, and using Python 3.6

In that section, within the for-loop, n is an integer (you loop over a range().

If you use the square brackets to access the Pandas DataFrame (or a Series, in this case), you are trying to access the index of the table. In this case, the index is a timestamp - have a look at the df in section Out [7].

So you need to get values by timestamp, or to use the row number itself, you must do as Vaalizaadeh said in a comment and use the .iloc accessor to the pandas object.

So just change shift = dominantTheta[n] to this: shift = dominantTheta.iloc[n].


Does testing a training dataset guarantee successful results? If I test an image that has been previously used to train a classification model, is it guaranteed to classify correctly?
My guess is that since the parameters have been trained with other images as well, there is no guarantee of getting a correct classification, just a high probability.

This is correct, there's no guarantee at all, not even a high probability. As usual It depends on the type of model, the data, the number and distribution of the classes.
However there's of course a higher chance that the instance would be correctly classified. That's why one shouldn't use a test set containing training instances to estimate the performance of the model, since there's a high risk the performance would be overestimated (data leakage).


Use correlation matrix scores as starting weights and bias inputs for neural network? I have a neural net that's generating an average 15% error across the three outputs it gives. My problem is two of the three really have about a 2% error while the third has around 40%.

I was wondering if anyone has used results from a correlation matrix on the raw data to set starting parameters for what features are valued more relative to the labels. If this is possible, or logical, how would you initialize this for two input features if your model's first layer has say eight nodes - or at least more than two? I'm using Keras btw, but even a theoretical explanation would be helpful.

If it doesn't make sense to follow this path, does anyone have recommendations at correctly the imbalance?

The way I see it is that NN can learn nonlinear features, so I wonder: does your correlation matrix on the raw data take into account nonlinear connections between features to see the full picture? Probably, it doesn't. As the topic is not about the single-layer nn (which would be an example of linear regression), the final connections (values of each weight) will be created by the contribution of many layers. Therefor it seems to me, that setting start weights on the proposed way wouldn't be high relevant.

Speaking about imbalancing, I'd recommend to experiment with loss functions. You can set higher penalty for mistakes on a certain class.


How to filter in pandas based on string value? I want to filter a pandas dataframe based on the id of the row. The id is a sequence of patient names. For example, I have columns in this format:
 ID | A | B

And the id values looks like TCGA-A6-6653-01Z-00-DX1.e130666d-2681-4382-9e7a-4a4d27cb77a4 but multiple repeating ones. I want to group the unique ids and do a mean operation on A and B.
Can anyone guide me?   Thanks

You can just groupby id and do mean for the remaining columns.
df.groupby('id').mean()