Is conditional GAN supervised learning? I am trying to understand this paper about conditional GAN, it says that extra information y (class labels) is given to the network. However, I cannot understand its usage during training or its benefits. As far as I know, GAN is unsupervised learning, in this case of extra information usage, can we say that the architecture is supervised?

I'm assuming you understand the original gan paper.

So there are 2 distribution at the start - first distribution that the original images follow and a random distribution that the fake images follow.

Discriminator task is to figure out which image came from what distribution, whereas generator is trying to learn the real distribution and make the random distribution similar to the real one.

Now giving image labels as input is like giving extra bit of information about the distribution. This doesn't change the game, it's the same unsupervised one.

There will two consequences of adding this extra bit of information :

Even the random distribution that the fake images follow will have some pattern. Hence convergence will be faster.
You can control the output of generator at test time by giving label for image you want to generate.


How can ReLU ever fit the curve of x²? As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations:

Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ being the activation function).
Recursive substitution $f(f(x))$.

Now with the above transformations a ReLU activation function should never be able to fit a x² curve. It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?

Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?

I am new to machine learning. So please pardon me if there are any blunders in anything I am assuming.

It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?

You are right, the neural network uses ReLu functions to approximate the output $f(x) = x^2$ and there is some error incurred, but you are forgetting that

Non-linear functions also do approximation since we use only some subset of activation functions and incur some error which can be as huge as with ReLus
There is also a number of neurons to consider. The more neurons/layers you have - the better the approximation can be (since more small-interval ReLus are fit to better approach the shape of $x^2$). If your input size grows, you should also consider increasing the number of neurons/layers, in this way the approximation will always be close enough.

Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?

The main reason for this is that even though other activation functions can approximate $x^2$ or something else better, they are not optimised as fast. Generally speaking, non-linear activation function decision surface is more complex than the loss created by ReLus and can contain better global minimum but this minimum is harder to find, and, thus, with the less sophisticated loss created by ReLus, we can find a better optima using some gradient descent procedure, and it will be much faster.


size of deep learning object detection pre-train models I am benchmarking five different models,

YOLO v2
YOLO v3
Cascade R-CNN
Faster R-CNN
Retina Net

In my comparison, I would like to know what the sizes of each model, to discuss the potential overfitting (if some are larger than others). It can be either the number of layers, the number of parameters. For Yolo models these are stated in the paper, e.g  YOLOv2 has 106 layers with 51,000,657 parameters, whereas YOLOv3 has 349 layers and 65,252,682 parameters.

I looked into the other models, but couldn't find similar numbers in their own papers. Therefore, I appreciate if you help me to figure out whether those three are simpler than YOLOs or where they stand in terms of complexity.

If you are researching different architectures, I would recommend you to look at https://www.modelzoo.co/. It contains plenty of links to implementations in different architectures. This can help you get an intuitive understanding of how the models are built via code in case the papers aren't intuitive.

I use PyTorch more so I focused on finding PyTorch implementations. Here are links to Github repositories for the models you listed:

YOLO v2
YOLO v3
Cascade R-CNN
Faster R-CNN
Retina Net

With PyTorch networks, you can count the number of weights/parameters in multiple ways, this StackOverflow question has multiple answers all showing different ways to do it. The simplest one being:

pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

In the post, there are also ways which detail the size of each layer in the network.


Solving a system of equations with sparse data I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to solve for the set of 40 coefficients that minimizes the total sum-of-square error between y and the predicted value.

My problem is that the matrix is very sparse and I do not know the best way to solve the system of equations with sparse data. An example of the dataset is shown below:

   y    x1  x2 x3 x4 x5 x6 ... x40
87169   14  0  1  0  0  2  ... 0
46449   0   0  4  0  1  4  ... 12
846449  0   0  0  0  0  3  ... 0
....

I am currently using a Genetic Algorithm to solve this and the results are coming out
with roughly a factor of two difference between observed and expected.

Can anyone suggest different methods or techniques which are capable of solving a set of equations with sparse data.

If I understand you correctly, this is the case of multiple linear regression with sparse data (sparse regression). Assuming that, I hope you will find the following resources useful.

1) NCSU lecture slides on sparse regression with overview of algorithms, notes, formulas, graphics and references to literature: http://www.stat.ncsu.edu/people/zhou/courses/st810/notes/lect23sparse.pdf

2) R ecosystem offers many packages, useful for sparse regression analysis, including:

Matrix (http://cran.r-project.org/web/packages/Matrix)
SparseM (http://cran.r-project.org/web/packages/SparseM)
MatrixModels (http://cran.r-project.org/web/packages/MatrixModels)
glmnet (http://cran.r-project.org/web/packages/glmnet)
flare (http://cran.r-project.org/web/packages/flare)

3) A blog post with an example of sparse regression solution, based on SparseM: http://aleph-nought.blogspot.com/2012/03/multiple-linear-regression-with-sparse.html

4) A blog post on using sparse matrices in R, which includes a primer on using glmnet: http://www.johnmyleswhite.com/notebook/2011/10/31/using-sparse-matrices-in-r

5) More examples and some discussion on the topic can be found on StackOverflow: https://stackoverflow.com/questions/3169371/large-scale-regression-in-r-with-a-sparse-feature-matrix

UPDATE (based on your comment):

If you're trying to solve an LP problem with constraints, you may find this theoretical paper useful: http://web.stanford.edu/group/SOL/papers/gmsw84.pdf.

Also, check R package limSolve: http://cran.r-project.org/web/packages/limSolve. And, in general, check packages in CRAN Task View "Optimization and Mathematical Programming": http://cran.r-project.org/web/views/Optimization.html.

Finally, check the book "Using R for Numerical Analysis in Science and Engineering" (by Victor A. Bloomfield). It has a section on solving systems of equations, represented by sparse matrices (section 5.7, pages 99-104), which includes examples, based on some of the above-mentioned packages: http://books.google.com/books?id=9ph_AwAAQBAJ&pg=PA99&lpg=PA99&dq=r+limsolve+sparse+matrix&source=bl&ots=PHDE8nXljQ&sig=sPi4n5Wk0M02ywkubq7R7KD_b04&hl=en&sa=X&ei=FZjiU-ioIcjmsATGkYDAAg&ved=0CDUQ6AEwAw#v=onepage&q=r%20limsolve%20sparse%20matrix&f=false.


Use single Convolution layer for the classification of two kinds of matrix? I have two kinds of matrix with 10*10 size, and each number of matrix is 24000, all matrix forms like this:

0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
1 0 0 1 1 1 1 1 0 0
1 1 1 1 0 0 0 1 1 1
1 1 0 0 0 0 0 0 1 0
1 0 1 1 1 1 1 1 1 0
0 1 0 0 1 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0

and I must classify them with a single convolution layer.
the convolution kernel size is 5*5, the strides are 5*6, the final output is 2*1,the Neural network structure below:

X_input = Input(shape=input_shape)
X=Conv2D(1,kernel_size=(5,5),strides=(5,6),use_bias=None)(X_input)
X=Activation('relu')(X)
X=Flatten()(X)
Y=Activation('softmax')(X)
model=Model(inputs=X_input,outputs=Y,name="JSmodel")
return model

and the final accuarcy is 85.6%. However, I'm confused whether I design the neural network is accurate?

A Convolution neural network is not justified here.
CNN is used for images because -

Flattening the data can create very high dimensional features

Exerpt from "Hands-on Machine Learning" book -
For example, a 100 × 100–pixel image has 10,000 pixels, and if the first layer has just 1,000 neurons (which already severely restricts the amount of information transmitted to the next layer), this means a total of 10 million connections.
2. Nearby pixels tell a lot about the feature. So by using convolution, Pooling, we are able to reduce the dimension without losing information. The best part is that learning(Identifying the features) is part of Backpropagation
3. Weight sharing reduces the parm count further

 In the case mentioned in the question -

Dimension is not so large i.e. 10x10 = 100
Data is sparse and binary. I doubt it can represent spatial features. CNN is not good for sparse dataset.
Only one kernel

I doubt real CNN is happening in the background. You are getting one Feature map and then it is flattened to work as a normal Neural network.
So, it would be better to use a normal neural network
Before that, you should try SVM once as it works well on the sparse dataset. Though data-count is a bit large but can try once.


How to do predict a new sms to be spam or not? I have trained a model for spam classification -

This is my code -

X_train, X_test, y_train, y_test = train_test_split(data['text'], data['label'], test_size = 0.4, random_state = 1)

cv = CountVectorizer()
cv.fit(X_train)
cv.fit(X_test)
X_train = cv.transform(X_train)
X_test = cv.transform(X_test)

model = LogisticRegression(solver='lbfgs')
model.fit(X_train, y_train)

After that I had also completed testing, it gives me an accuracy of about 97%
.

Now I want to add predict a new SMS/Email to be spam or not. What I am doing is -

new = 'Hey there you got a sale here on website'
new = cleanText(new)
cv.fit([new])
new = cv.transform([new])
model.predict(new)

It gives me an error

  ValueError: X has 4 features per sample; expecting 4331

Please tell me where I am going wrong?

You are refiting the CountVectorizer during prediction. Just remove the line

cv.fit([new])

and it should work.


Categorical data in Kmeans I need to perform clustering in a given dataset.
There are atrributes with numerical as well as categorical values.

What is the best way to convert categorical to numeric value ?

as an example one field is colour and the values are red, green, blue
so can I assign a  mapping like:

red : 1, green : 2, blue : 3 or
red : 11, green : 25, blue : 30

and if I provide a mapping like this will this affect the Euclidian distance for clustering ?

or is there any other way ?

Converting the categorical data into numerical data isn't really meaningful. Different mappings will give you different solutions.

There is an extension of K-means algorithm for categorical data called k-modes. You can read about K-modes in detail here. This article explains the difference between K-modes and converting data into numeric vectors and then running K-means.


What is difference between Nearest Neighbor and KNN? I was taking the tutorial of making Recommendation system , there I read that Nearest Neighbor is different from KNN classifier . Could anyone explain that what is Nearest Neighbor and how it is different between KNN ?

Not really sure about it, but KNN means K-Nearest Neighbors to me, so both are the same. The K just corresponds to the number of nearest neighbours you take into account when classifying.
Maybe what you call Nearest Neighbor is a KNN with K = 1.


Clustering tendency using hopkin's statistics As per the references
1.   http://www.sthda.com/english/wiki/assessing-clustering-tendency-a-vital-issue-unsupervised-machine-learning
2.   http://www.listendata.com/2016/01/cluster-analysis-with-r.html

The Hopkins statistics value close to '0' is also clusterable. Is this correct? Or the value should always be close to '1' for clusterable?

I don't think the Hopkins statistic is useful for this purpose at all.

It is essentially a test for a uniform distribution.

But not having a uniform distribution does not mean the data is suitable for cluster analysis.

For example a single Gaussian distribution (unimodal!) will score high on this test, but the data doesn't have multiple clusters but all points are from the same distribution.


Multi-inputs Convolutional Neural Network takes different number of images I am using Keras to build a CNN model that takes two types of images as inputs (input1, input2) and produce one output. The model classifies the two inputs into a class, and the number of images in each type are different. (The number of images in the input1 is 1500 and in input2 is 4000).

This is my code:

in1 = Input(...)
 x = Conv2D(...)(in1)
# rest of the model
 out1 = Dense(...)(x)

 in2 = Input(...)
x = Conv2D(...)(in2)
# rest of the model
out2 = Dense(...)(x)

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])

Image generator code:

input_imgen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator()

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False,
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=1,
                                          img_height=224,
                                          img_width=224)

My questions are:

How can I determine the value of steps_per_epoch and validation_steps if the number of the images in each type are different?

In general, How to deal with the multi input Convolutional Neural Network that takes different number of images?

You need to define what your training set is in this case. If a single input to your network is a pair of images $(a, b)$, and you only have 1500 examples of $b$, then you only have 1500 training examples.

If each image in $A$ can map to any arbitrary image in $B$, you can potentially re-use images in $B$ to create these input pairs. If that's not the case, which images in $A$ correspond to the image of interest in $B$?

Alternatively, if you don't want to (or it's not appropriate to) simply re-use the images, you could consider augmenting the smaller set by, for example, rotating, cropping, or otherwise manipulating the images in that dataset.

To re-initialize the generator, you would just execute this code again :

genX2 = generator.flow_from_directory(dir2, target_size = (img_height,img_width), class_mode = 'categorical', batch_size = batch_size, shuffle=False, seed=7)

When genX2.next() doesn't return anything. Alternatively, and more pythonically, you could use itertools.chain() to form a generator capable of returning the right number of observations:

n_times_through = math.ceil(4000/1500)
genX2_new = itertools.chain(*([genX2] * n_times_through)


Extremely dominant feature? I'm new to datascience. I was wondering how one should treat an extremely dominant feature.

For example, one of the features is "on"/"off", and when it's "off", none of the other features matter and the output will just always be 0. So should I drop all rows where it's "off" in my train/test data sets? I feel like I would get a better fit that way.

If I delete those rows, I'm concerned about how I would handle those rows in the test set. For example, I'd have to write code to loop through the data and put a 0 in the prediction column for those rows, as well as make sure everything else lines up. (This is all Kaggle related, so the training set is several columns of features and a y_column, whereas the test set doesn't have the y_column and we're supposed to predict it.)

I'm using Python and Scikit Learn's random forest, if that matters.

Rather than discarding the dominant feature (which will discard information), try reducing the number of features randomly selected when making each partition. In scikit's syntax this is max_features (mtry in R's randomForest). By default this is set to compare the square root of all features (http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), instead try much smaller (perhaps even 2 aka decision stump). You'll probably also need many more trees vs. higher settings. This will allow you to extract information from more subtle features without losing all that great information provided by the dominant feature.


Extract most informative parts of text from documents Are there any articles or discussions about extracting part of text that holds the most of information about current document.

For example, I have a large corpus of documents from the same domain. There are parts of text that hold the key information what single document talks about. I want to extract some of those parts and use them as kind of a summary of the text. Is there any useful documentation about how to achieve something like this.

It would be really helpful if someone could point me into the right direction what I should search for or read to get some insight in work that might have already been done in this field of Natural language processing.

What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization.

In a nutshell, TF-IDF tells you the relative importance of each word in each document, in comparison to the rest of your corpus. At this point, you have a score for each word in each document approximating its "importance." Then you can use these individual word scores to compute a composite score for each sentence by summing the scores of each word in each sentence. Finally, simply take the top-N scoring sentences from each document as its summary.

Earlier this year, I put together an iPython Notebook that culminates with an implementation of this in Python using NLTK and Scikit-learn: A Smattering of NLP in Python.


Semantic similarity between two or more sentences I need to determine how similar sentences (in meaning) are to one another.
In order to do it, I have been considering an algorithm (cosine similarity) to determine the similarity between sentences. I have thought as appropriate Word2vec or wordnet to build features for similarity.
If you have used this (or similar) approach, could you please provide me an example of use of word2vec/wordnet for similarity analysis?

Word2vec as the name suggests will create an embedding for each word in your sentence. In order to get a sentence level embedding you would need to average (or combine in some other way) the individual embeddings together.
An example of a model to generate sentence level embedding would be the Universal Sentence Encoder (USE). You may want to try it out and see if it can outperform a word-level model in your use-case.
The original paper can be found here: https://arxiv.org/abs/1803.11175
An example blog post leveraging USE: https://medium.com/@gaurav5430/universal-sentence-encoding-7d440fd3c7c7


tflite_convert a Keras h5 model which has a custom loss function results in a ValueError, even if I add it in the Keras losses import I have written a SRGAN implementation. In the entry point class of the Python program, I declare a function which returns a mean square using the VGG19 model:

# <!--- COST FUNCTION --->
def build_vgg19_loss_network(ground_truth_image, predicted_image):
    loss_model = Vgg19Loss.define_loss_model(high_resolution_shape)
    return mean(square(loss_model(ground_truth_image) - loss_model(predicted_image)))

import keras.losses
keras.losses.build_vgg19_loss_network = build_vgg19_loss_network
# <!--- /COST FUNCTION --->

(Vgg19Loss class shown further below)

As you can see, I have added this custom loss function in the import keras.losses. Why? Because I thought it could solve the following problem...: When I execute the command tflite_convert --output_file=srgan.tflite --keras_model_file=srgan.h5, the Python interpreter raises this error:

  raise ValueError('Unknown ' + printable_module_name + ':' + object_name)
  ValueError: Unknown loss function:build_vgg19_loss_network

However, it didn't solve the problem. Any other solution which could work?

Here is the Vgg19Loss class:

from keras import Model
from keras.applications import VGG19

class Vgg19Loss:
    def __init__(self):
        pass

    @staticmethod
    def define_loss_model(high_resolution_shape):
        model_vgg19 = VGG19(False, 'imagenet', input_shape=high_resolution_shape)
        model_vgg19.trainable = False
        for l in model_vgg19.layers:
            l.trainable = False
        loss_model = Model(model_vgg19.input, model_vgg19.get_layer('block5_conv4').output)
        loss_model.trainable = False
        return loss_model

I tried the code you posted the following way:

from keras import Model
from keras.applications import VGG19
import keras.backend as K

class Vgg19Loss:
    def __init__(self):
        pass

    @staticmethod
    def define_loss_model(high_resolution_shape):
        model_vgg19 = VGG19(False, 'imagenet', input_shape=high_resolution_shape)
        model_vgg19.trainable = False
        for l in model_vgg19.layers:
            l.trainable = False
        loss_model = Model(model_vgg19.input, model_vgg19.get_layer('block5_conv4').output)
        loss_model.trainable = False
        return loss_model

def build_vgg19_loss_network(ground_truth_image, predicted_image):
    loss_model = Vgg19Loss.define_loss_model(high_resolution_shape) # where is this variable coming from?
    return K.mean(K.square(loss_model(ground_truth_image) - loss_model(predicted_image)))

import keras.losses
keras.losses.build_vgg19_loss_network = build_vgg19_loss_network

print(keras.losses.build_vgg19_loss_network)  # <function build_vgg19_loss_network at 0x7f05e8e1cbf8>

I get no error messages and the function is assigned to the losses module. That means the problematic lines are probably not part of what you posted. It would be nice to know which line of code raises the error that you quoted.

However, I'm not sure where this high_resolution_shape argument on line 22 in your build_vgg_19_network function is coming from. If this is a global constant, it should be written in all uppercase letters separated by underscores to prevent confusion. If it is not defined it will throw a NameError sooner or later.

If I execute keras.losses.build_vgg19_loss_network(None, None) after running the code above, I get the following error message:

  NameError: name 'high_resolution_shape' is not defined

Edit: If this error happens only during TFLite conversion, it does so because custom objects are not yet supported by the TFLiteConverter in tensorflow 1.x. However, there is a commit on the tensorflow Github repo that addresses this issue and adds support for custom objects (see also the related pull request). It should be part of the official tensorflow v2.0.0-beta1.


why we sample when predicting with Recurent Neural Network I trained a Recurrent Neural Network to predict the next word in a sentence.

I trained and now I want to predict, but there is something I am not getting well.

I saw it in many tutorials even in the Keras documentation.

When we want to predict, we get the probabilities from the predict function and we pass it to the sample function bellow.

def sample(preds, temperature=1.0):
    # helper function to sample an index from a probability array
    preds = np.asarray(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

I am not getting what this function is doing.

I know that :

My predictions are an array of probabilities and from that, the word indice is the word with the largest probability.

From the function I can see that they are doing log, and then exp, that is what I am not getting.

Why are they doing that?

I saw this in many tutorials but no one explained it well.

Probably I should go back to my statistics and probabilities courses.

Please point me to the right direction.

It seems to me you have two questions:

Why use sampling to generate text from a trained RNN language model?
How does this particular sampling function from Keras work?

Why use sampling to generate text from a trained RNN language model

After training a language model, very often you would like to use the model to generate new text. For a word-level RNN language model, text is generated one word at a time. In each step, the model outputs a probability distribution over the entire vocabulary. With a toy vocabulary such as:

$$
vocabulary=\hspace{0.2cm} \begin{bmatrix}
           cat \\
           dog \\
           frog
         \end{bmatrix}
$$

the distribution output by the model at each generation step could look like

$$
predictions=\begin{bmatrix}
           0.7 \\
           0.1 \\
           0.2
         \end{bmatrix}
$$

each entry in the vector corresponds to one entry in our vocabulary.

After that, it is up to the user how to select a word from this distribution as the next word in the generated text. Two very different methods to pick a word from the distribution are:

Greedy search: take the word with the highest probability (will pick "cat")
Sampling: sample from the distribution, take into account probabilities (will pick a random word, but "cat" has the highest chance of being picked)

So, searching will return a very probable string (and the same one every time we try), while sampling will return somewhat likely, but more varied text. Often, this means that sampling leads to more "interesting" text.

How does this particular sampling function work?

As far as I can tell, the input for this sampling function is a vector of normalized probabilities after softmax, such as

np.array([0.7, 0.1, 0.2])

Here are some explanations about each line:

preds = np.asarray(preds).astype('float64')

This just makes sure that other array-like objects, such as Python lists, work as input, but are converted to the correct numpy array type.

preds = np.log(preds) / temperature

preds is a probability distribution that was created by applying the softmax function to the final output of the model network (those final values are called logits). Logits are assumed to be unnormalized log probabilities, applying softmax normalizes them.

This is a problem for softmax with temperature, because it is the unnormalized logits that are supposed to be divided by the temperature:

$$
q_i = \frac{exp(z_i/T)}{\sum_j{exp(z_j/T)}}
$$

So, np.log(preds) just moves the values back into logspace. Without this, we would not be scaling the actual network outputs, but exponentiated values.

exp_preds = np.exp(preds)
preds = exp_preds / np.sum(exp_preds)

Those two lines just apply the softmax function, regardless of what happens before or after.

probas = np.random.multinomial(1, preds, 1)

This function with those particular arguments is sampling a one-hot vector such as

$$
predictions=\begin{bmatrix}
       1 \\
       0 \\
       0
     \end{bmatrix}
$$

where the probability of each index being the 1 is dictated by preds.

return np.argmax(probas)

Finally, this returns the index of the largest element in the vector. This index will correspond to one word in our vocabulary.

I hope this helps you understand the matter, or perhaps express more clearly what you would need additional explanations for.


What is the best way to predict time series data? I have monthly price data for tomatoes for the last 9 yrs for a particular town and I'm looking to predict the prices of tomatoes 6 months into the future.

I had considered using Linear Regression in Tensorflow (something I learnt about a week ago), because there is a direct relationship between price and rainfall for the location in question. That is, high rainfall means a supply glut and a price drop, low rainfall means scarcity and price hikes.

However I've found that I can't get accurate weather forecasts for 6 months into the future and have to use some other way other than Linear Regression to predict future tomato prices.

What is the best way to do this?

You can find the seasonality in your data (if there exist) and by removing the seasonality, find the trend of tomato price and predict the future price using the seasonality data and trend data. If you are using python, you can find the seasonal library useful to apply this method to your data.

The advantage of this method to your method is you don't need to think over finding dependent features and any feature engineering.

Moreover, as you have mentioned in tags of your question, you can using RNN to predict the future value by as your data is time series and sequential intrinsically. You can find this article useful.


How to maximize recall score for specific label in multiclass classification? Thought I had solved the problem but I'm having inconsistent issues with it so reaching out here.
I have a multilabel classification problem with four labels ['--','-','+','++'] and with a basic random forest model, I have significant performance issues with one label '-', while the other three labels are performing pretty decently.
model = RandomForestClassifier(random_state=42)
model_name = 'RFC_base'

grid_params={}

pipe = GridSearchCV(make_pipeline(model),scoring='recall_weighted',cv=5,param_grid=grid_params,n_jobs=-1)

          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_base    0.848485    0.840909    0.871795    0.771429    0.893617

I'm well aware that GridSearchCV is over engineering, but I like to keep code consistent between tinkers while optimising.
I wanted to improve the recall score for '-', so created a custom scoring function that I thought would be maximising the recall score for '-'.
recall_neg_scorer = make_scorer(recall_score,average=None,labels=['-'],greater_is_better=True)

I've been tinkering with this to optimise it, hence trying to define the 'greater_is_better' param. So I do an actual GridSearchCV now changing some of the params.
model = RandomForestClassifier(random_state=42)
scaler = StandardScaler()

grid_params = {
    'randomforestclassifier__n_estimators': [81], #np.arange(start=60,stop=90,step=1),
    'randomforestclassifier__max_depth': np.arange(1,10,step=1),
    'randomforestclassifier__criterion': ['gini'], #,'entropy'],
    'randomforestclassifier__max_features': [7], #np.arange(1,X.shape[1]+1,step=1),
    'standardscaler': [StandardScaler(),MinMaxScaler(),None]
}

model_name = 'RFC_gscv_recall_neg'

pipe = GridSearchCV(make_pipeline(scaler,model),scoring=recall_neg_scorer,param_grid=grid_params,cv=5,n_jobs=-1)

What I'm finding is my scoring function doesn't actually optimise '-' when 'greater_is_better = True'
For instance, when GSCV runs on 'max_depth' it chooses 9 which gives :
          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_gscv_9  0.836364    0.818182    0.871795    0.742857    0.893617

Which is performing worse on '-' than even the base case. When I force max_depth = 8 it gives :
          recall avg    recall (++) recall (+)  recall (-)  recall (--)
RFC_force_8 0.842424    0.818182    0.871795    0.8        0.87234

When I choose 'greater_is_better = False' it actively tries to minimise the score. Is there something I'm screwing up here or is there a known issue that I'm missing out?
Also I'm a bit new to stack so let me know if there's something I'm missing.

I would highly suggest doing some deeper Analysis on why the ("-") class is beeing predicted. Meaning what do you predict for class ("-") when no the correct class. Analysing this you could see that you are ordering class ("-") as class "++". Than you can look at the Input data and feature engineer Features that discriminate between These two class. Forcing you model to do this indirectly is a fair try, but model cant learn whats not to be found in the data. With feature Engineering you can enhance the data so that discrimination is better.


Which method should be considered to evaluate the imbalanced multi-class classification? I am working on multiclass-imbalanced data. My dependent variable is highly skewed.

           Injury

2(No Injury)         208753
1(Medium Injury)      22318
0(severe Injury)       3394

I have used random forest algorithm with parameter "class_weight='balanced' " to manage the class 2 imbalance.

I get the below results when I use average='micro'.

 [[   34   107   688]
  [  148   778  4592]
  [  905  4635 46730]]
 Accuracy Score: 0.8110616374089428
 precision score: 0.8110616374089428
 Recall score: 0.8110616374089428
 AUC Score: 0.8582962280567071
 F1 score: 0.8110616374089428
 Kappa Score: 0.05522284663052324

For the average = 'macro', the results are below.

[[   31   125   684]
 [  157   838  4559]
 [  890  4694 46639]]
 Accuracy Score: 0.8104816009007626
 precision score: 0.3586119227436326
 Recall score: 0.3602869806251181
 AUC Score: 0.5253225798824679
 F1 score: 0.3592735337079687
 Kappa Score: 0.06376296115668922

So, which results should I consider to evaluate the model? If I have to consider the macro, then my model performance is really bad. Please suggest if there are any methods to improve the precision, recall and AUC score?

If I consider micro results, my precision, recall, f1 score is same. How can I justify this in the project?

I am interested in medium and severe injuries. But individual class precisions 0.0312(severe), 0.1409(medium) for macro are very low. The overall precision score is also very low 0.35. Is it possible to increase these scores? Or Is it fine to consider these low values as my final results for the project? Any suggestions are greatly appreciated. NOTE: I have tried SMOTE oversampling and ensemble cross-validation with different algorithms but I ended up having precision and recall scores less than 50.

Any help would be appreciated.

Thank you.

This answer explains how micro and macro are different and which one you should use.


Need help with LDA for selecting features I am currently selecting features of products by using LDA to group 6000 keywords of product into topics.
Here is the sample of my dataset after being organized into list of keywords for each product id.

I consider each id as a "document" and each keywords as the "word" in a "document" for the case of LDA.
It didn't work out as I expected as each topic have many identical keywords with different weight. I removed 100 most common keywords but there are still some identical keywords in the topics. Here is the sample output:

How can I deal with the identical keywords in my topics? and also how to deal with the 100 most common keywords I removed?

Here is the graph showing the frequency of the words in all "documents"

Each word only presents once in each document, but may present in different documents. I updated a new graph of the frequency of the words

Thank you so much. Any suggestion is appreciated.

Try before doing LDA look at the data - like doing TF, IDF and TFIDF analysis to identify such words which happen in all subject. If You have some taxonomy of Your product definition - consider using it. In my case it was really helpful.

I experimented with LDA topic modeling for recommendation systems purposes. I've few runs for offers in our marketplace service. It is in some aspect similar to Your problem with products and a list of keywords. But in our case, we do not have a product definition and an offer description is created by a seller. So You can imagine what tokens soup You will get :-)

The key point for me wast to analyze TF, IDF and TFIDF inside categories where I've tested LDA. E.g. for Toys category (http://allegro.pl/zabawki-11818?ref=simplified-category-tree) ~ .5M items, LDA based only on its names results in:

TOPIC:
lego    0.02282500803373657
puzzle  0.016679246682968853
gra     0.01092812347216676
klocki  0.010650330607355207
trefl   0.006797740719581059
zabawka 0.006365688767908592
lalka   0.0063546887775893105
maskotka        0.006111593588558912
dzieci  0.005888846995546501
24h     0.005736876716620104

TOPIC:
lego    0.02296167186908205
puzzle  0.016613064812376045
gra     0.010928234683331952
klocki  0.010607432835126579
trefl   0.006689061156804526
lalka   0.006317220992078405
zabawka 0.0062838718987015575
maskotka        0.006121026383268814
dzieci  0.0058569443925760144
24h     0.00578544965708585

TOPIC:
lego    0.02285507528449409
puzzle  0.016539723998111246
gra     0.010892673154789407
klocki  0.010624244094063881
trefl   0.006683424205358961
lalka   0.006350949499009139
zabawka 0.006324510823409019
maskotka        0.006148070280704085
dzieci  0.0058401580163702565
24h     0.005782372531889951
...

And the rest looks almost the same. Before running LDA I just did basic text preprocessing - our dictionary based item-name stop words removal and text tokenization (but without steaming). Even giving more terms for topic descriptions results doesn't look promising. Then, after exporting word frequencies with its offer categories by looking at data and plots I've decided to remove terms with TF-IDF above some threshold. Yes, above - I've used the calculation provided by Spark 1.3.1 implementation (HasingTF + IDF from mllib, no ml). After doing this I received:

TOPIC:
interaktywny    0.0026985965530064884
pony    0.0026721637823727343
dmuchany        0.0015777211309733249
baterie 0.0012447186564456534
pojazdy 0.0011171017074143481
thomas  9.476877418459519E-4
dinozaury       8.823212274401862E-4
monsters        7.426822230409613E-4
heroes  7.365256561824247E-4
ninjago 7.344344320326593E-4

TOPIC:
pony    0.0026867880330479327
interaktywny    0.0026803244251279693
dmuchany        0.0015797940843537916
baterie 0.001263424692131187
pojazdy 0.0010685733429671523
thomas  9.87888159025782E-4
heroes  8.26066846602493E-4
ptaszek 7.952454816383102E-4
dinozaury       7.849838129629457E-4
batman  7.408185413186286E-4

Where results started to differ from each other. Taking like last time only 10-terms for every subject description. Still, it's not perfect but better.

So, for me doing the experiments which few categories, the way to overcome problem was to remove it based on TF-IDF value. But, for every category the threshold was calculated separately. Mainly: mean(tf-idf) + 3*sd(tf-idf).

I know that Idf multiplication factor in Tf-Idf should resolve this problem by itself and the term should be punished for occurring in every document. But, using Spark implementation (idf = log((m + 1) / (d(t) + 1))) and our data it was simpler during a quick and dirty experiment to filter it that way.

When I find few sec. I will get back on this and place results with code and online to share this.


Aggregation of Discount I am trying to predict sales quantity of an item based on their attributes. Discount is one of those attributes. The problem is I am having different discounts in same period for same item .I  need to  consider sales for period wise i.e week  .While aggregating I am facing  problem

on differnt days item has different discounts.
Case1: Some items are having large variations in discounts like(0% and 50%)
Case2: some are having small variation like(40% and 49%).

-In case1 If we aggregate the discount and divide by sales quantity then there will be  decrease of  discount in one record and  increase in another.
- In case2 If we aggregate discounts and take average of that .It seems there is no large variations in discount by taking average.Which is not correct for first case.

In my data Sales is more sensitive to discounts.

Is there any way to get generalized discount to predict sales(with Random forest algorithm)?

Aggregate discount sequences are used to define the stacking logic for aggregate discounts. This logic can define:

The order in which aggregate discounts are applied.
Which aggregate discounts are mutually exclusive.
Which aggregate discounts can be "stacked," or applied in conjunction with one another.

You can use aggregate discounts within a quote or an order only if they are part of an aggregate discount sequence. Therefore, you must define an aggregate discount sequence even if you have only one aggregate discount. An aggregate discount can be a part of many aggregate discount sequences.

Aggregate discount sequences are linked with price list and discount matrix line items (like volume discounts and attribute adjustments). At run time, the pricing procedure picks the aggregate discount sequence to be used and evaluates the aggregate discounts to be applied.

The order in which aggregate discounts are applied within an aggregate discount sequence can affect the resulting price. For example, if aggregate discount (a) gives a discount of $100, and aggregate discount (b) gives you 10% off the price of the product, the resulting price would be lower if aggregate discount (b) is applied before aggregate discount (a).

To apply all the aggregate discounts in a sequence, you attach the sequence to either a price list or a product with component line items. A sequence includes all the aggregate discounts that apply to one price list or to one product with components in a price list.

Use the following procedure to define the aggregate discount sequence. This task is a step in Process of Creating Aggregate Discounts.


How to find correlation between time-series of different units? I have 3 time-series data.

NDVI(normalized difference vegetation index) mean
Precipitation
Temperature
All of these have their own unit.
Now I want to find similarity/correlation between NDVI and precipitation,
NDVI and temperature. Basically, my aim is to find "NDVI is more correlated with precipitation or temperature".
Should I normalize precipitation and temperature to NDVI values?

I recommend you to scale all three time series to unitless values, with zero mean and standard deviation of 1. After that, you can search for correlation among normalized time series. Python has StandardScaler function that could help with it.


LFR as synthesized data for dynamic community detection methods In recent methods for community detection in dynamic networks, LFR benchmark is used as dynamic dataset generator, but I thought it is for static community based data generation. For example in paper Overlapping Communities in Dynamic Networks: Their Detection and Mobile Applications, LFR is used. But I don't know how this dataset is generated.

Here is the code of this paper and ground truth included is only for one snapshot.

Question: How is the dataset generated?

I think the dataset is generated using dynamic network generator made available at this address which is dynamic version of static LFR. It's an excellent dynamic networked data generator.


BOVW - Combine vocabularies What I have so far

I have a set of images that I am trying to classify. I can also extract different feature descriptors from the images using algorithms such as hu moments, color histogram, and SIFT. I can the build a vocabularies from each of these algorithms.

What I do not understand is how can I combine these vocabularies together. From my understanding these feature descriptors can be of different size depending on what the features are. What would be the proper way to build this vocabulary from these different algorithms.

What is confusing me is how would we be combining feature descriptors from different algorithms which have different sizes? SIFT will give a m x 128 array where m is the number of feature descriptors of set size 128. A color histogram may give a count of each color in the range of 0-255. How would I combine these two in a meaningful way?

Here is the paper I was reading which gave me the idea of combining the vocabularies.

A Visual Vocabulary for Flower Classification

Here is what I was reading before:

Implementing Bag of Visual words for Object Recognition

If anyone can point me in the right direction I would greatly appreciate it.

I also found an interesting paper (with code) where they merged local and global descriptors.

Scene Recognition by Combining Local and Global Image Descriptors

According to the paper: They take the daisy features (local features), cluster the features into k clusters, then create a histogram with k bins. They normalize the histogram, once they do this they concatenate this vector (the histogram) with the normalized HOG vector (global features). So in the end we just cluster the local features and create histogram, then just concat with global vector and there we have a training sample.

Also there may be an alternate method where you just use local features to cluster: https://www.quora.com/How-do-I-merge-features-from-different-feature-extractors-i-e-color-histogram-and-SIFT-for-bag-of-visual-words#


Very low accuracy of new data compared to validation data I'm trying to train the neural network to predict the movement of a particular security on the market.

I teach on historical data collected for the year. At the entrance of the neural network candlesticks are served: close price and value

Before submitting, these data are normalized separately for each dataset. This happens with the z Score algorithm. Then the question immediately arises... output can not be obtained in the limit [0;1] or [-1;1] and can reach up to 10 or more (in both directions). Is that okay?

All sets are first shuffled, and then divided into two parts:: train and test (80:20)
The output is a class: the price will go up or down.

For train, the error is reduced to zero and the accuracy to 100%. For test, the error reaches a minimum of 0.28 and the accuracy reaches 90%

I tried to test this neural network to predict the next two months, which were not used in this neural network. They went through exactly the same normalization. However, the accuracy of this forecast is 0.5289256204750912. And the error - 1.8223290671986982

I know I'm doing something wrong, but I don't know what... I hope someone can help me figure this out.
PS: ZScore was used To the usual min, max. However, this particular success was not observed.

You are experiencing Data Leakage. In a comment, you explained that you shuffle your data before splitting into train/validation. For each validation point, you likely are showing the model data that is nearby temporally both before and after the time of the validation point. This is information that the model can not possibly hope to have when running in real time.

To alleviate this, I would keep the data in the correct time order, and instead take the validation data as a contiguous chunk of time. If you want to be the most careful, you could throw data away in a small window around the validation data, so that the training data won’t contain information about the edges of the validation data.

This appears to be a good resource, but I have not read it too thoroughly: https://www.kaggle.com/dansbecker/data-leakage


What does the long curve-shape t-SNE mean? I use 1-D CNN input 1*512 size time series data which randomly fragment segment, the output will classify input into 10 classes. After training the CNN, I apply t-SNE to the prediction which I fed in testing data.
In general, the output shape of the tsne result is spherical(for example,applied on MNIST dataset). But now I apply t-SNE on my own dataset. No matter how I adjust perplexity early, learning rate or maximum number of iterations. It will give me the result of long-shaped output, just like the pic below. Does the long-shaped t-SNE mean anything? Thanks everyone beforehand.

Edit: Explaining about dataset contents. p.s. before feeding into CNN for training, I randomly split it into training/testing dataset.

The lines therefore are the time series. t-sne is a very locally sensitive algorithm so every data point is very likely to show up nearest (in 2-D) to its nearest (N-D) neighbour.

In MNIST there is no sequence, so it doesn't show up as a line in 2-D. Every digit is written in a set of similar ways that cluster together. In your time-series data, if each time point is most similar to its next time point, what we expect is for them to be in some sequence - because you'd want point t=n to be closest to points t=n-1 and t=n+1  (if they are in fact the most similar).


Can I apply Dropout In layers other than Fully Connected layers in CNN I have read and seen that in CNN we apply DROPOUT layer between the FULLY CONNECTED layers to reduce overfitting.
Can we also apply the dropout layer between the CONV layers and the POOL layers. I have not seen models with this method applied. Will it help in overfitting when applied between these layers or are there any disadvantages to it?

In a CNN, each neuron produces one feature map. Since spatial dropout which is normally used for CNN's is per-neuron, dropping a neuron means that the corresponding feature map is dropped.
Pooling usually operates separately on each feature map, so it should not make any difference if you apply dropout before or after pooling.
Yes it should help with preventing overfitting. The added advantage is you can get uncertainty from the network at zero cost if using dropout.


Mixed effect random forest model for Python Windows Does anybody know if there is a Mixed effect random forest model for Python Windows? The merf package https://anaconda.org/search?q=merf+ seems to only be available on a linux environment? thanks!

I have tried installing "merf" on Windows 10 - it worked. I did it like this:

pip install merf


Model for predicting duration based on categorical data I am working on a model which will allow me to predict how long it will take for a "job" to be completed, based on historical data. Each job has a handful of categorical characteristics (all independant), and some historic data might look like:
JobID   Manager     City        Design          ClientType      TaskDuration
a1      George      Brisbane    BigKahuna       Personal        10
a2      George      Brisbane    SmallKahuna     Business        15
a3      George      Perth       BigKahuna       Investor        7

Thus far, my model has been relatively basic, following these basic steps:

Aggregate the historical data based on each category, calculating the mean, and counting how many times it occurs. From the previous example, the result would be:

Category        Value           Mean    Count
Manager         George          10.66   3
City            Brisbane        12.5    2
City            Perth           7       1
Design          BigKahuna       8.5     2
Design          SmallKahuna     15      1
ClientType      Personal        10      1
ClientType      Business        15      1
ClientType      Investor        7       1

For each job in the system, calculate the job duration based on the above. For example:

JobID   Manager     City        Design          ClientType
b5      George      Brisbane    SmallKahuna     Investor

Category        Value           CalculatedMean      CalculatedCount     Factor (Mean * Count)
Manager         George          10.66               3                   31.98
City            Brisbane        12.5                2                   25
Design          SmallKahuna     15                  1                   15
ClientType      Investor        7                   1                   7

TaskDuration    = SUM(Factor) / SUM(CalculatedCount)
                = 78.98 / 7
                = 11.283
                ~= 11 days

After testing my model on a few hundred finished jobs from the last four months, I calculated average discrepancies ranging from -15% to +25%.
In my actual model I have 15 categories, and am drawing historical data from ~400 jobs.
I think the largest issue (amongst others) is the simplicity of my model. Are their better/well established methods for calculating a value based on categorical data? And if not, how can I improve my predictions?
Related question here.

So, from what I understand of the question, you are asking how to model the duration of a job, given the input (which includes City and ClientType).
In this case you can use something like a feedforward neural network to model this problem. You might find, that using this methods, the level of prediction error will be lower than that produced by your model, which could act as a baseline to see if these models do work better for your problem.
When representing each of the categorical variables, we use something called one-hot encoding. Then we concatenate these categorical variables into one n-dimensional input vector, which represents all the features for one example.


Least Squares optimization The cost function given as $\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$ is used to evaluate the weights $\beta$. Here $X$ is the data and $Y$ is the output. On taking the derivative, we get the estimates of the weights. This is a Least Squares formulation.

1) Can Least Squares (LS) be used when the observation (outputs) $y_i$, $i=1,2,..,N$ number of examples are categorical? I don't quite get the picture how classification problems using LS works in terms of derivative for categorical cases.

2) Can LS be used when the data $X$ is a one-hot encoding? Would the formulation and derivative be the same?

Coming to your first question:

Yes, you can but it's not advisable to use LS as a cost function for classification task since the optimization problem becomes non-convex! More on it, since your model in logistic regression would be sigmoid(a non-linear function) which means your cost function will have a lot of local optima!(think of a surface of the cost function as a Himalayan valley!), So, when you will use Gradient descent(an iterative approach) for minimizing your cost function, GD will be stuck somewhere in the local minima instead of global minimal, so you will not learn the best model parameters for your model on your data! Therefore, we change the cost function from LS to Log loss for classification task that ensures a nice bell-shaped curve surface of the cost function where the gradient descent can reach to global minima and give you the corresponding best model parameters which you can happily take it later to do predictions(on test-set)!
More on it here

Much more theoretical understanding of it you could find in ISLR at page 129 4.2 Why Not Linear Regression?


Why is the kernel of a Convolutional layer a 4D-tensor and not a 3D one? I am doing my final degree project on Convolutional Networks and trying to understand the explanation shown in Deep Learning book by Ian Goodfellow et al.
When defining convolution for 2D images, the expression is:
$$S(i,j) = (K*I)(i,j) = \sum_{m}\sum_{n}I(i+m,j+n)K(m,n)$$
where $I$ is the image input (two dimensions for widht and height) and $K$ is the kernel.
Later it states that the input is usually a 3D tensor. This is because usually the input image has multiple channels (say, red, green and blue channels). Furthermore, it says that it can also be a 4D-tensor when the input is seen as a batch of images, where the last dimension represents a different example, but that they will omit this last dimension for the sake of simplicity.  I understand this.
Let now $I_{i,j,k}$ be the input element with row $j$, column $k$ (height and width) and channel $i$. The output of the convolution is a similarly-structured 3D-tensor $S_{i,j,k}$.
Then, the generalized convolution expression for this 3D-tensor input is
$$S_{i,j,k} = \sum_{m,n,l} I_{l,j+m-1,k+n-1}K_{i,l,m,n}$$
where "the 4-D kernel tensor $K$ with element $K_{i,j,k,l}$ giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input, with an oﬀset of $k$ rows and $l$ columns between the output unit and the input unit".
I am completely lost in this definition. Why is the Kernel 4-D and not 3-D (which is a logical generalization of the first formula)? What is the analogous to the sentence "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input" in the initial 2D Kernel tensor? I think that is the main things that has to be understood to understand the 4D-kernel.

The first formula you quote is for an image with one input channel and one output channel, it just focuses on height and width. In this case, if we consider a 5x5 convolution, the Kernel will just have size 5x5, $m$ and $n$ and going from -2 to +2.
Now if our input has 3 channels (RGB, but could be feature maps). we need to use each channel as an input, and the weights will be different for each input map. So K becomes 3-D : 3x5x5. This new dimension corresponds to the $l$ in your formula.
If we want to have 10 outputs, we need 10 different ways of convoluting the input feature maps, so our Kernel will have one more dimension, leading to a 10x3x5x5 size for the Kernel. This last dimension corresponds to the $i$ of your formula.
To recap, the 4 dimensions of the Kernel $i, l, m, n$ stand for :

$i$ : Output dimension
$l$ : Input dimension
$m, n$ : height and width

About this sentence "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $j$ of the input".
I feel it is wrong as I would rather interpret it as : "giving the connection strength between a unit in channel $i$ of the output and a unit in channel $l$ of the input"


Time series regression forecast next month from now (random forest,Lasso,Ridge) I have a dataset about hedge funds. Its include data from 2010 january to 2019 december. This data are monthly financial ratios of hedge funds such as sharpe,alpha,beta,sortino and monthly returns of hedge funds. I normalized the relative returns of each fund. And I want to estimate these monthly returns. Using these ratios. Now Im using machine learning regression models, I created this function "$$Y_{t+1} = X_{0,t}+...+ X_{12,t}$$" for the regression model. "$Y_{t+1}$" shows the normalized relative return of the hedge fund in the following month(t+1)(next/future), "$X_{0,t}$" variables show the financial ratios of that fund in the t month(now). I thought about this function to predict future month's returns before data came so I want to predict the next month with the current data.Because the data comes as of the end of the month. I have to guess the return for the next month. As an example, I would like to forecast the return on hegde funds at the end of January 2011 with data from January 2010 to December 2010. Without ever seeing the data for January 2011. Is this regression function written correctly.

Does using $Y_{t+1}$ break the model and is there a model or mathematical function that you can suggest? for "predict future month's returns before data came so I want to predict the next month with the current data."

Additonal Infos:

Some additional information there are 2889 months of data, I keep all 27 different funds in a database, and I train a single regression model, not for each fund separately.I used the regression function I showed above($Y_{t+1} = X_{0,t}+...+ X_{12,t}$).

But it didn't work well on any algorithm, so the R-square score was negative. why do you think it didn't work and how can I fix it ?
I used both sliding window and expanding window as validation. I use dummy variables for fund names/tickers

Sample Data:

You could try to predict difference $dy = Y_{t+1} - Y_{t}$, not the $Y_{t+1}$ itself. Intuitively, the weather today is good starting prediction for the weather tomorrow.

Secondly, with time series, the seasoning is quite important. You usually would like to use models like ARIMA that takes this into account.

Lastly, it could be just don't enough data to make good predictions. I don't know what is these indexes exactly, but I'm not sure you can use data from one fund to predict another. This leaves you with about 100 points of training data (8 years monthly) and it's not a lot.
I would recommend you to solve 1-d (predict one fund by only it's data) with $dY$ trick, then try ARIMA


Choosing an embedding feature dimension I'm trying to tackle a classification problem with a neural net tensor using flow.
I have some continuous variable features and some categorical features.
The continuous features are normalized using sklearn's StandardScaler.
For the categorical features I am using a series of embedding features that I'm concatenating together with my continuous features.

The embedding features are created like so :

 airline = tf.feature_column.categorical_column_with_hash_bucket(
        'AIRLINE', hash_bucket_size=10)

then :

 tf.feature_column.embedding_column(airline, 8)

However I am having trouble choosing my embedding feature output size. I understand this transforms my sparse one hot encode "airline" feature into a float vector of size 8.

Is there a heuristic I can use to choose an embedding feature size ?

My neural net's accuracy remains stuck at 31%. It doesn't seem to be learning even after a 100 epochs. Could the size of the embedding features be a cause for such a behaviour ?

I think post below is a good resource.

https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html

Basically, all categorical variable is initially converted to one-hot encoding, then layer size defined by dimension argument is stacked on top of one-hot encoding; thus learning optimal representation of categorical variable based on specified dimension.

There is general rule in the blog post to take the 4th root of the number of categories.

Another approach is to perform MDS to inspect your categorical variables to decide dimensions.


How to apply a different Loss function to one specific Label? I got a recurrent neural network in Keras, which classifies on 14 labels.
The first label is the most important one and should be predicted with the highest accuracy. The other labels don't have to be very accurate, they should just help the network to generalize better.

I thought of using the "Mean absolute error" for the first and the "mean squared error" for the other labels.

How can I integrate this in Keras?

Using mean absolute error might damage your output. You must do some regularization. One thing you can do would be to assign custom weights for each class metric.

Though in general, I would strongly advise against doing that. If this class is so important do separate model in one-vs-all fashion. Be aware you'll be overfitted.


TF2.0 DQN has a loss of 0? I am having a hard time understanding why my loss is constantly a zero when using DQN.
I'm trying to use the gym environment to play the game CartPole-V0.

In the code below,
r_batch indicates rewards sampled from the replay buffer, and similarly s_batch, ns_batch, and dones_batch indicate the sampled state, next states, and if the "game" was done. Each time, I sample 4 samples from the buffer.

# This is yi
target_q = r_batch + self.gamma * np.amax(self.get_target_value(ns_batch), axis=1) * (1 - done_batch)
# this is y
target_f = self.model.predict(s_batch)
# this is the gradient descent computation
losses = self.model.train_on_batch(s_batch, target_f)

The values of target_q are:

[ 0.42824322  0.01458293  1.         -0.29854858]

The values of target_f are:

[[0.11004215 1.0435755 ]
 [0.20311067 2.1085744 ]
 [0.413234   4.376865  ]
 [0.24785805 2.6716242 ]]

I'm missing something here, and I don't know what..

OK, a bit more digging led me to this:

The TD target or "target value" gets its name because by updating a Q table or training a NN with it as a ground truth, the estimator will output values in future closer to the supplied value. The estimator "gets closer to the target".

So, I have to update the prediction network with the "ground truth".

    for i, val in enumerate(actions):
        predict_q[i][val] = target_q[i]


Predict time of dispatch for marketing campaign What would be appropriate models/algorithms/strategies for predicting best individual send times for marketing campaigns based on past response timestamps?

Data:

Given for example

===========================================================
  customer campaign    campaign_time       response_time
-----------------------------------------------------------
1   100       a     2017-01-01 06:50:01 2017-01-01 08:02:21
2   101       a     2017-01-01 06:50:01 2017-01-01 16:45:31
3   101       a     2017-01-01 06:50:01 2017-01-02 07:20:00
4   100       b     2017-01-07 06:30:21 2017-01-08 08:15:21
5   101       b     2017-01-07 06:30:21 2017-01-07 17:00:12
6   100       c     2017-01-14 06:43:55 2017-01-14 07:59:44
7   101       d     2017-01-21 14:02:01 2017-01-21 16:50:01
-----------------------------------------------------------

two customers 100&101,
four past campaigns a-d.
with each campaign having different times of dispatch,
and multiple,one or no response time(s) (e.g. buying a product) for customers and campaigns

Goal:

Assuming that

campaign_time can vary for 100 and 101 (personalized times
of dispatch), and
past response times are an indicator for when
customers are most receptive for a campaign

I would like to predict the best next campaign_time ( 2017-01-28 ??:??:??) for each customer based on past response_times, so that the number of respondents per campaign is maximized.

Anyone having any experience with something similar or any ideas where to start? I'd be happy to hear some ideas.

To simplify things, I'd consider the first response_time the most valuable one (=> should be predicted) and I'd also abstract from weekdays (=> it's about predicting time 0:00-23:59, marked by the ? above); however it would be nice to have a continous prediction instead of a discretized one (like suggested here).

I might suggest a change to your theoretical setup:

It sounds like you are trying to maximize an equation f(t), where t is the time of dispatch, and f(t) is the risk adjusted chance of purchasing (as you note it can be 0-many).

In that case, it is likely better to use a setup like this:

===========================================================
  customer campaign    campaign_time       total_responses_within_1week
-----------------------------------------------------------
1   100       a     2017-01-01 06:50:01       0
2   101       a     2017-01-01 06:50:01       3
3   101       a     2017-01-01 06:50:01       2
4   100       b     2017-01-07 06:30:21       1
5   101       b     2017-01-07 06:30:21       1
6   100       c     2017-01-14 06:43:55       1
7   101       d     2017-01-21 14:02:01       1
-----------------------------------------------------------

Unless you have specific internal data or journal research that directly indicates a relationship between speed of response and likelihood/number of purchases, I might avoid including it.

Now, if you need to estimate time of response for issues like labor or inventory, you could use a probabilistic model, but I'll avoid making specific recommendations there, as my relative strength is in regression models. You'd want the function to output a probabilistic curve over a given time period, and then sum all of those curves together (e.g. to estimate that during Weds morning, you will have 80-109 orders, so need 5 customer service reps).

If your goal is purely response frequency (within a relevant time), time of response doesn't need to be modeled, however.


How does the bounding box regressor work in Fast R-CNN? In the fast R-CNN paper (https://arxiv.org/abs/1504.08083) by Ross Girshick, the bounding box parameters are continuous variables. These values are predicted using regression method. Unlike other neural network outputs, these values do not represent the probability of output classes. Rather, they are physical values representing position and size of a bounding box.

The exact method of how this regression learning happens is not clear to me. Linear regression and image classification by deep learning are well explained separately earlier. But how the linear regression algorithm works in the CNN settings is not explained so clearly.

Can you explain the basic concept for easy understanding?

A very clear and in-depth explanation is provided by the slow R-CNN paper by Author(Girshick et. al) on page 12: C. Bounding-box regression and I simply paste here for quick reading:

  Moreover, the author took inspiration from an earlier paper and talked about the difference in the two techniques is below:

  After which in Fast-RCNN paper which you referenced to, the author changed the loss function for BB regression task from regularized least squares(ridge regression) to smooth L1 which is less sensitive to outliers!. Also, you embed this smooth L1 loss in the multi-task loss function so that we can jointly train for classification and bounding-box regression that wasn't done before in R-CNN or SPP-net!

  However, the same author has changed the loss function again in the upcoming paper faster-RCNN
  Later, in FCN
  Many a time, in order to learn about a topic, you need to do backtracking through research papers! :) Hope it helps!


Why do we divide the regularization term by the number of examples in regularized logistic regression? So this is the formula for the regularized logistic regression cost function:

$x^{(i)}$ - the $i$'th training example

$\theta_j$ - the parameter of the $j$'th feature

$m$ - the number of training examples

$n$ - the number of features

$y_{(i)}$ - the actual outcome of the $i$'th training example, such that $y
\in \{0, 1\}$

$\lambda$ - the regularization term

$h_\theta$ - the hypothesis function that produces a prediction in the interval $(0, 1)$

$h_\theta(x^{(i)})$ - predicted value of the $i$'th training example, such that:

$h_\theta(x^{(i)}) = \sigma(\theta^{T}x^{(i)})$, where $\sigma(z) = \frac{1}{1+e^{-z}}$ (the sigmoid/logistic function)

My question is about that last term:

From my understanding, in order to do regularization, we need to find the sum of all the squares of the parameters $\theta$. That is clear enough. Also we multiply this sum by the regularization term $\lambda$, which we can change if we think the data is overfit/underfit. Good. Then, for convenience we divide this term by $2$ such that when we take the derivative we will get rid of that $2$ that would come from the exponent of $\theta$. All clear so far. BUT, why in the world do we also divide by $m$ (the number of training examples)? We divide the left term by $m$ in order to find the average error, and that makes sense since we have $m$ examples, therefore $m$ errors and after we find the sum of these $m$ errors, we would need to divide by $m$ to get the average error. But in this right term that I am confused about, we find the sum of the squares of the features, and the number of features is $n$. If we want to find the average of all the $\theta_j^2$ wouldn't we need to divide by $n$ instead of $m$, since we have $n$ features. Why does it make sense to divide that sum by $m$ and shouldn't we divide it by $n$?

There are several possible intuitions behind that as explained here, but the strongest argument for me is the following one.

The „best“ form of the loss function is the one that requires the least amount of tweaking, e.g. for different dataset sizes. Dividing the regularization term by the number of samples reduces its significance for larger datasets. And, indeed, since regularization is needed to prevent overfitting, its impact should be reduced (in favor of the impact of the data itself) if a larger amount of data is available.

This way the same $\lambda$ value has better chances for working with the whole dataset as well as with a small part of it without adjusting.


Why does the generator produce output in a different scale than the training sample? I am currently trying to train a GAN (based on proGAN) to produce images in a Vaporwave-style which is quite distinct.

The results so far have been underwhelming, which I suspect might be due to a very small training sample (only 909 images).

However in trying to optimize the model and my code I stumbled upon the following problem:

The output of the generator model has a different scale than my training samples!

My input data are image arrays scaled from 0-1, however the generator produces arrays scaled from -1 to 2!

Why does this happen? And is it a problem at all?

My code

You can find my code and the output so far here:

Vaporgan - Kaggle Notebook

If the last layer of your output has no activation (e.g. a linear layer, a convolution), then its range is unbounded. Supervised training may lead the network to learn the appropriate output range. This does not seem to be the case with your GAN.

When you need the output of the generator to be constrained to a specific range of value, the safest approach is to simply force it by having the last layer use an activation function that does so by construction. As you need the output to be in $[0, 1]$, an appropriate activation function for the last layer of the generator would be a Sigmod.


Random Forest, Duplicating Data increases Accuracy. Why? I duplicated my training data for the random forest classifier (Sklearn) and the accuracy of the prediction increased by about 3%. Why?

Without further details, my guess is that your dataset is unbalanced (not the same number of samples in both classes), so that duplicating the data increases the # samples in the most represented class. Depending on how you configured the Random Forest, it may aim at predicting correctly the most represented class and have less concern about the under represented one. In this case, by duplicating the dataset, you put even more pressure on predicting the most represented class and that's what it does, thus increasing accuracy.

As a note, I believe this would not happen if you were training on a dataset and testing on a different one (samples of the test set, on which you assess performance, should be different from the ones in the train set).


Confusion matrix of UNET image sgemenation model I have used Unet model for image segmentation. I have used RGB images and corresponding image masks and at output i got corresponding region of interest. Now i want to find confusion matrix of this model and from this i want to evaluate other parameters. Can some help? I could not get what are predicted and actual values which are required to build confusion matrix. Thanks

If your masks have only one channel and in the case of binary segmentation, you can easily compute a confusion matrix from one image thanks to:

TP: ground truth and predicted pixel are of class 1 (object)
TN: ground truth and predicted pixel are of class 0 (background)
FP: ground truth pixel=0 while predicted pixel=1
FN: ground truth pixel=1 while predicted pixel=0

On a dataset, you can get the confusion matrix from chosen iou thresholds.
For example if your iou threshold is 0.5:

TP += 1 for each image where iou(ground_truth, predicted) >= 0.5
FP += 1 for each image where iou(ground_truth, predicted) < 0.5 and predicted != empty
FN += 1 for each image where iou(ground_truth, predicted) < 0.5 and gt != empty

It is just a suggestion, there is maybe better ways to deal with that.


stable set PCA while adding features Is it possible to have a PCA setup (or any other dimensionality reduction technique) in a way that adding new features wouldn't require retrain downstream models that were trained on that particular PCA?

The idea is that (hopefully) we could have new features that would improve PCA in a way that wouldn't require retraining every time we add new features. If this isn't possible as explained here, how to achieve this in another way?

We add new features regularly by adding new datasources, or add some new features through feature engineering. Downflow models rely on PCA of that feature space. By training models on PCA and not on features directly simplifies workflows, and also removes burden of retraining models if it was possible to have a PCA (or another feature extraction process) that maps original features to this "stable" subspace.

If you're prepared to use a neural net, it might be worthwhile looking into spatial pyramid pooling.

A spatial pyramid pooling layer can take inputs of varying sizes (eg from  a convolutional layer) and convert them to an output of a fixed size (to pass on to a fully connected layer for instance). This is probably most useful where features are spatially related in some way.

Alternatively, you could continue simply implementing pca on your inputs, selecting the N dimensions with the highest variance for each example, but train the model continuously through reinforcement learning.

As there is more to how data is distributed along an axis than the variance, it is unlikely that any one pretrained model will perform well when your preprocessing step is constantly swapping out features, which is why a reinforcement model would be needed.  Combined with the pca, hopefully that won't take up too much overhead.


Python function returning a 4x4 matrix instead of a floating number like in an equivalent Octave function in a Linear Regression problem I am trying to translate code from Octave to Python, and I am stuck. I am aware they are libraries out there such as scikit-learn etc., but for my own learnin,g I would like to be able to implement the cost function from scratch. Furthermore, I managed to solve some issues by looking into numpy documentation, but I am still stuck with one last one. The function doesn't return the expected result.
I am providing everything you need to reproduce below. Nothing more is needed.
The python function returns a 4x4 matrices instead of a float like it does in Octave and I get this:
Actual results: an numpy.ndarray
Expected results: a floating number

The function in computeCost(X, y, theta) in Octave:

function J = computeCost(X, y, theta)
    m = length(y);
    J = (((X*theta)-y)'*((X*theta)-y))/(2*m);
end

The function compute_cost(X, y, theta) in Python:

from numpy import dot

def compute_cost(X, y, theta):
    m = len(y)
    ans = (X * theta) - y
    product = dot((1/(2*m)), dot(ans, ans.T))
    return product

The tests:

import unittest
import numpy as np
from ml.ml_utils import compute_cost

class TestComputeCost(unittest.TestCase):
    data = np.loadtxt('../data/data1.txt', delimiter=',')
    x = data[:, 0]
    y = data[:, 1]
    m = len(y)
    X = np.vstack((np.ones(m), x))

    def test_compute_cost_with_theta_zeros(self):
        theta = np.zeros((2, 1), dtype=int)
        result = compute_cost(self.X, self.y, theta)
        self.assertEqual(round(result, 2), 32.07,
                         'Result is wrong!')

    def test_compute_cost_with_theta_values(self):
        theta = np.array([[-1], [2]])
        result = compute_cost(self.X, self.y, theta)
        self.assertEqual(round(result, 2), 54.24,
                         'Result is wrong!')

if __name__ == '__main__':
    unittest.main()

data1.txt

6.1101,17.592
5.5277,9.1302
8.5186,13.662
7.0032,11.854
5.8598,6.8233
8.3829,11.886
7.4764,4.3483
8.5781,12
6.4862,6.5987
5.0546,3.8166
5.7107,3.2522
14.164,15.505
5.734,3.1551
8.4084,7.2258
5.6407,0.71618
5.3794,3.5129
6.3654,5.3048
5.1301,0.56077
6.4296,3.6518
7.0708,5.3893
6.1891,3.1386
20.27,21.767
5.4901,4.263
6.3261,5.1875
5.5649,3.0825
18.945,22.638
12.828,13.501
10.957,7.0467
13.176,14.692
22.203,24.147
5.2524,-1.22
6.5894,5.9966
9.2482,12.134
5.8918,1.8495
8.2111,6.5426
7.9334,4.5623
8.0959,4.1164
5.6063,3.3928
12.836,10.117
6.3534,5.4974
5.4069,0.55657
6.8825,3.9115
11.708,5.3854
5.7737,2.4406
7.8247,6.7318
7.0931,1.0463
5.0702,5.1337
5.8014,1.844
11.7,8.0043
5.5416,1.0179
7.5402,6.7504
5.3077,1.8396
7.4239,4.2885
7.6031,4.9981
6.3328,1.4233
6.3589,-1.4211
6.2742,2.4756
5.6397,4.6042
9.3102,3.9624
9.4536,5.4141
8.8254,5.1694
5.1793,-0.74279
21.279,17.929
14.908,12.054
18.959,17.054
7.2182,4.8852
8.2951,5.7442
10.236,7.7754
5.4994,1.0173
20.341,20.992
10.136,6.6799
7.3345,4.0259
6.0062,1.2784
7.2259,3.3411
5.0269,-2.6807
6.5479,0.29678
7.5386,3.8845
5.0365,5.7014
10.274,6.7526
5.1077,2.0576
5.7292,0.47953
5.1884,0.20421
6.3557,0.67861
9.7687,7.5435
6.5159,5.3436
8.5172,4.2415
9.1802,6.7981
6.002,0.92695
5.5204,0.152
5.0594,2.8214
5.7077,1.8451
7.6366,4.2959
5.8707,7.2029
5.3054,1.9869
8.2934,0.14454
13.394,9.0551
5.4369,0.61705

Change the compute_cost function to the following to match dimensions for matrix multiplications:
def compute_cost(X, y, theta):
    m = len(y)
    ans = np.dot(X.T, theta).T - y
    product = np.dot((1/(2*m)), np.dot(ans, ans.T))
    return product[0, 0]

Result for the data given:
32.0727338775
54.242455082

PS: one can simplify the transpose operations further..


Is the PR AUC invariant under label flip? The ROC-AUC curve is invariant under a flip of the labels. I don't know if its a famous result so I will give the proof below. My question is if the PR-AUC curve also has this property. I have not been able to prove or disprove it yet.
The reason this is important is that when data sets are highly imbalanced ROC-AUC curves perform well mechanically and it is better to look at PR-AUC curves. This obviously raises the question of a better curve for PR-AUC for one class being better for the other class as well or not.
=== Proof that ROC-AUC is invariant under label flip ====
$$
\begin{array}{|c|c|c|}
\hline
&predicted~positive & predicted~negative \\
\hline
\hline
Positives & TP(\xi) & FN(\xi) \\
\hline
Negatives & FP(\xi) & TN(\xi) \\
\hline
\end{array}
$$
where $\xi \in [0,1]$ is the output of a classifier.
For a fixed dataset we have
$$
TP(\xi) + FN(\xi) = P \\
FP(\xi)+TN(\xi) = N
$$
where $P$ and $N$ are actual number of negatives and positives. The ROC-AUC defined wrt to this definition of positives and negatives is
$$
A = \frac{1}{PN} \int_0^1 TP(\xi)~d FP(\xi)
$$
Now lets look at what happens under a label flip. We get
$$
\begin{eqnarray}
\tilde A &=& \frac{1}{PN} \int_1^0 TN(\xi)~d FN(\xi) \\
&=& - \frac{1}{PN} \int_1^0 (N-FP(\xi)~d TP(\xi) \\
&=&  \frac{1}{PN} \int_0^1 (N-FP(\xi)~d TP(\xi) \\
&=& \frac{1}{P} TP(\xi) |_0^1 - \frac{1}{PN} (TP(\xi))(FP(\xi))|_0^1 + A \\
&=& A
\end{eqnarray}
$$
where the last equality follows from $TP(0)=0$, $TP(1)=P$ and $TN(1)=N$.

I propose to look at this a bit differently. The different classification status of the two classes are related as follows:

$TP_1=TN_0$
$FP_1=FN_0$
$FN_1=FP_0$
$TN_1=TP_0$

From there we have:
$$FPR_0=\frac{FP_0}{FP_0+TN_0}=\frac{FN_1}{FN_1+TP_1}=1-\frac{TP_1}{FN_1+TP_1}=1-TPR_1$$
and naturally $FPR_1=1-TPR_0$ for the same reasons. Therefore when one switches the labels on a ROC curve, one obtains a mirror image of the curve using the top-left to bottom-right diagonal as the symmetry axis. It follows that the AUC is identical.
However this does not hold with a PR curve, because there is no such symmetry between precision and recall of the two classes. A counter-example should be enough to prove this:
Suppose a test set with 100 instances, 90/10 of which have true class 0/1. Let's assume the classifier predicts all the instances as class 0 except 2 instances as class 1, one true 0 and one true 1.
$$
\begin{array}{|c|c|c|}
\hline
&predicted~0 & predicted~1 \\
\hline
\hline
true~0 & 89 & 1 \\
\hline
true~1 & 9 & 1 \\
\hline
\end{array}
$$
Thus we have:
$P_0=0.92, R_0=0.99, P_1=0.5, R_1=0.1$
[edited] Assuming there is only this point on the PR curve and the two following extreme cases:

Predict every instance as class 0: $P_0=0.9, R_0=1, P_1=NaN, R_1=0$
Predict every instance as class 1: $P_0=NaN, R_0=0, P_1=0.1, R_1=1$

Note: I'm not totally sure about the proper way to deal with $P=NaN$ but I think it's ok to assume P=1 in order to put these points on the curve (this is not essential to the point anyway).
It's easy to see that the AUC is very different whether the focus class is 0 or 1:

In the former case the curve is entirely in the top part of the graph and the area is almost the maximum.
In the latter the curve is mostly in the bottom part of the graph and the area is less than 0.5.


Preparing custom dataset for object detection using ML Seeking clarity on single class object detection model using ML. I have prepared a custom database for this purpose up to 400 images which is split in 80%-20% as training and testing data-set. These are top view only images. The data collection followed the basic guidelines provided at here.

The objective now is to detect the zebra crossing in below contexts.

Model is failing in terms of accuracy. Although it identifies the class but fails to localize correctly in some contexts(greater extent).The desirable result is in blue whereas the model throws up red.

What changes are required to training dataset to rectify this?

EDIT1:-
Tensorflow object detection API is used for this task. Detection accuracy attained is above 90%. Looking forward for suggestions to fix Localisation issue.

EDIT2:-
The illustrations here are only for outlining the issue.All my images are real world pics.

I have the following suggestion:

The size of your data-set is small, you should increase the sample size. You also have to use data augmentation methods for improving the performance, translation, orientation and blurring are highly recommended.
For your task I guess the most popular approach is what which is discussed in YOLO paper, and it is highly recommended using that if you want to have object detection not just localization and recognition for good performance.
I also suggest you providing real data for your task. The images of first row are artificially made. You have to provide real data which your classifier is going to face. The distribution of your test time should be like validation and train time if you want to have good performance.


Implement MLP in tensorflow There are many resources online about how to implement MLP in tensorflow, and most of the samples do work :) But I am interested in a particular one, that I learned from https://www.coursera.org/learn/machine-learning. In which, it uses a cost function defined as follow:

$
J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ -y_k^{(i)} \log((h_\theta(x^{(i)}))_k - (1 - y_k^{(i)}) \log(1 - (h_\theta(x^{(i)}))_k \right]
$

$h_\theta$ is the sigmoid function.

And there's my implementation:

# one hidden layer MLP

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1))

W_out = tf.Variable(tf.random_normal([512, 10]))
y_ = tf.matmul(h1, W_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_) - (1 - y) * tf.log(1 - y_), 1)
loss = tf.reduce_mean(cross_entropy)
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
    s.run(tf.initialize_all_variables())

    for i in range(10000):
        batch_x, batch_y = mnist.train.next_batch(100)
        s.run(train_step, feed_dict={x: batch_x, y: batch_y})

        if i % 100 == 0:
            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
            print('step {0}, training accuracy {1}'.format(i, train_accuracy))

I think the definition for the layers are correct, but the problem is in the cross_entropy. If I use the first one, the one got commented out, the model converges quickly; but if I use the 2nd one, which I think/hope is the translation of the previous equation, the model won't converge.

You made three mistakes:

You omitted the offset terms before the nonlinear transformations (variables b_1 and b_out). This increases the representative power of the neural network.
You omitted the softmax transformation at the top layer. This makes the output a probability distributions, so you can calculate the cross-entropy, which is the usual cost function for classification.
You used the binary form of the cross-entropy when you should have used the multi-class form.

When I run this I get accuracies over 90%:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets('/tmp/MNIST_data', one_hot=True)

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
b_1 = tf.Variable(tf.random_normal([512]))
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1) + b_1)

W_out = tf.Variable(tf.random_normal([512, 10]))
b_out = tf.Variable(tf.random_normal([10]))
y_ = tf.nn.softmax(tf.matmul(h1, W_out) + b_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_), 1)
loss = tf.reduce_mean(cross_entropy)
train_step = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
    s.run(tf.initialize_all_variables())

    for i in range(10000):
        batch_x, batch_y = mnist.train.next_batch(100)
        s.run(train_step, feed_dict={x: batch_x, y: batch_y})

        if i % 1000 == 0:
            train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
            print('step {0}, training accuracy {1}'.format(i, train_accuracy))


What's the default Scorer in Sci-kit learn's GridSearchCV? Even if I don't define the scoring parameter, it scores and makes a decision for best estimator, but documentation says the default value for scoring is "None", so what is it using to score when I don't define a metric or list of metrics?

From the User Guide:

By default, parameter search uses the score function of the estimator to evaluate a parameter setting. These are the sklearn.metrics.accuracy_score for classification and sklearn.metrics.r2_score for regression...


How to interpret classification report of scikit-learn? As you can see, it is about a binary classification with linearSVC. The class 1 has a higher precision than class 0 (+7%), but class 0 has a higher recall than class 1 (+11%). How would you interpret this?
And two other questions: what does "support" stand for? The precision and recall scores in the classification report are different compared to the results of sklearn.metrics.precision_score or recall_score. Why is that so?

The classification report is about key metrics in a classification problem.

You'll have precision, recall, f1-score and support for each class you're trying to find.

The recall means "how many of this class you find over the whole number of element of this class"
The precision will be "how many are correctly classified among that class"
The f1-score is the harmonic mean between precision & recall
The support is the number of occurence of the given class in your dataset (so you have 37.5K of class 0 and 37.5K of class 1, which is a really well balanced dataset.

The thing is, precision and recall is highly used for imbalanced dataset because in an highly imbalanced dataset, a 99% accuracy can be meaningless.

I would say that you don't really need to look at these metrics for this problem , unless a given class should absolutely be correctly determined.

To answer your other question, you cannot compare the precision and the recall over two classes. This only means you're classifier is better to find class 0 over class 1.

Precision and recall of sklearn.metrics.precision_score or recall_score should not be different. But as long as the code is not provided, this is impossible to determine the root cause of this.


Predicting household energy consumption? I have a fairly simple dataset of energy consumption values generated every half hour. I want to train a model to predict the energy consumption at a particular time.

How do I model time values?

At first sight, the total acumulated energy consumption seems to have a linear relation with time, so I suggest to try a linear regression at first. There are several libraries you can use to code it. I recommend you do it with pandas and sklearn, here is an answer related to this: answer.

If the relation is not linear, so I could try with a more complex model (but I suggest to keep simplicity at first). Since you are trying to predict a temporal serie, I would try with an LSTM model. Here is a tutorial to implement an LSTM neural network with keras.