text:
Solution to classify product names
I have a bunch of training data for classifying product names, around 30,000 samples. The task is to classify these product names into types of product, around 100 classes (single words).
For example:
dutch lady sweetened uht milk => milk
samsung galaxy note 10        => electronics
cocacola zero                 => softdrink
...

All words in inputs are indexed to numbers, and so classes. I've tried to use tf.estimator.DNNClassifier to classify them but no good results. The outcome is just an accuracy of 4% which is no meaning.
Should it be I'm in the case that classes (Y values) are distributed kinda randomly and too hard to do multi-time linear separation?
Are there any existing solutions to classify a list of names? like my product names?

If you're looking for an existing solution, the best approach I found was using a TF-IDF model, check out the links below which have similar examples which should be easily adapted for your dataset.
https://www.kaggle.com/selener/multi-class-text-classification-tfidf#targetText=Text%20classification%20(multiclass)&targetText=With%20the%20aim%20to%20classify,one%20of%20the%20product%20categories).
https://github.com/susanli2016/Machine-Learning-with-Python/blob/master/Consumer_complaints.ipynb
However if you specifically want to go for a DNN approach, there are a few options you can take for a multi-class text classification. Try looking into a simple CNN classifier, which is a relatively lightweight approach, computationally speaking, yet showing pretty good results:
https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f
Alternatively, you can use a word2vec or a doc2vec model to map your sentences to unique vectors, and then put them through a regression algorithm.
https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568


text:
Train on big dataset (1mil + images)
I am in the process of collecting a huge dataset of Human poses captured images to create a model to classify poses.
My question is how will I be able to train on this massive dataset? I have multiple GPUs and Multiple machines access (Also have GCP).
What would be the best way to train on such huge dataset?
Thanks.

Welcome to Ai Community. Please read and follow the posting guidelines in the help documentation, as suggested when you created this account. Minimal, complete, verifiable example applies here. We cannot effectively help you until you post a sneak peak of  what you have tried out and accurately specify the problem.Sometimes,we leave out questions unanswered due to lack of a brief but descriptive summary of the problem as the title of the question might say.

is your data stored in raw ASCII text, like a CSV file?
Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.
There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.
Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.
In some cases, you may need to resort to a big data platform.
That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.

My data is in CSV having two columns, img_path and label. Image_path points to an actual image on which I have to train my CNN Model. So loading millions of images in memory is the main issue.

how did u label your images ? did u use any tool or software ?.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why do I get a straight line as an output from a neural network?
I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:

Data I use are typical standardised tabular numbers. The architecture is as follows:
model.add(Dropout(0.2))
model.add(Dense(units=512, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(0.3))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=128, activation='relu'))
model.add(Dense(units=1))

adam = optimizers.Adam(lr=0.1)

model.compile(loss='mean_squared_error', optimizer=adam)

reduce_lr = ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.9,
    patience=10,
    min_lr=0.0001,
    verbose=1)

tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))

history = model.fit(
    x_train,
    y_train,
    epochs=500,
    verbose=10,
    batch_size=128,
    callbacks=[reduce_lr, tensorboard],
    validation_split=0.1)

It seems to me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value, but I am not sure.
I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?

Have you tried experimenting with a lower learning rate as a starting point? You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.

I have now and it worked out! Lowering initial learning rate to 0.005 eliminated straight line as an output. You can post it as an answer :) Thanks!

and if you have some idea why this happens so, with greater learning rate, please do share as well.

You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Can Neural Networks be considered as "Strong AI"?
I've been reading on the differences between "Strong" and "Weak "AI.
I was wondering, where do Neural Networks (especially deep ones) fall in this spectrum? Can they be considered "Strong AI"? If not, is there any model that can be considered "Strong AI"?

In this excerpt there is something very misleading

The principle behind Weak AI is simply the fact that machines can be made to act as if they are intelligent. For example, when a human player plays chess against a computer, the human player may feel as if the computer is actually making impressive moves. But the chess application is not thinking and planning at all. All the moves it makes are previously fed in to the computer by a human and that is how it is ensured that the software will make the right moves at the right times.

Intelligent systems do not in general make use of a look up table. For example, the first chess computer Deep Blue made use of complex evaluation functions.
Neural networks could be trained to act as these evaluation functions.
Additionally, there are planning agents - this is a very large sub-field in AI.
Below again the article mistakes all artificial agents for simple deterministic Turing-like machines

Weak AI is focused towards the technology which is capable of carrying out pre-planned moves based on some rules and applying these to achieve a certain goal but, Strong AI is based on coming up with a technology that can think and function very similar to humans.

A highly recommended book is "Artificial Intelligence: A modern Approach." This reference covers many important sub-fields of AI and provides enough information to realize when people have mistaken the operations of their laptop as the final measure of artificial agency.
Here Comes the hand waving
The strong vs weak AI dichotomy is fueled by what appears to be a fundamental question: Is there something "special" about humanity that sets us apart from the rest of the material realm?
If we are just a complex arrangement of matter with no metaphysical properties then in theory we only need to solve two problems for strong AI:

Powerful enough hardware
Knowing how to correctly assemble said hardware

Justification of 1 & 2:
In the absence of metaphysical properties, we know that a correct arrangement of matter will result in the epiphenomenal emergence of human like experience - this idea is strongly supported by Big Bang theory and the Theory of Evolution.
Thus, all that is left is finding how to correctly arrange matter (hardware) so as to ensure said emergence.
Though, this philosophical thread quickly descends into the rabbit hole where we must tread softly when discussing whether a synthesized agent actually has qualia. Subjectivity is arguably the fundamental characteristic of human experience and we can't even assure ourselves that another human has this - we just accept that they do.
I hope that I have clarified where the science and philosophy stand.


text:
How does an LSTM output the correct dimensions for classes?
Take the below LSTM:
input: 5x1 matrix
hidden units: 256
output size (aka classes, 1 hot vector): 10x1 matrix

It is my understanding that an LSTM of this size will do the following:
$w_x$ = weight matrix at $x$
$b_x$ = bias matrix at $x$
activation_gate = tanh($w_1$ $\cdot$ input + $w_2$ $\cdot$ prev_output + $b_1$)
input_gate = sigmoid($w_3$ $\cdot$ input + $w_4$ $\cdot$ prev_output + $b_2$)
forget_gate = sigmoid($w_5$ $\cdot$ input + $w_6$ $\cdot$ prev_output + $b_3$)
output_gate = sigmoid($w_7$ $\cdot$ input + $w_8$ $\cdot$ prev_output + $b_4$)
The size of the output of each gate should be equal to the number of hidden units, ie, 256. The problem arrises when trying to convert to the correct final output size, of 10. If the forget gate outputs 256, then it is summed with the element wise product of the activation and input gate to find the new state, this will result in a hidden state of size 256. (Also in all my research I have not found anywhere whether this addition is actually addition, or simply appending the two matrices).
So if I have a hidden state of 256, and the output gate outputs 256, doing an element wise product of these two results in, surprise surprise, 256, not 10. If I instead ensure the output gate outputs a size of 10, this no longer works with the hidden state in an element wise product.
How is this handled? I can come up with many ways of doing it myself, but I want an identical replica of the basic LSTM unit, as I have some theories I want to test, and if it is even the slightest bit different it would make the research invalid.

I found this Quora answer to be helpful in understanding the dimensions at each point in the LSTM cell. Dimensions of LSTM

Thankyou, this is a good source. So from this it leads me to believe that the output of an LSTM must be equal to the number of hidden units multiplied by the batch size? This doesn't sound right

After some more research, I have found the answer.
An LSTM is comprised of 1 LSTM cell that is continuously updated by passing in new inputs, the hidden state and the previous output. All nodes inside the LSTM cell are of size hidden_units. That means the output of the activation gate, input gate, forget gate and output gate are all of size hidden_units. As an extension of this, the size of the hidden state is also equal to hidden_units.
The problem arises when the desired output is not equal to that of hidden_units. This is fixed simple and easily, by slapping a basic feed forward net onto the end of the output of the LSTM, so even if it outputs 256, you can convert it to n nodes. These final output nodes are typically put through a softmax with cross-entropy loss applied.
I came to the conclusion by looking at the code included here: https://towardsdatascience.com/lstm-by-example-using-tensorflow-feb0c1968537
In which you can see the final output has its own separate weights and bias'.


text:
Class imbalance and "all zeros" one-hot encoding?
I tried this example for a multi class classifier, but when looking at the data I realized two things:

There are many examples of "all zeros" vectors, that is, messages that don't belong in any classification.
These all-zeros are actually the majority, by far.

Is it valid to have an all-zeros output for a certain input? I would guess a Sigmoid activation would have no problems with this, by simply not trying to force a one out of all the "near zero" outputs.
But I also think an "accuracy" metric will be skewed too optimistically: if all outputs are zero 90% of the time, the network will quickly overfit to always output 0 all the time, and get 90% score.

are you sure that this is not an effect of processing mistake to have all zeros?

also if you have all zeros in training data so maybe add additional column that will be = 1 when this situation happens else 0

I understand your questions as follows:
Is it valid to have an all-zeros output for a certain input?
Yes its possible in some cases like:

In the tutorial "jigsaw-toxic-comment-classification-challenge", the data is taken from wikipedia comments, so generally there is no toxic behavior in people's comment, it is very rare that a person posts "bad" comments in an informative source like wikipedia, this might lead to all the labels being zero for a particular example.
In a single label classification like predicting whether a person has a rare disease,the dataset would contain labels that are mostly "zero"-(no disease)-skewed towards zero

This happens in datasets where the positive output to be predicted is a very rare case.
I guess your second question is -whether "accuracy" is a good way of evaluating a model for this type of a problem
You are right, in such a case even a simple program/model that outputs zeros for all the inputs would achieve an accuracy of more than 90%, so here accuracy is not a good metric to evaluate the model on.
You should go through the metrics f1_score,recall, and precision which are ideal for this type of problems.
Basically here we are interested in "out of those which are predicted positive, how many are really positive -precision" and "out of those which should have been predicted positive, how many are really predicted positive"
If my definitions seem confusing to understand, please got through the link below:
f1_score/recall/precision
Hope I was helpful


text:
How would an AI learn idiomatic phrases in a natural language?
After an AI goes through the process described in How would an AI learn language?, an AI knows the grammar of a language through the process of grammar induction. They can speak the language, but they have learned formal grammar. But most conversations today, even formal ones, use idiomatic phrases. Would it be possible for an AI to be given a set of idioms, for example,

Immer mit der Ruhe

Which, in German, means 'take it easy' but an AI of grammar induction, if told to translate 'take it easy' to German, would not think of this. And if asked to translate this, it would output

Always with the quiet

So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?

Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation should handle this. I don't an AI knows grammar of a language - a translation AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora  with examples of idioms - https://github.com/facebookresearch/MUSE

Can you make this into an answer?

Short answer: Yes.
TL;DR
In the presence of good datasets this can be accomplished with a pipeline.
Long Answer
In reality an idiom is a series of words which is supposed to have a semantic meaning that is not denoted by the literal reading (source). This means that any system that is used must be capable of considering multiple words at a time. Additionally, some idioms are context dependent. Example:

The fisherman broke the ice with his tool.

Are we to believe that this is a very suave fisherman?

So, it is possible to teach an AI to use idiomatic phrases to keep up with the culture of humans?

Observe that humans do not come linguistically "pre-loaded" with idioms. So we can safely assume that idiom usage is a learning task and that the only way for them to keep up is for them to keep learning. So if we solve the idiom learning task we just need to keep our agent online or periodically retrain it on nascent corpora.
One difficulty is that, in the absence of a label, a metaphor could be easily mistaken for an idiom and vice versa. So semantic outlier (sorry it's not free) approaches  may suffer from precision issues. Example:

She's a thorny wildflower (metaphor - could easily be an idiom)
She's a diamond in the rough (idiom - could easily be a metaphor)

Though, idioms will most likely be repeated if a dataset is large whereas a "custom metaphor" is less likely to repeat.
Additionally, some idioms (eg bite the bullet or break a leg) do not have readily available "interpretable semantics" that allow us to extract their intended meaning. For example, if one did not know the idiom "cut me some slack" one could think:
"Slack implies loosening or to make less tight/taut. I was being very uptight. They probably want me to loosen up and not be so critical."
Of course the human understanding of it might happen in a flash and not follow such a delineated path. The idea is that some NLP pipeline might be constructible that satisfactorily handles idioms in some specific use cases (example of a pipeline). For example, one module might attempt to process outliers like "diamond in the rough" which have said interpretable semantics. Though, something like "bite the bullet" may have to be labelled with the correct semantics.
I've only scratched the surface of this. Natural language understanding is already a hard problem - and idioms are thus a tough task in a tough task. I hope that this motivates the reading of some more thorough articles. I have gathered some articles that can be used as a springboard into the literature.
Here's a source that uses a dictionary type approach to train the model to recognize idioms. Excerpt:

For identification, we assume data of the form ${(⟨p_i,d_i⟩,y_i) : i = 1...n}$ where $p_i$ is the phrase associated with definition $d_i$ and $y_i ∈ \{literal, idiomatic\}$.

This source provides pseudo-code for idiom extraction.
This source describes a dataset to help solve the idiom difficulties.

Natural language processing is already a hard problem, to be more precise, I would say natural language understanding is a hard problem.

@nbro Indeed. Thanks for the input. I want my answers to be as precise as possible. see edit

Do you have access to parallel corpora in source and target language that translates idioms correctly? Neural machine translation.  (NMT) should handle this. NMT uses deep learning to match sequences/pairs of words in one language to another and is now the state of the art method for translation AI.
I don't think an AI knows grammar of a language. A translating AI knows patterns but not necessarily grammar in the sense that we learn in school as children. Here's a potential approach that should work give a large enough corpora with examples of idioms - github.com/facebookresearch/MUSE


text:
Will a .h5 file trained with Xception model work with Resnet50?
I have been running my 2013 server box since 2 weeks ago for training an AI model.
I set up 30 epochs to run but since than it only ran 1 epoch as my PC config is super slow. But it generates 1 .h5 file.
My question is will this .h5 file that I trained with Xception model work for Resnet 50?

If I understand your question correctly, you are asking whether you could load the saved weights of a trained model with the Xception architecture on a a Resnet50 architecture.
Short answer: No
Long answer: Xception and Resnet50 have very different architectures.
Here is a paper comparing multilple CNNs including Xception and Resnet50:
https://www.researchgate.net/publication/330478807_Deep_Feature-Based_Classifiers_for_Fruit_Fly_Identification_Diptera_Tephritidae/figures?lo=1
As you can see, the architectures are quiet different between Xception and ResNet50. Thus when you train a model, you are changing the weights and bias of the different layers of the model. If you switch models, and they are similar like let's say VGG16 and VGG19, you could import part of the weights for the layers which are similar between the two models.
As far as I know, I don't know of a function which can do this operation in tensorflow or keras.
But in your case, the architectures are extremely different between Xception and ResNet and there seem not to have any layers in common.
In general, one train a model, save the weights of the training, import back the weights, and use the same model for the next round of training/testing/predictions.
Hope that helps!

Awesome happy to hear that! I hope other people also try to answer the question so we get more insights.

exactly~! let's see whether someone could come up with different ideas :)

btw, what do you call the trained .h5 file? the saved wrights fill?? do we have some shorter names?

Either the saved weights file (in tensorflow): model.save_weights("yourfilename.h5")
Or the saved model:
model.save("yourfilename.h5"). The saved model contains all the layers, weights, biases and the optimizer features.


text:
Accuracy too high too fast?
I have a simple text classifier, with the following structure:
    input = keras.layers.Input(shape=(len(train_x[0]),))

    x=keras.layers.Dense(500, activation='relu')(input)
    x=keras.layers.Dropout(0.5)(x)
    x=keras.layers.Dense(250, activation='relu')(x)
    x=keras.layers.Dropout(0.5)(x)
    preds = keras.layers.Dense(len(train_y[0]), activation="sigmoid")(x)

    model = keras.Model(input, preds)

When training it with 300,000 samples, with a batch size of 500, I get an accuracy value of .95 and loss of .22 in the first iteration, and the subsequent iterations are .96 and .11.
Why does the accuracy grow so quickly, and then just stop growing?

By any chance you are running it on a Jupyter notebook, can you share more of your code, especially about the training part ?

I'm not. I'm new to this. I don't even know what a jupyter notebook is :P

Without more information I don't think we can help you. A few things I can thinkg of: First of all, are you sure you mean iteration and not epoch? Secondly, are your data balanced (i.e. same number of samples in each class)? If not it is possible, if let's say 95% of your samples belong to a single class that your model is predicting this class only and achieving 0.95 accuracy. Finally, it is possible that it is just a very simple task and your model actually scores so high.

It actually depends on a couple of things here -

How many output classes do you have? If you have only 2 or 3 classes, it is a very easy task for the classifier that you have built. So, it is highly possible that convergence has occurred.
As @Djib2011 mentioned already, if your input training set is not balanced and is heavier with one of the output classes (95%), then this accuracy you see makes sense but note that your model won't do well in production.
Do not try to evaluate your model on the basis of your training accuracy. Test it on data your model has never seen before and then evaluate the classification accuracy making sure that your training/testing data is not heavy with one of the classes.

As you have trained your model in batch_size of 500. Weights has been updated for each batch therefore 600 times(300000/500) by the end of one epoch.
So, Your model generalized well. Check the predictions. If Predictions are well. Your model is ready.

It can be normal and there might be nothing wrong with your model. If there is a very strong and clear correlation in your data(good separability) then a network can achive very high accuracy very fast. After reaching some value learning gets harder.


text:
Generate credit cards dataset for locating number region
Currently I'm working on a project for scanning credit card and text extraction from cards. So first of all I decided to preprocess my images with some filters like thresholding, dilation and some other stuff. But it was not successfully for OCR of every credit cards. So I learned a lot and I found a solution like this for number plate recognition that is very similar to my project.
In the first step I want to generate a random dataset like my cards to locate card number region, and for every card that I've generated I cropped two images that one of them has numbers and another has not. I generated 2000 images for every cards.
so I have some images like this:
 (does not have numbers)
 (has numbers)
And after generating my dataset I used this model with tensorflow to train my network.
    model = models.Sequential()
    model.add(layers.Conv2D(8, (5, 5), padding='same', activation='relu', input_shape=(30, 300, 3)))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(16, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Conv2D(32, (5, 5), padding='same', activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))

    model.add(layers.Flatten())
    model.add(layers.Dense(512, activation='relu'))
    model.add(layers.Dense(2, activation='softmax'))

Here is my plot for 5 epochs.

I almost get 99.5% of accuracy and It seems to be wrong, I think I have kind of overfitting in my data. Does it work correctly or my model is overfitted ? And how can I generate dataset for this purpose ?

I am assuming the question you are asking is how to prevent over-fitting on the maximum accuracy. Your graph does show that your model over-fits.
There is a couple of different methods to prevent over-fitting from happening. You can specify training to stop after a certain amount of epochs. In your case it seems to be 2 or 3 epochs. Take care as a new initialization might require more or less epochs to reach optimal accuracy thus it is required run your model a couple of times to determine the correct amount of epochs.
You can also specify the accuracy you want your model to reach before you stop training, this can be dangerous as a local-minima could be found below your expected accuracy thus resulting in your training to run indefinitely. Also your model could have become more accurate then your expected accuracy but stopped prematurely due to the model reaching your stopping-condition.
You can combine the two. Terminate training once it reach your expected accuracy otherwise terminate it if it reaches a certain amount of epochs. This way you prevent your training from happening indefinitely.
In Tensorflow you would want to build your own custom training seen here
The method you are interested is
epoch_accuracy = tfe.metrics.Accuracy()

In their example they get the accuracy after each epoch, you would want to create your own batches and apply it there.
If you really want to dive deeper, you can implement methods for the model to detect when it starts over-fitting such as looking at the standard deviation of the accuracy across a certain measure (usually epochs but your model starts over-fitting after 5 so its not a good measure in your case, maybe do standard deviation over a certain amount of batches trained) if the accuracy leaves the standard deviation space then it might be good idea to stop training. The problem with this technique is that your model has to over-fit a bit to know it needs to stop. The other problem is that your model might seem it is over-fitting but was just a dip in accuracy and was stopped prematurely. (There is methods around this but can't recall them)
Another technique is to add drop out. This introduce noise and pushes your model out of a local-minima. Also forces nodes that was ignored to be used and optimized. This surprisingly works well but is not a surefire way to prevent over-fitting. This technique is built into Tensorflow and don't need custom training. You can find more reading about Tensorflow dropout here, also they have more techniques on how to overcome over-fitting but their solutions is to change the model and your model seems to be fine.
tf.keras.layers.Dropout(0.2),

These are just some techniques I listed but there is a lot more out there. Unfortunately there is no concrete method to prevent it but steps can be taken to stop well before over-fitting occurs. The first two techniques are mostly used together.

Thanks for your response, yes you are right,  my model is overfitted, I will try your ways for preventing overfitting , and you mean is my dataset okay ? Can I also prevent overfitting with add some noise or blur to my dataset ?

I am not to sure what the impact would be. If I have to predict the outcome it might rip your weights out of a local minima but bring down overall accuracy.


text:
Structure discrepancy of an LSTM?
I've found multiple depictions of how an LSTM cell operates. See 2 below:

and

Each of these images suggest the hidden state is utilised differently. On the top diagram, it is shown that the hidden state is added along with the previous output and current input to both the forget gate and the input gate. The bottom image suggests the input and forget gates are calculated only using the previous output and current input. Which is it?
Also, when the previous output is fed in for the current layer, is this before or after it has been reshaped to the final output size and been put through a softmax?

There are different variants of LSTM, in most ML packages now of days you'll probably see what's shown in the bottom picture. For more details, intuition and motivations please see this paper.
It is not reshaped and there is no softmax layer. That is all done outside the LSTM.


text:
Why isn't my Neural Network based calculator working?
I am playing around with neural networks in Tensorflow and I figured an interesting test would be whether I can write a calculator using a Tensorflow Neural Network.
I started with simple addition and it kinda worked (so given 2, 4 it would get around 5.9 or 6.1).
Then I wanted to add the ability to calculate using "+", "-", and "*".
Here is the code I came up with in the end:
import numpy as np
import tensorflow as tf
from random import randrange

def generate_input(size):
    nn_input = []
    for i in range(0,size):
        symbol = float(randrange(3))
        nn_input.append([
                float(randrange(1000)),
                float(randrange(1000)),
                1 if symbol == 0 else 0,
                1 if symbol == 1 else 0,
                1 if symbol == 2 else 0,
                ])
    return nn_input

def generate_output(input_data):
    return [[generate_single_output(i)] for i in input_data]

def generate_single_output(input_data):
    plus = input_data[2]
    minus = input_data[3]
    multiplication = input_data[4]

    if (plus):
        return input_data[0] + input_data[1]

    if (minus):
        return input_data[0] - input_data[1]

    if (multiplication):
        return input_data[0] * input_data[1]

def user_input_to_nn_input(user_input):
    symbol = user_input[1]
    return np.array([[
            float(user_input[0]),
            float(user_input[2]),
            1 if symbol == '+' else 0,
            1 if symbol == '-' else 0,
            1 if symbol == '*' else 0,
            ]])

if __name__ == '__main__':
    model = tf.keras.models.Sequential([
        tf.keras.layers.Dense(64, activation='relu', input_shape=(5,)),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(1),
        ])

    model.compile(tf.keras.optimizers.RMSprop(0.001), loss=tf.keras.losses.MeanSquaredError())

    input_data = np.array(generate_input(10000))
    output_data = np.array(generate_output(input_data))

    model.fit(input_data, output_data, epochs=20)

    while True:
        user_calculation = input("Enter expression (e.g. 2 + 3):")
        user_input = user_calculation.split()
        nn_input = user_input_to_nn_input(user_input)
        print(model.predict(nn_input)[0][0])

The idea is built on this tutorial: https://www.tensorflow.org/tutorials/keras/basic_regression
The input is 5 fields: number 1, number 2, plus?, minus?, multiplication?
Where the last 3 are simply 1 or 0 depending on whether that is the calculation I am trying to do.
As an output for say [1,4,1,0,0] I would expect 1 + 4 = 5
for [1,4,0,1,0] I would expect 1 - 4 = -3 etc.
For some reason though the numbers I am getting are completely off and seem random.
Basically I am trying to understand what I went wrong?
The data being input to the NN seems correct and the model is based on the model used in the tutorial I quoted (and the problems seem fairly similar so I expect if one would work the other would too).

A neural network is not good at selecting a function based on those 3 input parameters, because of the way a neuron is setup.
What you should do is either make a neural network for each operation, or use different input neurons for each operation. E.g. 2 input neurons for the addition operation, 2 for the multiplication, and 2 for the minus. 6 inputs in total of which 4 will always be 0.
This will make it easier for the neural network to calculate the result.

Thanks :) That seems to work to an extent. Now addition and subtraction work but multiplication doesn't work at all. Why exactly isn't a neural network not good at selecting a function the way I had it? Also any idea why multiplication wouldn't be working here?

I looked into it and it seems a simple multiplication function is very hard to approximate for a neural network with certain activation funtions (such as relu). And this is only learning multiplication, not trying to do the other operations in the same network.
link:https://www.sololearn.com/Discuss/1467062/can-a-neural-network-learn-multiplication

Thanks Lustwelpintje, that's what I've discovered too. I'll continue looking into it and see if I can figure out a way to do it


text:
What is the difference between a stationary and a non-stationary policy?
In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.
What is the difference between a stationary and a non-stationary policy? How do you formalize both? Which problems (or environments) require a stationary policy as opposed to a non-stationary one (and vice-versa)?

This Wikipedia article should also be useful.

A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy), or a conditional density, $\pi(A \mid S)$ (a stochastic policy). A non-stationary policy is a policy that is not stationary. More precisely, $\pi_i$ may not be equal to $\pi_j$, for $i \neq j \geq 0$, where $i$ and $j$ are thus two different time steps.
There are problems where a stationary optimal policy is guaranteed to exist. For example, in the case of a stochastic (there is a probability density that models the dynamics of the environment, that is, the transition function and the reward function) and discrete-time Markov decision process (MDP) with finite numbers of states and actions, and bounded rewards, where the objective is the long-run average reward, a stationary optimal policy exists. The proof of this fact is in the book Markov Decision Processes: Discrete Stochastic Dynamic Programming (1994), by Martin L. Puterman, which apparently is not freely available on the web.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Neural network does not give out the required out put?
Made a neural network using tensor flows that was supposed matches an Ip to one of the 7 type of vulnerabilities and gives out what type of vulnerability that IP has.

    model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(50, activation=tf.nn.relu),
  tf.keras.layers.Dense(7, activation=tf.nn.softmax)
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(xs, ys, epochs=500)

The output of print(model.predict([181271844])) when this command is executed should be one of the numbers from 1 to 7 but the out put its gives is

[[0.22288103 0.20282331 0.36847615 0.11339897 0.04456346 0.02391759
    0.02393949]]

I can't seem to figure out what the problem is.

xs is the list of IPs and ys is its corrensponding vulnerability from 0 to 6 (seven in total)

The numbers you are seeing as output are a probability vector. This is a common output format for multi-class classification models.
In this case, you can interpret the vector as saying:

22% chance of class 1
20% chance of class 2
37% chance of class 3
11% chance of class 4
4% chance of class 5
2% chance of class 6
2% chance of class 7

If you want to get a concrete label out of this, the easiest choice is to compute and return the index of the maximum element.


text:
How to change this RNN text classification code to become text generation code?
I can do text classification with RNN, in which the last output of RNN (rnn_outputs[-1]) is used to matmul with output layer weight and plus bias. That is getting a word (class name) after the last T in the time dimension of RNN.
The matter is for text generation, I need a word somewhere in the middle of time dimension, eg.:
t0  t1    t2  t3
The brown fox jumps

For this example, I have the first 2 words: The, brown.
How to get the next word ie. "fox" using RNN (LSTM)? How to convert the following text classification code to text generating code?
Source code (text classification):
import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0      t1      t2
british gray    is => cat (y=0)
0       1       2
white   samoyed is => dog (y=1)
3       4       2
'''
Bsize = 2;
Times = 3;
Max_X = 4;
Max_Y = 1;

X = [[[0],[1],[2]], [[3],[4],[2]]];
Y = [[0],           [1]          ];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  Y[I][0] /= Max_Y;

#model
Inputs   = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,      1]);

#single LSTM layer
#'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Inputs);
#'''

#multi LSTM layers
'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
]);
Hidden2 = Layers(Inputs);
'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden1,Weight3) + Bias3);

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed = {Inputs:X, Expected:Y};
for I in range(1000): #number of feeds, 1 feed = 1 batch
  if I%100==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed);
print("\nEval:");
print(Results);

print("\nDone.");
#eof

I found out how to switch it (the code) to do text generation task, use 3D input (X) and 3D labels (Y) as in the source code below:
Source code:
import tensorflow as tf;
tf.reset_default_graph();

#data
'''
t0       t1       t2
british  gray     is  cat
0        1        2   (3)  <=x
1        2        3        <=y
white    samoyed  is  dog
4        5        2   (6)  <=x
5        2        6        <=y
'''
Bsize = 2;
Times = 3;
Max_X = 5;
Max_Y = 6;

X = [[[0],[1],[2]], [[4],[5],[2]]];
Y = [[[1],[2],[3]], [[5],[2],[6]]];

#normalise
for I in range(len(X)):
  for J in range(len(X[I])):
    X[I][J][0] /= Max_X;

for I in range(len(Y)):
  for J in range(len(Y[I])):
    Y[I][J][0] /= Max_Y;

#model
Input    = tf.placeholder(tf.float32, [Bsize,Times,1]);
Expected = tf.placeholder(tf.float32, [Bsize,Times,1]);

#single LSTM layer
'''
Layer1   = tf.keras.layers.LSTM(20);
Hidden1  = Layer1(Input);
'''

#multi LSTM layers
#'''
Layers = tf.keras.layers.RNN([
  tf.keras.layers.LSTMCell(30), #hidden 1
  tf.keras.layers.LSTMCell(20)  #hidden 2
],
return_sequences=True);
Hidden2 = Layers(Input);
#'''

Weight3  = tf.Variable(tf.random_uniform([20,1], -1,1));
Bias3    = tf.Variable(tf.random_uniform([   1], -1,1));
Output   = tf.sigmoid(tf.matmul(Hidden2,Weight3) + Bias3); #sequence of 2d * 2d

Loss     = tf.reduce_sum(tf.square(Expected-Output));
Optim    = tf.train.GradientDescentOptimizer(1e-1);
Training = Optim.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Feed   = {Input:X, Expected:Y};
Epochs = 10000;

for I in range(Epochs): #number of feeds, 1 feed = 1 batch
  if I%(Epochs/10)==0:
    Lossvalue = Sess.run(Loss,Feed);
    print("Loss:",Lossvalue);
  #end if

  Sess.run(Training,Feed);
#end for

Lastloss = Sess.run(Loss,Feed);
print("Loss:",Lastloss,"(Last)");

#eval
Results = Sess.run(Output,Feed).tolist();
print("\nEval:");
for I in range(len(Results)):
  for J in range(len(Results[I])):
    for K in range(len(Results[I][J])):
      Results[I][J][K] = round(Results[I][J][K]*Max_Y);
#end for i
print(Results);

print("\nDone.");
#eof


text:
Unable to replicate Figure 2.1 from "Reinforcement Learning: An Introduction"
The author explains in 2.2 Action-Value Methods:

To roughly assess the relative effectiveness of the greedy and $\varepsilon $-greedy methods, we compared them numerically on a suite of test problems. This is a set of 2000 randomly generated n-armed bandit tasks with n = 10. For each action, a, the rewards were selected from a normal (Gaussian) probability distribution with mean Q*(a) and variance 1. The 2000 n-armed bandit tasks were generated by reselecting the Q*(a) 2000 times, each according to a normal distribution with mean 0 and variance 1. Averaging over tasks, we can plot the performance and behavior of various methods as they improve with experience over 1000 plays, as in Figure 2.1. We call this suite of test tasks the 10-armed testbed.

But doing my best, the replication yields something nearer to :

I think I am misunderstanding how the author took the averages.
Here is my code:
from math import exp

import numpy
import matplotlib.pyplot as plt

def act(action, Qstar):
    return numpy.random.normal(Qstar[action], 1)

def run(epsilon):
    history = [0 for i in range(1000)]

    for task in range(1, 2000):
        Qstar = [numpy.random.normal(0, 1) for i in range(10)]
        Q = [0 for i in range(10)]
        for t in range(1, 1001):
            if numpy.random.randint(0, 100) < epsilon:
                action = numpy.random.randint(0, len(Q))
            elif t == 0:
                action = 0
            else:
                averages = [q/t for q in Q]
                action = averages.index(max(averages))

            reward = act(action, Qstar)
            Q[action] += reward

            history[t-1] += reward

    return [elem/2000 for elem in history]

if __name__ == '__main__':
    plt.plot(run(10), 'b', label="ɛ=0.1")
    plt.plot(run(1), 'r', label="ɛ=0.01")
    plt.plot(run(0), 'g', label="ɛ=0")
    plt.xlabel('Plays')
    plt.ylabel('Reward')
    plt.legend()
    plt.show()

You are calculating the average reward for each action (i.e. bandit arm) incorrectly. You cannot calculate this simply with a list comprehension, and you need to keep a second list storing the number of times each action was taken.
The correct calculation is to divide the total reward obtained from each action by the number of times that action was taken. You are dividing all actions by the total number of actions taken. Adding:
...
counts = [0 for i in range(10)]
...
else:
    averages = []
    for i in range(0, 10):
        averages.append(Q[i]/counts[i] if counts[i] > 0 else 0)
    ...
...
counts[action] += 1

results in working code, and I can generate this graph using 200 samples per method, which looks like a noisier version of the one in the figure you referenced:


text:
Why is awareness of itself such a point when speaking about AI?
Why is awareness of itself such a point when speaking about AI? Does it always mean a starting point for apocalyptic nightmares to occur when such a level is reached or is it just a classical example about what could be a really abstract thing that machine cannot easily posses?
I would sleep my nights far more calmfully if the situation was the latter, and I understand the first does not automatically  happen. The main thing I would like to discover is the starting point - which approach came first historically? Or is there another view point in the historical first occurrence of self awareness term?

Chimps, Elephants, Dolphins and other animals are suspected to be self-aware (by measures such as the mirror test). Either that means you will sleep less calmly fearing the dolphin apocalypse, or that there is a more nuanced interpretation of the term than the now classic sci-fi trope of machines "awakening". I'm hoping someone with some background on research in this topic can answer  . . .

https://www.quora.com/Is-self-aware-Artificial-Intelligence-possible has 65 answers claiming whether self awareness can be achieved or not. This is a hot topic and visions of the results vary a lot. I wanted to know what is the origin for such a hype for such a feature of AI.

Neil Slater has it right, there most probably is no fear of AI self awareness as some starting point of evil series of things to happen.
Wikipedia [1] puts self awareness talks to sci-fi section, among stories, not a real thing. Self-awareness is among a list of terms that tries to make machines or aliens similarly human than ordinary people and uses that as a method of story telling.
Self-awareness or other humanlike skills that we posses that the machines don't have yet and will not have in near future can twist minds and make a seed of conspiracy theories, but at least the exhaustive Wikipedia overview on topic did not speak anything about AI.
Maybe the concept of humanlike behaviour materialises on our minds as a term of self-awareness but my source puts the origin to different category.
[1] https://en.m.wikipedia.org/wiki/Self-awareness


text:
How can I teach a computer to play N64 games using Neural Nets?
I would like to work on a project where I teach an NN to play N64 games.
To my current understanding, I would need an emulator?
I can do the Machine Learning side of it, im just unsure how I can give the NN access to the game's controls, such as left, right, up or down?
Where could I find more information on doing so and is using an Emulator the right path to take?

The common options are:

Implement a simple simulation of the game, perhaps without graphics. Train the agent on the simulation. This is usually not very satisfying, because the agent won't play the real game.
Use an emulator that allows you to inspect the game's memory, and input values directly at the controls. BizHawk is a good choice for SNES and I think also for N64 emulation. Use a pipe to connect this program with your agent. Pass across the pipe any key values from the game's memory that represent the state of the game, and send back actions that represent the inputs the agent makes in that state. Usually this is the best option.
Scrape the screen in real time. Tools like SkyScraper will allow you to capture the raw pixels displayed in a window. You should then be able to pass these pixels to your agents via a pipe. You may be able to use tools like Selenium to pass values back from the agent to the game. This route is usually a lot more intensive than option 2, but is the most realistic. It may also require very good hardware to keep up with the game in real time.


text:
Confused with backprop in pytorch with BCE loss
I've a prediction matrix(P) of dimension 3x3 and one-hot encoded label matrix(L) of dimension 3x3 as shown below.
    |0.5 0.3 0.1|      |1 0 0|
P = |0.3 0.2 0.1|  L = |0 1 0|
    |0.2 0.5 0.8|      |0 0 1|

each column in 'P' corresponds to prediction of a label in 'L'

How is the BCELoss calculated using pytorch?, my experimentation by giving these two matrices as parameters to loss function yielded me poor results and pytorch's loss calculation function doesn't disclose on how loss calculation is done for this case.
How is the loss averaged for each instance and across the a batch?
if loss is calculated column wise and averaged for each instance and across the batch, then how can loss be backprop'd in pytorch?

Thanks in advance.

BCELoss ( Binary Cross Entropy Loss) is used for binary classifier, which is a neural network that have a binary output, 0 or 1. It is not used for multi-output neural network like your case. For that kind of networks, you can use MSELoss or CrossEntropyLoss as your loss for the network.
For the calculation of BCE, it is shown on pytorch documentation.
https://pytorch.org/docs/stable/nn.html#BCELoss
For across batch, it either sum the loss or take the mean. you can set that through the reduction="" argument.
As i said, the loss is either summed or taken the mean. Then there will be a single value so pytorch can do backdrop.

Hope I can help you


text:
Is there any measure of separability of classes?
I want to know if there is a measure of how well two classes in Y are separable (linearly or not) based on their features in X. Easiest way of explaining this is to compare it to correlation coefficients, the higher the correlation the higher possiblity for successful regression based on given feature (at least in theory).
Is there any measure that will tell me how well classes are separated based on input data features, before training a ML model?

You can use a different measure for the decision tree. Anyhow, it's an ML model.

Are you thinking something like Information Gain?
Information Gain basically uses the concept of information entropy to determine if splitting a variable is useful.


text:
Why feed actions in later layer in Q network?
I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network:

Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details)

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?
Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?
Is this just a small little tweak or a more significant improvement?

Is this just a small little tweak or a more significant improvement? This seems pretty major.. I tested this with and without this tweak and without it the networks just wouldn't learn at all. But it may depend on the environment too, I'm not sure.

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?

Yes.

Why would you do that? To have the first layer focus on learning the state value independent of the selected action? How would that help?

Neural networks hidden layers learn representations that get progressively closer to a linear relationship with the target, layer by layer.
So the first layer would not be learning the state value per se, but some representation of the state that was better related to the action value at the output.
Neural network architectures are often established by experimentation, so I expect here they tried the idea and the performance was OK. If they do not give alternative architectures in the paper, then the precise reason is not clear.
I can try a few guesses:

Concatenating the actions with the states in the input layer would result in more parameters for the neural network to achieve the same accuracy, running slightly slower. That's because more weights would be required to link the input layer with the first hidden layer.
Separating the layers of state and action inputs is a form of regularisation, as the first layer has to produce features that are useful for all possible actions.
You don't want the neural network to construct a reverse map $\pi(a|s) \rightarrow Q(s,a)$, you want it to independently assess the action values to do its job as a critic. By having the states and actions presented in different laters, this reduces the chance of the neural network finding shortcuts due to "recognising the policy" to predict the values. That's because the state and action pairings may repeat (so be recognised), but the first layer activations change over time (so even with repeated state/action pairs, this is a new representation to learn from).

Is this just a small little tweak or a more significant improvement?

I don't know, and suggest looking through previous work by the same authors in case they describe the approach in more detail.


text:
Can we say: the more we increase the numbers of cross validation the less likely it is that we overfit?
Based on the answer of my previous question:
How can I avoid overfitting when doing parameter tuning?
Can we say: the more we increase the numbers K of cross validation the less likely it is that we overfit?

In general, no.
There is a tradeoff between making the validation set for each fold smaller, and having more folds in total.
As an example, if you have $N$ folds for $N$ datapoints, each fold will have only a single datapoint in its validation set. The validation accuracy of a model on a single datapoint is not a reliable estimator for the test performance of the model. In fact, you can construct examples where the error is arbitrarily large.
For this reason, people sometimes use Bootstrap Validation if they need a very large number of folds. In practice though, most people just us 10 folds, and that's "good enough".


text:
Can we combine Off-Policy with On-Policy Algorithms?
On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: Kinds of RL Algorithms - Spinning up - OpenAI)
Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here an example:

Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm?

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017).
Citing the paper:

The core idea is to use the first-order Taylor expansion of the critic
  as a control variate, resulting in an analytical gradient term through
  the critic and a Monte Carlo policy gradient term consisting of the
  residuals in advantage approximations. The method helps unify policy
  gradient and actor-critic methods: it can be seen as using the
  off-policy critic to reduce variance in policy gradient or using
  on-policy Monte Carlo returns to correct for bias in the critic
  gradient.

The authors provide an open source implementation of it in https://github.com/shaneshixiang/rllabplusplus
There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NIPS 2017).
The Related Work section in both papers might be also worth looking at.

Dear @user5093249, thank you very much for your detailed and definitely interesting answer. I'll definitely have a look at the papers. I'm especially interested in combining existing algorithms (like TD3 and PPO) instead of developing new ones that combine on- and off-policy approaches.

@RayWalker your approach seems to be in line with what the authors suggest. In fact, in the first paper, they state that "since Q-Prop uses both on-policy policy updates and off-policy critic learning, it can take advantage of prior work along both lines of research". They implemented it on top of TRPO-GAE for example.

Ok great, thank you very much for the additional information. I've now had a look at both abstracts and accepted your answer. It very well answers the main question in the topic.  And the Q-Prop paper just jumped to the top of my reading list : )


text:
How to evaluate a Deep Q-Network
Good day, it's a pleasure having joined this Stack.
In my master thesis I have to expand a Deep Reinforcement Learning Network, to be precise a Deep Q-Network, which is used to control machines in an electrical grid for power quality management.
What would be the best way to evaluate if a network is doing a good job during training or not? Right now I have access to the reward function as well as the q_value function.
The rewards consist of 4 arrays, one for each learning criteria of the network. The first tuple is a hard criteria (adherence mandatory) while the latter 3 are soft criteria:
Episode: 1/3000 Step: 1/11 Reward: [[1.0, 1.0, -1.0], [0.0, 0.68, 1.0], [0.55, 0.55, 0.55], [1.0, 0.62, 0.79]]
Episode: 1/3000 Step: 2/11 Reward: [[-1.0, 1.0, 1.0], [0.49, 0.46, 0.67], [0.58, 0.58, 0.58], [0.77, 0.84, 0.77]]
Episode: 1/3000 Step: 3/11 Reward: [[-1.0, 1.0, 1.0], [0.76, 0.46, 0.0], [0.67, 0.67, 0.67], [0.77, 0.84, 1.0]]

The q_values are arrays which I do not fully understand yet. Could one of you explain them to me? I read the official definiton of Q-Values positive False Discovery Rate. Can these values be used to evaluate neural network training? These are the Q-Values for step 1:
Q-Values: [[ 0.6934726  -0.24258053 -0.10599071 -0.44178435  0.5393113  -0.60132784
  -0.07680141  0.97968364  0.7707691   0.57855517  0.16273917  0.44632837
   0.00799532 -0.53355324 -0.45182624  0.9229134  -1.0455914  -0.0765233
   0.37784138  0.14711905  0.10986999  0.08918551 -0.8189287   0.14438646
   0.8869624  -0.43251887  0.7742889  -0.7671829   0.07737591  0.2569678
   0.5102049   0.5132051  -0.31643414 -0.0042788  -0.66071266 -0.18251896
   0.7762838   0.15322062 -0.06284399  0.18447408 -0.9609979  -0.4508798
  -0.07925312  0.7503184   0.6858963  -1.0436649  -0.03167241  0.87660617
  -0.43605536 -0.28459656 -0.5564517   1.2478396  -1.1418368  -0.9335588
  -0.72871417  0.04163677  0.30343965 -0.30024529  0.08418611  0.19429305
   0.44063848 -0.5541725   0.5740701   0.76789933 -0.9621064   0.0272104
  -0.44953588  0.13415053 -0.07738207 -0.16188647  0.6667519   0.31965214
   0.3241703  -0.27273563 -0.07130697  0.49683014  0.32996863  0.485767
   0.39242893  0.40508035  0.3413986  -0.5895434  -0.05772913 -0.6172271
  -0.12423459  0.2693861   0.32966745 -0.16036317 -0.36371914 -0.04342368
   0.22878243 -0.09400887 -0.1134861   0.07647536  0.04724833  0.2907955
  -0.70616114  0.71054566  0.35959414 -1.0539075   0.19137645  1.1948669
  -0.21796732 -0.583844   -0.37989947  0.09840107  0.31991178  0.56294084]]

Are there other ways of evaluating DQNetworks? I would also appreciate literature about this subject. Thank you very much for your time.

could you provide more detail? what is the action space (how many actions) does your problem have?

Q-values represent expected return after taking action $a$ in state $s$, so they do tell you how good it is to take an action in the specific state. Better actions will have larger Q-values. Q-values can be used to compares actions but they are not very meaningful in representing performance of the agent since you have nothing to compare them with. You don't know the actual Q-values so you can't conclude if your agent is approximating well those Q-values or not.
Better performance metric would be the average reward per episode/epoch, or average reward in last $N$ timesteps for continuing tasks. If your agent is improving its performance then it's average reward should be increasing. You said that you have rewards per state and that some of them represent more important criteria then others. You could plot the average reward per episode by doing some kind of weighted linear combination of criteria rewards
\begin{equation}
\bar R = \bar R_1 w_1 + \bar R_2 w_2 + \bar R_3 w_3 + \bar R_4 w_4
\end{equation}
where $\bar R_i$ is the average episode reward for criteria $i$.That way you can provide more importance to some specific criteria in your evaluation.

Thank you for this elaborate answer. I will try out implementing average rewards.


text:
How does the math behind heat map filters work?
I am working on an app that generates heat/ thermal map given a picture. i have been able to get what i expected using python opencv builtin function cv2.applyColorMap(img, cv2.COLORMAP_JET). Everything works exactly as expected. But i want to understand how applyColorMap works at the back end. I am aware how several image filters (like image, edge filters) work by convolution / cross correlation with appropriate kernals, but i can't seem to pull the same concept for color maps.
For this question lets consider a color map where we want :
Brightest ones: (RED COLORED)
MEDIUM INTENSITY ONES: (YELLOW COLORED)
LOW INTENSITY ONES: (BLUE COLORED)
What i have done:
I tried dividing the pixels into 3 categories and replaced each pixel with the either of the colors (RED, YELLOW, BLUE ) depending upon it's value from gray scale image( 0-255).
This approach had a problem that there were solid 3 colors in the image with no variation in intensity of the individual color while in a good heat map there is blend of colors ( it decreases or increases ) based upon the intensity . I want to achieve that effect. I would appreciate any help or any lead to understand how heat maps work .

About your question concerning ColorMaps:
A cv2 ColorMap is basically just a lookup table which directly maps the intensity values of the input image to a predefined RGB color. In its essences it is exactly what you did by categorizing and associating with a specific color value.
Most of the cv2 ColorMaps just have a little more detail, most of them have either 64( "Rainbow", "Hot", ...) or 256 ("Jet", "Magma", ...) steps, some like "Winter" have 11. They come from the GNU Octave or Matlab color palettes.
If you really want to build this lookup table yourself the easiest way to achieve a high level of granularity for the individual steps is to use the HSV color space
import numpy as np
import cv2

N_STEPS = 20
h = np.linspace(0, 180, N_STEPS, endpoint=False) # cv2 Hue range is [0, 179]
s = np.ones_like(h)*255   # cv2 saturation range is from [0, 255]; adjust it to your liking
v = np.ones_like(h)*255   # cv2 value range is from [0, 255]; adjust it to your liking
hsv_colormap = np.dstack([h,s,v])

rgb_colormap = cv2.cvtColor(np.uint8(hsv_colormap), cv2.COLOR_HSV2RGB)

IMG_SIZE = 100
#intensities = np.random.random((IMG_SIZE,IMG_SIZE)) # from 0 to 1
intensities = np.array([np.linspace(0,1, IMG_SIZE, endpoint=False), ]*IMG_SIZE) # from 0 to 1
intensity_indices = np.uint8(N_STEPS*intensities) # map insity ranges to discretized intervals

color_mapped_intensities = rgb_colormap[0,intensity_indices,:]
cv2.imshow('colormap', color_mapped_intensities)

cv2.waitKey(0)

you can play around with the Hue values ranges to feed into the color map
Happy color mapping to you !


text:
How to add more than 1 agent in one generation with Q Learning
Sometimes the agent learns a bit slow and you want to have multiple agents in one generation. And at each episode you'll draw on the screen only the best of them or all of them. How is that possible?
For clarification purposes, please watch this video on youtube at time 4:10.
I need just a theoretical approach, I'll try the coding myself :).
Thanks for any answer! I really do appreciate it! :)

The video you linked is not using reinforcement learning (RL). It is using genetic algorithms (GA).
GA is designed around using multiple agents and picking the best performing to move forward to next generation. With this approach, it is common to want to only view the best performing agents, as the learning mechanism uses the same selection process - the best agent is the output of the algorithm*.
Whilst you can run muliple agents to collect more data easily enough in RL, they typically won't perform better or worse other than by random chance. The best performance is not indicative of the agent's overall performance in the way that GA is, because there is only one agent. It would not be as meaningful to pick out the best ones for display. Instead, after a certain number of training episodes, you should take the agent so far, stop it running exploratory moves (set $\epsilon = 0$ if you are using $\epsilon$-greedy exploration). Then render the behaviour of that agent.
If you want to compare RL versus GA for learning efficiency, then one arguably fair comparison would be to render the agent each time it has trained on a number of episodes in RL equal to the population size used in the GA version. If you are only showing results after every 100 generations of GA, then multiply number of training episodes by 100 to compare using the same amount of data input.
You can also, separately to this concern, run RL with multiple exploring agents at once. If you want to have parallel training with multiple environments running in RL, then you will need a distributed environment. Each instance of the environment would run one agent, and collect training data. You have a rough choice between:

Feeding everything observed into one central experience replay memory and a single training loop routine samples from the whole memory and updates the agent. This should work OK for Q-learning.
Calculate update gradients on each distributed environment, collate them centrally and update the agent on a mean gradient update step before sending out the updated agent to all the distributed systems. This is the approach typically used by A3C and A2C which benefit from having multiple agents running at once.

In both cases, the latest parameters of the agent (the neural network weights) need to be regularly copied out to each environment so that each instance can work as much as possible with a current policy.
Setting up a distributed learning environment for RL is more work than for GA, because you need to move a lot more data between agents to complete learning, whilst for GA you only need to measure fitness. However, you should find that Q learning can be a lot more efficient (in terms of number of simulations required) than a GA-based approach for many control problems.

* As an aside, this can be a weakness of GA if you are running in an environment with random choices or events - the GA can select as it's "best" result at any stage something that is less optimal but that was lucky. On average over many generations this effect should be removed, because the same agent won't be luck every generation, but it does mean in some environments that you will get an over-estimate of performance unless your fitness assessment is very thorough (or perhaps run separately every so many generations, to double-check). Similar concerns occur in RL too, but do not affect which agent you select for assessment or display, since there is only ever one set of agent parameters at any time.

@DuttaA: If your GA finds a policy in an environment, then you can definitley compare the policies for performance. It doesn't matter how you produced a policy at all. Deeper comparisons are harder, which is why it is only a suggestion that a fair-ish approach is to compare policies after considering the same amount of data. Other comparisons might be total CPU (where GAs do better because the update steps are less expensive) or development time etc.

@DuttaA: It is also possible to apply GAs to optimisation problems that are not really suitable for RL, but in this answer I am assuming that there is a control problem suitable for both. You are right GA does not have anything to say about policy functions directly. But provided you can model the function in a way that can be evolved (see NEAT) then a GA can be used to search for optimal results of that function

@DuttaA As Neil says, in a similar fashion to algorithms such as simulated annealing, GAs are really a very general approach for solving optimization problems. You just need some way of comparing one solution to another (e.g. you don't need gradient-based information).

Thank you very much for taking the time to explain this to me! I really do appreciate it! I'll study more in GA and see what's that about more precisely. Thanks a lot! :)


text:
Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi_1$.
I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script. However, when computing the actions on the same test set, i realised the actions were very different.
Is this supposed to be normal when training deep q networks or is there something that I am missing ?
I am using prioritised experience replay when training the model.

Have you measured the expected returns for the two different policies (separately from training runs)? If not, may be worth doing so and adding that detail to the question, as if the returns are similar it points to a different kind of answer to if they were really different

Second clarifying question - are you calculating actions given arbitrary test states? Do those test states actually occur naturally when the agents behave optimally?

not yet because it is not straight forward to measure the value of the learnt policy because i would have to use off policy reinforcement learning evaluation techniques like Weighted importance sampling to do so.

regarding second question, I have trajectories of a behaviour policy, and i split them into training and test sets. After training the Q network only on the train set, I feed the test set into the Q network and obtain the actions via an argmax for each state

@calveen: Is all this because you have one real environment, that your agents are not allowed to act in, and no way to accurately simulate it?

I am implementing reinforcement learning on retrospective healthcare data to discover optimal treatment strategies. Hence it is difficult to simulate the learned target policy on actual patients.

OK, So it would be hard to use weighted importance sampling in any case, because you don't have access to behaviour policy function $b(a|s)$ - and chances are that there are multiple policies in use in your training data. There are off-policy evaluation techniques that don't require importance sampling though. Q-learning used for policy evaluation is one of them.

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):

Neural network weight initialisation
Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
Minibatch sampling from experience replay
Stochastic choices in the environment (does not apply in your case)
Other stochastic parts of training that may be in use, such as dropout regularisation

Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.

I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script.

This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:

Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.
Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.

You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.
Here are a couple of additional tests that may help:

Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.
Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.

If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?
Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.
You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.

Thanks Neil for the tips :-) I realised that it could be different because of prioritised replay sampling and initialisation of random normal weights. I wouldn't be able to believe if those random parameters affect the results :/

can prioritised experience replay be made deterministic ? because the sampling is based on probabilities ?

@calveen: Any "random" process in machine learning can be made deterministic by setting the RNG seeds. That won't necessarily make things better for your end result though. A Q function that is consistently inaccurate has limited use

Just to clarify again, for the 1st test, we compute the q value for the action chosen by the behaviour policy at some state for each run of the script and we see whether the q values for that particular action are similar ?
For the second test, we use the q network from 1st run to compute the q function for the action chosen by the q network from the second run over some states and vice versa. The q values output by these 2 networks should be similar ?

@calveen: Yes. Note those tests I am suggesting are not formal parts of Q-learning theory. They are just metrics that may help you decide whether you have a problem with stability of your learning algorithm.

is there a particular reason why network weight initialisation might cause problems because the algorithm when it converges would have similar weights ? due to gradient descent update

@calveeen: NN initialisation makes a huge difference, and even if the NN predicts the same values, if it was initialised differently the weights are likely to be very different. It is a major source of difference between training runs.


text:
Are there any conferences dedicated to artificial general intelligence?
Similarly to What are the scientific journals dedicated to artificial general intelligence?, are there any conferences dedicated to artificial general intelligence?

There are several conferences dedicated to AGI or human-level intelligence, such as

AGI Conferences (organized by AGI society)
Biologically Inspired Cognitive Architectures (organized by BICA Society)
Advances in Cognitive Systems
IEEE Task Force on Towards Human-like Intelligence

The conferences focus on topics such as cognitive architectures, autonomy, creativity, lifelong learning, and formal models of general intelligence. There are also journals associated with these conferences (see [1], [2], [3]). If you want to know more about them, I suggest that you go to their websites.

In the past, there was also this symposium and this track.


text:
Why is learning $s'$ from $s,a$ a kernel density estimation problem but learning $r$ from $s,a$ is just regression?
In David Silver's 8th lecture he talks about model learning and says that learning $r$ from $s,a$ is a regression problem whereas learning $s'$ from $s,a$ is a kernel density estimation. His explanation for the difference is that if we are in a stochastic environment and we are in the tuple $s,a$ then there might be a 30% chance the wind will blow me left, and a 70% chance the wind will blow me right, so we want to estimate these probabilities.
Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?

NB the link should take you to roughly where he mentions this.

Is the main difference between these two problems, and hence why one is regression and the other is kernel density estimation, because with the reward we are mainly concerned with the expected reward (hence regression) whereas with the state transitioning, we want to be able to simulate this so we need the estimated density?

Yes.
An expected reward function from $s,a$ is all you need to construct valid Bellman equations for value functions. For example
$$q_{\pi}(s,a) = r(s,a) + \gamma\sum_{s'}p(s'|s,a)\sum_{a'}\pi(a'|s')q(s',a')$$
is a valid way of writing the Bellman equation for action values. You can derive this from $r(s,a) = \sum_{r,s'}rp(r,s'|s,a)$ and $q_{\pi}(s,a) = \sum_{r,s'}p(r,s'|s,a)(r + \gamma\sum_{a'}\pi(a'|s')q(s',a'))$ if you have the equations in that form.
However, in general there is no such thing as an "expected state" when there is more than one possible outcome (i.e. in environments with stochastic state transitions). You can take a mean of the state vector representations over the samples you see for $s'$ but that is not the same thing at all and could easily be a representation of an unreachable/nonsense state.
In some cases, the expectation $\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$ where $x(s)$ creates a feature vector from any given state $s$, $x(s): \mathcal{S} \rightarrow \mathbb{R}^d$, can be meaningful. The broadest and most trivial example of this is for deterministic environments. You may be able to construct stochastic environments where there is a good interpretation of such a vector, even if it does not represent any reachable state.
Simple one-hot encoded states could maybe made to work like this by representing a probability distribution over states (this would also require re-interpretations of expected reward function and value functions). That is effectively a kernel density function over discrete state space.
In general knowing this $\mathbb{E}_{\pi}[x(S_{t+1})|S_t=s, A_t=a]$ expected value does not help resolve future rewards, as they can depend arbitrarily on specific state transitions.

Thanks - I understood why you wouldn't want to take the expected value of the state distribution, I just wanted to double check why we weren't interested in the full distribution of the rewards, just the expected value.

@DavidIreland: I probably focussed too much on the state distribution issue and not enough on expected reward then. I may come back and address that

For me the comment you made about only needing expected reward to be able to construct the Bellman equations for enough but the new edit is very clear.


text:
In vanilla policy gradient is the baseline lagging behind the policy?
Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16)

Initialize policy parameter θ, baseline b
for iteration=1, 2, . . . do

Collect a set of trajectories by executing the current policy
At each timestep in each trajectory, compute

the return $R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$
the advantage estimate $\hat{A}_{t} = R_{t} - b(s_{t})$

Re-fit the baseline, by minimizing $\lVert b(s_{t}) - R_{t} \rVert^{2}$

summed over all trajectories and timesteps.

Update the policy, using a policy gradient estimate $\hat{g}$,

which is a sum of terms $\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$

At line 6, advantage estimate is computed by subtracting baseline from the returns
At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
At line 8, we update the policy using advantage estimate from line 6

So is the baseline expected to be used in the next iteration when our policy has changed?
To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

So is the baseline expected to be used in the next iteration when our policy has changed?

Yes.

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.
You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.
Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that  policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.
For more on this, see Sutton & Barto, chapter 13, section 13.4.

To be honest I was also struggling with this question for quite some time and I think that it was written this way mostly because of historic reasons. Once you have the advantage $A_t = \sum_{i=t}^{T} r_{i+1} - V_\phi(s)$, you can use it directly to update both the value and the policy networks:
\begin{align}
&w = w + \alpha A_t \nabla V_\phi(s) \\
&\theta = \theta + \alpha A_t \nabla \log \pi_\theta(a|s)
\end{align}
To see why this is true note that you update the value network using the squared-error loss $L^{VF} = 0.5(\sum r_{t+1} - V_\phi(s))^2$, and when you differentiate this you get simply $A_t$. And the loss for the policy network is simply $L^{PG} = \log \pi_\theta(a|s) A_t$.
Essentially you evaluate the value network only one time, instead of two times, which probably back at the time when this algorithm was invented (e.g. in the 1990s) was a big deal.
If I were to implement vpg with baseline, I would first update the value network and only then calculate the advantage to update the policy network, i.e. I would reshuffle the steps so that I use the updated baseline. However, I would never use vpg with baseline. I don't think anyone has ever used it for something practical and I don't know of any papers that show SotA results with it.
Note that if you have the value network, instead of using it merely as a baseline, you would be much better off using it to bootstrap the estimate of the return, i.e. $A_t = r_{t+1} + V(s_{t+1}) - V(s_t).$
If you take a look at the pseudocode for this algorithm you can see that, again, people update the value network after they compute the advantage. Here, you could again update the value network before that, but now you have to be extra careful. When computing the advantage you have to use the old value network for bootstrapping and the new value network only for the baseline: $A_t = r_{t+1} + V_{old}(s_{t+1}) - V_{new}(s_t).$ A very brief explanation for this is given at the end of page 55 in John Schulman's phd thesis that you shared.
So in this case, I would just go on and use the old network and not overthink it.
Even better than a one-step bootstrap, you could do an n-step bootstrap: $A_t^{(n)} = r_{t+1} + r_{t+2} + \cdots + r_{t+n} + V(s_{t+n}) - V(s_t)$ and then use a generalized advantage estimation $A^{GAE} = \sum_{n=1}^{\inf} (1-\lambda) \lambda ^ {n-1} A_{t}^{(n)}$. When computing $A^{GAE}$ the value network is used both for bootstrapping and as a baseline and it is not possible to distinguish between the two uses because they are cancelled out in a telescoping sum. This means that the only correct way would be to update the value network after you compute the advantage.
So you can see that once you have the value network you get from a poor algorithm to state-of-the-art with one simple equation. See here if you want to read more.

authors: Rachid ELAÏDI رشيد العايدي


text:
Continuous control with DDPG: How to eliminate steady state error?
Currently I'm working on a continuous control problem using DDPG as my RL algorithm. All in all, things are working out quite well, but the algorithm does not show any tendencies to eliminate the steady state control deviation towards the far end of the episode.
In the graphs you can see what happens:
In the first graph we see the setpoint in yellow and the controlled continuous parameter in purple. In the beginning, the algorithm brings the controlled parameter close to the setpoint fast, but then it ceases its further efforts and does not try to eliminate the remaining steady state error. This control deviation even increases over time.

In the second graph, the actual reward is depicted in yellow. (Just ignore the other colors.) I use the normalized control deviation to calculate the reward: $r = \frac{\frac{|dev|}{k}}{1+\frac{|dev|}{k}}$.
This gives me a reward that lies within the interval $]0, 1]$ and has a value of $0.5$ when the deviation $dev$ equals the parameter $k$. (That is the parameter $k$ kind of indicates when half of the work is done)
This reward function is relatively steep for the last fraction of the deviation from $k$ to $0$. So it would definitely be worth the effort for the agent to eliminate the residual deviation.
However, it looks like the agent is happy with the existing state and the control deviation never gets eliminated. Eventhough the reward is stuck at ~0.85 instead of the maximum achievable 1.

Any ideas how to push the agent into some more effort to eliminate the steady state error?
(A PID controller would exactly do this by using its I-term. How can I translate this to the RL-algo?)
The state presented to the algo consists of the current deviation and the speed of change (derivatve) of the controlled value. The deviation is not included in the calculation of the reward function, but in the end we wat a flat line with no steady state deviation of course.
Any ideas welcome!
Regards,
Felix

After some research on the subject, I found a solution to my problem of steady state errors in continuous control using DDPG:
I added a reward component based on the integral of the error. This yields maximum reward if the error integral is zero and yields lower rewards down to 0 if the integral has some value in it.
This integral component in the reward was found to be very effective, but introduces some overshoot after set-point changes. By limiting the integral to quite small quantities, this behavior could be overcome.
All these findings are detailed out in the "Reward Engineering" section of my master's thesis. Please have a look into https://github.com/opt12/Markov-Pilot/tree/master/thesis
I'll be glad to get feedback on it.
Regards, Felix

Could you figure out a workaround for this problem of the steady-state error using DDPG?
I'm currently facing the same problem. My application is Satellite attitude control, and no matter which cost function variation I use, the resulting controlled system maintains a constant steady-state error.
It seems to be a limitation of the algorithm. But I can't understand why this happens. Hence, I've decided to use PPO for now. But would like to know if you could solve this problem using DDPG.
All the best,
Wilson

Hey Wilson, yes, I could eliminate the steady state error with DDPG sucessfully by adding an integral component to the reward. I'll post a link to my master's thesis next week, where this is described in detail.
You stated, that this is a DDPG-inherent problem which my not affect e. g. PPO. Do you have any references where to read about this?

Hi Felix! That's great news! Yes, I will definitely want to read your master's thesis and understand better how you did it. Yes, a reference I read recently is this article https://arxiv.org/abs/1804.04154, it is about the attitude control for a UAV. The author compares different RL algorithms with  a conventional PID, and you can see there that the DDPG indeed presented a steady-state error and the PPO was the one that performed the best. However, in my own experiments,  the one that worked better for me so far was the improved version of DDPG, called TD3.


text:
Is better to spend parameters on weights or bias?
If a neural network has a limited number of neuron parameters to find, -let's say only 1000 parameters-, it is generally better to spend the parameters on weights or neuron bias?
For example, if each neuron has 2 weights and one bias, it uses 3 parameters per neuron, so only 333 neurons would be available.
But if each neuron uses no bias parameter, then 500 neurons are available with 1000 parameters.
I'm concerned with overfiting by using too many parameters, so I want to minimize the number of parameters meanwhile maximizing the quality of the result.

Are you just interested in 1. having or not biases or 2. in general, how many connections a neuron has? From your title, it seems like you're trying to understand whether you should or not have biases. However, from the body of your post, it seems like you're also interested in how you should define all the connections (including biases) between neurons (e.g. if a neural network should be sparse or not).

@nbro I'm asking about the choice between (more neurons) or (less neurons with bias parameters). I guess that more neurons require more computing, and more cycles for convergence, but more neurons can achieve higher accuracy. If the constrain is the number of parameters, then I guess that more neurons should be preferable.

The capacity of a network is proportional to the number of parameters and not necessarily the number of neurons, so I think you're making a possible wrong assumption here. I think your question is more related to the sparsity of the parameters (i.e. how many connections you have and how they affect learning).

First of all, your estimates are a bit off. If you have 300 neurons, you won't have just 2 weights per neuron, but much more, assuming full connectivity
Bias isn't just an extra parameter to fit, it is an important adjustable parameter that sets the offset of the separating hyperplane represented by each neuron. Think of the simple equation $ax+b$, there's no way to shift the line unless you use the $b$ (bias) part.
This would be especially important for small number of nodes and classification tasks (think perceptrons etc)

Thank you for your answer. I said 300 neurons with 2 weights just as an illustrative example. Do you mean that the bias parameter cannot be replaced by using more neurons without bias? Or just that it is more efficient to use bias than adding neurons?

@tutizeri the bias parameter cannot be replaced by using more neurons without bias, think of it as fitting a line to separate some points, you can rotate the line with the slope (weights) but it still passes through the origin, if your data points are not distributed such that they can be separated by this line, you will need to move the line. This is for one neuron. So, the bias is an important parameter without which your weights can be useless.


text:
Is there any programming practice website for beginners in Reinforcement Learning
I am doing an online course on Reinforcement Learning from university of Alberta.
It focus too much on theory. I am engineering and I am interested towards applying RL to my applications directly.
My question is, is there any website which has sample programmers for beginners. Small sample programs.
I have seen several websites for other machine learning topics such as CNN/RNN etc. But the resources for RL are either limited, or I couldn't find them

The course you are referring to does have programming assignments for most of the studied concepts/algorithms. Did you check those?

Are you looking for something like this https://github.com/dennybritz/reinforcement-learning or a course where they teach RL with a focus on the implementation part too? Do the answers to this question answer your question? If yes, feel free to upvote them (including the question), then I will close this question a duplicate of that one.

Firstly, since you are a beginner, I strongly recommend you start reading Sutton's book. It is a really great book.
Then, some tutorials:
udemy rl
udemy deep rl
rl-with-tensorflow
learndatasci
stackabuse


text:
How do LSTM or GRU gates learn to specialize in their desired tasks?
While I was studying the equations for the computation inside GRU and LSTM units, I realized that although the different gates have different Weight matrices, their overall structure is the same. They are all dot products of a weight matrix and their inputs, plus bias, followed by a learned gating activation. Now, the difference between computation depends on the weight matrices being different from each other, that is, those weight matrices are specifically for specializing in the particular tasks like forgetting/keeping etc.
But these matrices are all initialized randomly, and it seems that there's no special tricks in the training scheme to make sure these weight matrices are learned in a manner that the associated gates specialize in their desired tasks. They are all random matrices that kept getting updated with gradient descent.
So how does, for example, a forget gate learn to function as a forgetting unit? Same question applies to others as well. Am I missing a part of the training for these networks? Can we ever say that these units learn truly disentangled functions from each other?

It comes down to the order they're computed in, and what they're used in. I will be referring to the LSTM in this answer.
Looking at the forget gate, you can see that it has the ability to manipulate the cell state. This gives it the ability to force a forget. Say (after training) it sees a super important input that means some previous data is irrelevant (say, like a full stop). This forget gate, while it might not force a forget, has the ability to force one, and will likely learn to.
The input gate ultimately adds to the cell state. This gate doesn't have direct influence over the cell state (it can't make it 0, like the forget gate can), but it can add to it and influence it that way. So it is an input gate.
The output gate is used to interpret the hidden state, and get it ready to be combined with the cell state for a final output at that time step.
While these gates all use sigmoid functions, are all initialised randomly and have the same dimensionality, what their output is used in, and the order they're computed in gives them a certain role to play. Initially, they won't conform to this role, but logically as they learn, they likely will.

There really isn't a completely clear reason why you need a hidden state over the cellstate. IIRC the structure of the LSTM allows for better gradient propagation(supposedly).

It does, the cell state acts as a gradient highway as looking through a series of LSTM cells, the gradient is only affected by 3 operations each time step, and so allows for longer distance relationships to remain in tact.


text:
What's the right way of building a deep Q-network?
I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space.
I tought that the action $a_t$ should have been included as an input of the neural network, togheter with the state. It also made sense to me as when you have to compute the argmax or the max w.r.t. $a_t$ it was like a "standard" function. Then I've seen some examples of networks that had as inputs only $s_t$ and that had as many outputs as the number of possible actions. I quite understand the logic behind this (replicate the q-values pairs of action-state) but is it  really the correct way? If so, how do you compute the $argmax$ or the $max$? Do I have to associate to each output an action?

Do I have to associate to each output an action?

You are absolutely correct! In DQN, each output node of the neural network will be associated with one of your possible actions (assuming a finite discrete action space). After passing an input through the network, the value of each output node is the estimated q-value of the corresponding action. One benefit of this architecture is that you only need to pass the input through the neural network once to compute the q-value of each action, which is constant in the number of actions. If you were to include an action as an input to the neural network along with an observation, then you would need to pass an input for each action, which scales linearly in the number of actions. This is mentioned in paragraph 2 of Section 4.1 in the original DQN paper (https://arxiv.org/abs/1312.5602)

Is it really the correct way? If so, how do you compute the argmax or the max?

It's one possible way that is used in many popular algorithms such as DQN. To find the argmax, you simply take the action corresponding to the output node with highest q-value after passing an input through the network.