text:
How to make NN distinguish between two types of functions (data)?
I have a neural network which is trying to predict two types of functions in a noisy setting. The input is an array, and the output is also an array. The two types of functions I am trying to predict are the following:

a linear function with negative slope
a significantly steeper exponential function.

Example inputs for each would look like

[10, 9, 8, 7]
[11, 1, 0.1, 0]

Sometimes my neural network predicts well, such as the following:
Linear
Other times my neural network completely messes up and predicts the exponential decay instead:
Poor Prediction on Linear
As you can imagine, this makes the final result a very poor one.
I'm using a NN architecture of NCF because I'm setting up from a matrix setting. That is, the arrays I have generated come from a noisy matrix setting of low rank.
Any suggestions to improve the result? When I only use one function (e.g. just linear or just exponential) I get superb results. However, mixing both those functions together randomly causes my neural network to perform poorly, as described in my question.

How about dividing the problem?
You can first train a classification model that predicts the type of function (linear or exponential).
Then you can use your seperately trained nn depending on the classification output.
P.S.
I'm not sure why you would use a neural network for this problem. Fitting a linear/exponential function seems to be a relatively simple problem that can be achieved using traditional ML approaches.

As DKDK said, Indeed one could fit both linear and exponential function and see which one has smaller residual, without using any complex AI.
But OTOH this could be a great toy-problem for learning about neural networks. You could have a network with these parts:

A network with a final sigmoid activation, which predicts whether the function is linear or not. Let's call its prediction C for "class".
An additional "data path" within the same network, which predicts linear functions. Let's call its prediction L for "linear".
An additional "data path" within the same network, which predicts exponential functions. Let's call its prediction E for "exponential".
There is just a single input but two outputs. The first output is C and the second one is C * L + (1 - C) * E.

Then you can train the whole network end-to-end, but this model could output a mix of 20% linear and 80% exponential values. If this is a problem then I'd use three separate networks as DKDK suggested.
Edit: Now that I read the question again I noticed that all your linear functions have a negative slope? Well surely they will be easy to identify by comparing the first and last element of the list. Then this is just a normal regression problem.


text:
Are these book example CNN results realistic?
I've been following a deep learning book and the current section I'm on is about convolutional neural networks. The author presents some code to create a basic CNN with about 1 million parameters, which he manages to train to 99.2% accuracy within 12 epochs on the full MNIST dataset.
His output looks like this:
Train on 60000 samples, validate on 10000 samples
Epoch 1/12-loss:0.2800 acc:0.9147 val_loss:0.0624 val_acc:0.9794
Epoch 2/12-loss:0.1003 acc:0.9695 val_loss:0.0422 val_acc:0.9854
Epoch 3/12-loss:0.0697 acc:0.9789 val_loss:0.0356 val_acc:0.9880
Epoch 4/12-loss:0.0573 acc:0.9827 val_loss:0.0282 val_acc:0.9910
Epoch 5/12-loss:0.0478 acc:0.9854 val_loss:0.0311 val_acc:0.9901
Epoch 6/12-loss:0.0419 acc:0.9871 val_loss:0.0279 val_acc:0.9908
Epoch 7/12-loss:0.0397 acc:0.9883 val_loss:0.0250 val_acc:0.9914
Epoch 8/12-loss:0.0344 acc:0.9891 val_loss:0.0288 val_acc:0.9910
Epoch 9/12-loss:0.0329 acc:0.9895 val_loss:0.0273 val_acc:0.9916
Epoch 10/12-loss:0.0305 acc:0.9909 val_loss:0.0296 val_acc:0.9904
Epoch 11/12-loss:0.0291 acc:0.9911 val_loss:0.0275 val_acc:0.9920
Epoch 12/12-loss:0.0274 acc:0.9916 val_loss:0.0245 val_acc:0.9916
Test loss:
0.02452171179684301 Test accuracy: 0.9916

Using this code.
Running that same code on my machine, after 12 epochs, I'm barely at 0.6 accuracy. I did have to modify a couple function calls as keras has changed a couple things since the book came out. I'm going crazy trying to figure out why his is so quick. He presents the results as "this is what should happen when the code is run." I understand it had already went through almost 500 gradient descent steps by the time epoch 1 is complete, but is that really enough to reach 98% accuracy right off the bat??

I was able to run the code without "any" modifications on Tensorflow 2.4.0, just had to replace the imports:
import keras
from keras.datasets import mnist
...

->
import tensorflow.keras as keras
from tensorflow.keras.datasets import mnist
...

Output:
Epoch 1/12
469/469 [==============================] - 4s 7ms/step - loss: 2.2914 - accuracy: 0.1260 - val_loss: 2.2425 - val_accuracy: 0.2915
Epoch 2/12
469/469 [==============================] - 3s 7ms/step - loss: 2.2375 - accuracy: 0.2383 - val_loss: 2.1710 - val_accuracy: 0.4693
Epoch 3/12
469/469 [==============================] - 3s 7ms/step - loss: 2.1665 - accuracy: 0.3439 - val_loss: 2.0758 - val_accuracy: 0.5645
Epoch 4/12
469/469 [==============================] - 3s 7ms/step - loss: 2.0750 - accuracy: 0.4286 - val_loss: 1.9489 - val_accuracy: 0.6155
Epoch 5/12
469/469 [==============================] - 3s 7ms/step - loss: 1.9510 - accuracy: 0.4964 - val_loss: 1.7919 - val_accuracy: 0.6633
Epoch 6/12
469/469 [==============================] - 4s 8ms/step - loss: 1.8107 - accuracy: 0.5401 - val_loss: 1.6096 - val_accuracy: 0.7082
Epoch 7/12
469/469 [==============================] - 3s 7ms/step - loss: 1.6469 - accuracy: 0.5737 - val_loss: 1.4162 - val_accuracy: 0.7475
Epoch 8/12
469/469 [==============================] - 3s 7ms/step - loss: 1.4888 - accuracy: 0.6095 - val_loss: 1.2314 - val_accuracy: 0.7727
Epoch 9/12
469/469 [==============================] - 3s 7ms/step - loss: 1.3350 - accuracy: 0.6409 - val_loss: 1.0702 - val_accuracy: 0.7961
Epoch 10/12
469/469 [==============================] - 3s 7ms/step - loss: 1.2118 - accuracy: 0.6651 - val_loss: 0.9390 - val_accuracy: 0.8161
Epoch 11/12
469/469 [==============================] - 3s 7ms/step - loss: 1.1072 - accuracy: 0.6855 - val_loss: 0.8373 - val_accuracy: 0.8291
Epoch 12/12
469/469 [==============================] - 3s 7ms/step - loss: 1.0208 - accuracy: 0.7061 - val_loss: 0.7568 - val_accuracy: 0.8370

Test loss: 0.7567713856697083
Test accuracy: 0.8370000123977661

Running it 10 times (it took about 30 seconds / model on 1080 Ti):
Test loss: 0.71385968 +/- 0.0471146
Test accuracy: 0.84305001 +/- 0.00868576

So clearly these are very different results than the author got, but also significantly higher than what You got. He might have used different parameters with the Adadelta optimizer, or a different version of it.
Usually I've had the best results with Adam (default parameters, learning rate = 0.001). I tested running it 5 times with these results:
Test loss: 0.02858894 +/- 0.00151724
Test accuracy: 0.99186 +/- 0.00058172

Adam is 3 years newer than Adadelta (2014 vs. 2011).

Thank you so much! This really clears things up. I think 0.6 was the lowest I got. My more recent ones were in the same range as yours. It was just the instant high accuracy and low loss from his data that was really throwing me off. However, I'm able to mimic that using Adam like you said. Happy to see everything's running as its supposed to. Guess I'll stick to Adam for now.

According to my experience, it is possible to reach 99%+ accuracy on MNIST within a few epochs using a simple CNN. MNIST is really an easy dataset. So, it's likely that you've broken something as you're modifying author's code.

I didn't really "modify" much. The compiler couldn't find the to_categorical() or the Adadelta() functions in the imported libraries, so I imported tensorflow itself and prefixed the function call with it.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Simple Polynomial Gradient Descent algorithm not working
I am trying to implement a simple 2nd order polynomial gradient descent algorithm in Java.  It is not converging and becomes unstable.  How do I fix it?
public class PolyGradientDescent {
public static double getValue(double input) {
    return 3 * input * input - 4 * input + 3.5;
}

public static void fit() {
    double x0 = Math.random();
    double x1 = Math.random();
    double x2 = Math.random();
    double size = 15;
    double learningrate = 0.0001;

    for(int i = 0; i < 400; i++) {
        double partial_x2 = 0;
        double partial_x1 = 0;
        double partial_x0 = 0;
        for(double x = 0; x < size+0.001; x++) {
            double xx = x * x;
            double y_predict = xx * x2 + x * x1 + x0;
            double delta = getValue(x) - y_predict;
            partial_x2 += xx * delta;
            partial_x1 += x * delta;
            partial_x0 += delta;
        }
        x0 = x0 + (2 / size) * partial_x0 * learningrate;
        x1 = x1 + (2 / size) * partial_x1 * learningrate;
        x2 = x2 + (2 / size) * partial_x2 * learningrate;
        System.out.println(x0 + "\t" + x1 + "\t" + x2 + "\t" + "\t" + partial_x2 + "\t" + partial_x1 + "\t" + partial_x0);
    }
    for(double x = 0; x < size+0.001; x++) {
        System.out.println("Y: " + getValue(x) + ", Y_Predict: " + (x2 * x * x + x1 * x + x0));
    }
}

public static void main(String[] args) {
    fit();
}
}

I tested your code in python and it works just fine, when I decrease the learning rate (divided by 100) by a bit more (and the epochs multiplied by 100).
I also changed the way the derivative was calculated to make it more mathematically correct :)
import random

def getValue(x):
    return 3 * x * x - 4 * x + 3.5

def fit():
    x0 = random.randrange(-100, 101) / 100
    x1 = random.randrange(-100, 101) / 100
    x2 = random.randrange(-100, 101) / 100
    size = 15
    learningrate = 0.000001

    for i in range(40000):
        partial_x2 = 0
        partial_x1 = 0
        partial_x0 = 0
        for x in range(16):
            xx = x * x
            y_predict = xx * x2 + x * x1 + x0
            delta = getValue(x) - y_predict

            # for the partial derivatives, I pulled the sign and the 2 into this step, and also devided the term later by -2, because this would be the true derivative
            partial_x2 -= 2 * xx * delta
            partial_x1 -= 2 * x * delta
            partial_x0 -= 2 * delta

        x0 = x0 - (1 / size) * partial_x0 * learningrate
        x1 = x1 - (1 / size) * partial_x1 * learningrate
        x2 = x2 - (1 / size) * partial_x2 * learningrate

    for x in range(16):
        print("Y: " + str(getValue(x)) + ", Y_Predict: " + str(x2 * x * x + x1 * x + x0))

fit()

Thanks for your help.  I appreciate.  Does the factor of 2 come from chain rule applied to the ordinary least squares cost function?  If so, that makes sense.  I ran your function and it performs similar to mine.  In fact, I think they are equal, no?  I'm a little disappointed at the performance of this algorithm.  Here, we are only trying to fit 15 points which align perfectly on a 2nd order polynomial and even after 1 million iterations, the delta between y and y-predict is still greater than 1 for a couple of points.  Is that expected?

Yes you are right, and they should be equal as I just pulled the -2 from one term into the other. Your algorithm is inefficient because you are working with polynomials which produce large values. The label for X: 15 is Y: 618.5. That means if your function predicts 0 in the beginning, partial_x2 becomes 2 * 15 * 15 * 618.5. That's why you need the small learning rate. What you could try is gradient clipping where you clip the partial_xn values to be between e.g. -20 and 20 and then you can choose a much higher learning rate. I have tried it and it worked quite well. I hope this helps :)

thanks.  I found this helpful too -- added momentum. https://www.cs.toronto.edu/~lczhang/321/notes/notes08.pdf

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Are any non-injective activation functions used?
All activation functions I know of are injective, which I think makes sense.
But are there cases where non-injective activations can be useful?

Do you care about clarifying what you mean by "injective" and which activation functions do you have in mind (because maybe there's a misunderstanding)?

Injective like in https://en.wikipedia.org/wiki/Injective_function. And I don't have examples in mind. Like of course |x| or sin(x) would be possible, but I'm unsure of what would be gained when using them.

There is at least Swish, which is defined as $f(x) = x \cdot \text{sigmoid}(\beta x)$.

...This suggests that Swish can be loosely viewed as a smooth function
which nonlinearly interpolates between the linear function and the
ReLU function. The degree of interpolation can be controlled by the
model if β is set as a trainable parameter.

There is an other paper which introduces Growing Cosine Unit, defined as $f(x) = x \cdot \text{cos}(x)$.

The experiments (Table 1-3) show that use of GCU activation for
convolutional layers and ReLU for the dense layers provides the best
performance among all architectures considered. This is particularly
evident on the VGG-16 network trained on the Imagenette dataset, where
the GCU models outperform all ReLU architectures by 7+%. The models
with GCU in the convolutional layers also converge faster during
training as highlighted by Fig 7-8.

In their experiments this GCU performed significantly better than Swish, or any other activation function.

Hm interesting. Maybe bad question as I don't have experience with academics: On Google scholar it only shows 2 citations of this paper. Is this inaccurate, or do responses just take some time, or is this result not as important as it seems?


text:
Does the state space of an MDP change in these two examples?
In the classic Atari environments, like that introduced in the original DQN paper, the state space is the set of all possible images that the Atari emulator can produce (or more generally just any RGB image, potentially stacked to better represent the environment). This makes sense as the CNN in the DQN is trained end-to-end with the RL signal, and so the Q-Fuction looks directly at the image as input.
Now, in methods such as CURL that look to pre-train the CNN and treat it as an encoder, does the state space change here? My thinking is that, if the pre-trained encoder is a function $\psi: \mathcal{I} \rightarrow \mathbb{R}^d$, where $\mathcal{I}$ is the space of images, then the state space is now $\mathbb{R}^d$. The rationale for my thinking is that now the agent directly observes the vector in $\mathbb{R}^d$ rather than the image from the encoder, and the state space should be what the agent observes (even though this vector is a representation for the image).

Whilst engineering solutions in reinforcement learning, I think it is common to discuss the concept of state space loosely, in terms of what the search space looks like for the algorithm, and what compromises are OK even though they technicaly make the problem a POMDP.
In terms of definitions relating to the MDP, the state space has well-defined meaning. It is the set of all state values that can occur in the environment. That set/space can be mapped into different domains, but it remains the same size of space in terms of the set for any bidirectional mapping.
Once you start to implement a state representation in a real system, in a practical agent, you often need to compromise regarding this definition. Even in purely mathematical treatments, it may not be convenient to determine all the theoretically reachable states. Determining them can be more complex than the optimal control problem. So it is very common to over-specify the state space.
Atari games don't reach states where they produce arbitrary images. Their output during gameplay is on a relatively small manifold embedded in image space. Despite this, the over-specification in image space is useful, because we have good toolkits for working with it, including CNNs for learning generalising function outputs when images are used as inputs.
Another compromise seen in the original Atari DQN is missing state. Only using the image, even when stacked, can mean a certain amount of state is not being used. Depending on the game, this state could be important enough that a POMDP would make a better model, and the images would move from a state space representation to an observation space representation (as an aside, stacking images to include velocity information could be replaced by a sequence-aware model such as RNN, and this is similar to POMDP approach, building an internal state representation separate from observations).
In both cases - over-specified state space, and missing state - the state space of the problem is not changed. When implementing the agent, you know the representation space you are using, and expect it to have good coverage of all possible states, but often do not know the precise underlying state space of the MDP.
This further gets confounded by feature engineering. I would treat the embedding by pre-trained CNN as a form of feature engineering. In theory it could reduce the dimensionality of the optimisation problem significantly, speeding up learning, but there is always a risk that the pre-training misses key features due to differences that are important in RL context having a low weighting in unsupervised learning of the embeddings.
So does converting an RL problem that works with images from vision-based observations, to work with embeddings of those images reduce the state space? I would say no, the problem definition is not changed, it has the same state space as before. However, the separation of concerns (vision processing vs policy or value prediction), and lower dimension space for generalising has still done something useful. It may help with generalisation, as similar states may be be closer in the embedding space than they are in the larger image space.
Loosely speaking you could say that CURL "reduces state space" and most people would understand what you meant in practical terms. I would personally caveat that with e.g. "effectively reduces state space" or perhaps "makes it easier for the agent to generalise its experience across the state space".

thanks for this Neil, a great answer as always!


text:
Does the summing or averaging of the weight gradients have anything to do with the cost function used?
I've been trying to implement my own neural network library and have been wondering if:

The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)

The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.

or if the summing or averaging of the weight gradients has nothing to do with the loss function used?
I feel like the answer would be clear if I knew other loss functions, if it did matter that would mean the weight gradients should be averaged for MSE and summed for SSE in mini-batch gradient descent?

seems at least a partial dup of this. let me know.

hi, I've seen that one before. I don't think it answers my question.

What you stated looks correct :-
The SSE loss function includes the summation of the errors in the other training examples of the mini-batch (each training example's loss in the mini-batch is summed for one big loss)
SSE = Summation of (y-y')squared for all samples in mini-batch
The MSE loss function averages the loss of each individual training example in the mini-batch, and then all those losses are averaged based on the mini-batch size.
MSE = (1/m)SSE

You can format the formulas and math symbols in this answer with latex. I would recommend that you do that.


text:
Action-value estimation of deterministic policies with Monte Carlo method
In Monte Carlo-based action value estimation  problem for a deterministic policy (estimation of $q_{\pi}(s,a)$),the estimation problem seems not to be well-defined because  $q_{\pi}(s,a)$ by definition means the value of an arbitrary action $a$ at a given state $s$
when initial action $a$ is applied at that state and then following actions from policy $\pi$ at
the next states. But, in a real application under a given deterministic policy $\pi$, how can you choose the initial
action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?

But, in a real application under a given deterministic policy $\pi$, how can you choose the initial action $a$ arbitrarily at state $s$ because it is already fixed by the policy $\pi$: $a=\pi(s)$?

You ignore $\pi$ for the action selection of the action you need to evaluate. This is well-defined, it is the expected future return for taking action $a$ in state $s$ and thereafter following the policy $\pi$.
So you may have a problem estimating that value, unless you also force that action choice during training in order to observe what transitions and rewards follow (that you would never see following the deterministic policy). However, it is well-defined conceptually.
A very simple way to force the assessment of all state/action pairs is exploring starts: Pick an arbitrary state/action pair to evaluate, then follow the transition and policy rules from that point on until the espidode end. This will give you a Monte Carlo sample of the value for the starting point, that you can use to update the estimate.
If you are not able to use exploring starts, or otherwise take actions different from the supplied deterministic policy, then you may be stuck. You would only have data of certain $s,a$ pairs. You could estimate $Q(s,a)$ for those pairs, and not for others.

@ Neil Slater In a real application, how can you pick an arbitrary state/action pair to evaluate? Maybe, the  state/action pair you chose does not exist in episodes.

The answer includes this - start your episode from the arbitrary state/action pair. That is called exploring starts. If you only have historical data, or are not able to set up the environment in particular states, then your options are limited. Probably you will not be able to complete the Q function, given your other constraint of a fixed deterministic policy. But maybe that's not important. You could still create a state value function for all reachable states for instance.

@user3489173 You don't say in the question what your goal is. If you have a specific goal and constraints you must meet, I may be able to explain whether it is logically possible.

@ Neil Slater. Assume that our application is a real-time application and the historical data of this application contains n number of episodes where n is a large number. I wonder how  you can  estimate action value function using monte carlo methods.

@user3489173 If you have historical data using a fixed policy, you can only estimate action values of the actions that were taken. If the policy was deterministic, that will mean you know nothing about any alternative actions. You can fill in $Q(s,a)$ estimates for the actions that were observed, but not anything else.

@ Neil Slater. Perfect explanation! You must write a reinforcement learning book!


text:
Solving logical pattern puzzles with Machine Learning?

I found this kind of problem while reading about some web tests companies use to screen applicants. It is a puzzle where you need to guess what comes inside?
Looking at this made me wonder,
How would you approach solving problems like this using Machine Learning?
I assume that if people can recognize the pattern with a bit of training, so should A.I. be able to as well.
If anyone knows anything that could be a good starting for me to solve this problem, please let me know. Any opinion will be appreciated! (whether this is possible or not, how difficult it is, computer vision models, algorithms etc., cool projects like this etc.)
I've seen other people solve logic games like here
But visual cues get complicated like this often in this type of puzzles.

These specific puzzles are called Raven's Progressive Matrices, a common psychometric tool used to measure the IQ. Knowing that is easy to find references. Here just a few examples, but the literature of course is broader, visual reasoning is quite an old AI topic.

Similarity-Based Reasoning, Raven’s Matrices, and General Intelligence
RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
Solving Raven’s Progressive Matrices with
Multi-Layer Relation Networks

From a technical perspective the approaches are not that impressive, classic CNN (and of course more fancy architectures) trained on detecting which answer amongst the given ones is the most "logically" similar to the images forming the incomplete sequence. Notice also that the task is reasonably tractable because we have a set of given answers, without that the story would be much different.
Also, not really related to the question title, but if you're interested in machine leaning and abstract reasoning a much more impressive task to tackle is numerical sequences prediction, and a fresh new paper managed to achieve incredible results in this task as well.

I did not know it was called that way! I am happy to know that these tasks are already covered quite well by deep learning! Thanks for the extra info on numerial sequences!


text:
Training a regression model on a set of values in 0-1 range to give 0-1 continual values
I have a textual dataset that has a set of real numbers as labels: L={0.0, 0.33, 0.5, 0.75, 1.0}, and I have a model that takes the text as input and has a Sigmoid output.
If I train the model on this data, will the model keep generating labels that exactly equal one of the values in L? or it might generate, as an example, 0.4?
If not, is there a solution for that?

To clarify: You want the model to be able to generate a 0.4 if that would be the best estimate given the input data?

yes @NeilSlater

As long as you train the model with a proper loss function for regression the model will learn to output any continuous values, not restricted to and most likely not exactly equal to the labels your providing during training, rather an approximation of them based on the level of generalization the model manage to learn.
The range of values the model can learn to output will also depend on the final activation function of your model. Using a sigmoid is indeed a good choice if your labels belong in the range 0-1.
If instead you don't want the model to learn continuous values then you need to frame the task as a classification problem, and convert your set of finite real values labels into a discrete representation, for example using one hot encoding and then train the model with a loss suited for classification.

Many machine learning models used for regression will interpolate their predictions as you seem to want, and can return target values not seen in the training set.
For example, basic linear regression will do this provided at least one of the input variables is also continuous. Also neural networks.
There are some cases where it may not happen:

Decision tree-based algorithms (such as random forests) approximate with combinations of step functions, so most will output a set of discrete values. However, in practice this set will include many values that are not in the training set, and it is usually not a real concern, if your goal is accurate predictions for unseen data.

If all the input features are discrete, then there will be a discrete set of output values, equal in number to all the possible combinations of input. Again, this may not be a practical concern in your case, and will very likely generate outputs that are not in the training set.


text:
Off-policy every visit MC prediction algorithm for estimating $V\approx v_{\pi}$: is this algorithm correct?
Below is  the off-policy every visit MC prediction algorithm for estimating $V\approx v_{\pi}$,
which I took from coursera.
It seems to me that this algorithm is not correct. Let me explain why. Assume that $t=T-1$ and let $\pi(A_{T-1}|S_{T-1})=0$,
which means there is no possibility for target policy $\pi$ to go from $S_{T-1}$ to terminal state
via action $A_{T-1}$. Hence, based on this observation we should have $V(S_{T-1}) \approx v_{\pi}(S_{T-1})=0$. However, the algorithm in the pseudoce returns $V(S_{T-1}) \approx v_{\pi}(S_{T-1})=R_{T}$. What do you think?

The weighting calculation ignores how the current action $A_t$ was chosen.
The action being evaluated does not need its probability adjusted by importance sampling, only future actions. This is because you have "already decided" to evaluate/update a particular $Q(s,a)$ and any probabilities of being in that part of the trajectory do not come into play (they do affect approximators such as neural nets, but that is a secondary effect that is often ignored). More formally, updating $Q(s,a)$ is conditional on you having a record $S_t = s, A_t = a$, so you do not take into account the probabilities of that happening. Whilst the return from that point on is a sample from a distribution including all the probabilities for observing $A_{t+1}$ to $A_{T-1}$ which you do need to adjust for if you are observing results from a different policy to the one you want to update.
An off-by-one relationship where you only weight for future actions is maintained simply by having the $W$ update at the end of the loop, after the Q update for the same timestep. In fact $W$ is being updated ready for calculating the weightings to apply to the previous timestep (previous because the loop is decrementing $t$). At $t = T-1$ there are no future actions, and you have not been around the loop even once yet.
When you get to the subject of off-policy TD control later on, you can also ignore how the last action in the partial trajectory was chosen, because you will use the target policy's action instead. Which means single-step Q learning doesn't use importance sampling at all.


text:
Unclear fact about difference between Gradient Descent to Stochastic Gradient Decent in wikipedia
From wikipedia page it mentioned:

To economize on the computational cost at every iteration, stochastic
gradient descent samples a subset of summand functions at every step.
This is very effective in the case of large-scale machine learning
problems

And later on it demonstrates the pseudo code with shuffle, and not sub-setting the samples.
I don't get it how come applying shuffle improves the Gradient descent:

Can somebody shed some light on this issue?

That paragraph is incomplete and unclear indeed.
Let's crack the difference with a concrete example: logistic regression.
The objective we want to minimize is:
$J_{train}(\theta)=\frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{i})-y^{i})^{2}$
Notice how this loss formulation requires us to iterate trough all our data to sum the squared errors. Gradient descent does precisely that:
$\theta_{j} := \theta_{j} - \alpha\frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{i})-y^{i})x^{i}_{j}$
Also here, to compute a single descent step we are iterating through to all our data, while keep summing the gradients. This is why normal gradient descent is so memory inefficient.
One work around to this problem is to redefine our initial loss $J$ in order to allow us to perform a descent step without having to look at all our data. We usually call this new formulation cost:
$cost(\theta, (x^{i}, y^{i})) = \frac{1}{2}(h_{\theta}(x^{i})-y^{i})^{2}$
This allows us to rewrite also our descent step in a way that doesn't require to look at all our data.
$\theta_{j} := \theta_{j} - \alpha\frac{1}{2}(h_{\theta}(x^{i})-y^{i})x^{i}_{j}$
Since the sum now it's gone it means that we don't need to sum gradients or keep in memory extra stuff, so this is much more efficient. And this is basically Stochastic gradient descent.
Problem is that because we're updating on a fraction of our data at the time, order becomes relevant. If our data are ordered in some strange fashion, we'll risk to introduce biases in our parameters do to biases in the subset we selected to perform the update. And even if other subsets of data would normally account for that, now it's not the case anymore, cause we already performed an update step, and when applying our model to the new subset the parameters are therefore already biased. The only strategy we have to prevent this is precisely randomizing the order of our data.

thank you for taking the time and explaining your answer.


text:
What does it mean by Generalization?

Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning

What does it mean by Generalization in this article?

Before asking a question, do a little bit of research.  See https://ai.stackexchange.com/help/how-to-ask. The answer to this question can be easily found by pasting your question on google.

The term  generalization  refers to the model's ability to adapt and react appropriately to new, unpublished data that was drawn from the same distribution as the one used to build the model . In other words, generalization examines how well a model can digest new data and make correct predictions after being trained on a training set.

How well a model is able to generalize is key to its success.
If you train a model too well on the training data, it will be unable to generalize. In such cases, it will end up making wrong predictions when receiving new data.

When training machine to learn from data, it is typical withhold a set of data that the machine never sees. This withheld dataset is known as the test set. The performance of the learned model on the test set gives an idea of how well the model has learned the function that generated the data in the first place as opposed to overfitting to the training dataset. This measure of performance is typically referred to as generalization error and this is what is being referred to in the article.


text:
How to construct input dependent convolutional filter?
I am constructing a convolutional variational autoencoder for images, starting out with mnist digits. Typically I would specify convolutional layers in the following way:
input_img = layers.Input(shape=(28,28,1))
conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img)
conv2 = keras.layers.Conv2D(64, (3,3), strides=2, padding='same', activation='relu')(conv1)
...

However, I would also like to construct a convolutional filter/kernel that is fixed BUT dependent on some content related to the input, which we can call an auxiliary label. This could be a class label or some other piece of relevant information corresponding to the input. For example, for MNIST I can use the class label as auxiliary information and map the digit to a (3,3) kernel and essentially generate a distinct kernel for each digit. This specific filter/kernel is not learned through the network so it is fixed, but it is class dependent. This filter will then be concatenated with the traditional convolutional filters shown above.
input_img = layers.Input(shape=(28,28,1))
conv1 = keras.layers.Conv2D(32, (3,3), strides=2, padding='same', activation='relu')(input_img)

# TODO: add a filter/kernel that is fixed (not learned by model) but is class label specific
# Not sure how to implement this?
# auxiliary_conv = keras.layers.Conv2D(1, (3,3), strides=2, padding='same', activation='relu')(input_img)

I know there are kernel initializers to specify initial weights https://keras.io/api/layers/initializers/, but I'm not sure if this is relevant and if so, how to make this work with a class specific initialization.
In summary, I want a portion of the model's weights to be input content dependent so that some of the trained model's weights vary based on the auxiliary information such as class label, instead of being completely fixed regardless of the input. Is this even possible to achieve in Keras/Tensorflow? I would appreciate any suggestions or examples to get started with implementation.

Why wouldn't you want to learn also the context (auxiliary data) dependent kernel?

Here is one way of achieving this. This network is an autoencoder, with extra auxiliary_convs. The active convolution depends on the input image's class, since each convolution layer's output is multiplied with the one-hot encoded class input.
import matplotlib.pyplot as plt
import numpy as np

from tensorflow import keras
from tensorflow.keras import layers
import tensorflow.keras.backend as K

BN = layers.BatchNormalization

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)

# Convert class vectors to binary class matrices
num_classes = y_train.max() + 1
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# Build the model
inputs = [keras.Input(shape=x_train.shape[1:]), keras.Input(shape=y_train.shape[1:])]
dim = 8

x = inputs[0]
x = BN()(layers.Conv2D(dim, kernel_size=3, activation='relu')(x))
x = BN()(layers.Conv2D(dim, kernel_size=3, activation='relu')(x))

auxiliary_convs = [layers.Conv2D(dim, kernel_size=3, padding='same', activation='relu')
                   for _ in range(num_classes)]

x_auxs = [conv(x) * inputs[1][:,ix:ix+1,None,None]
          for ix, conv in enumerate(auxiliary_convs)]

x = K.sum(K.concatenate([x_aux[:,:,:,:,None] for x_aux in x_auxs]), axis=-1)
x = layers.AveragePooling2D(pool_size=2)(x)

x = BN()(layers.Conv2D(dim, kernel_size=3, activation='elu')(x))
x = layers.AveragePooling2D(pool_size=2)(x)

x = layers.Flatten()(x)
x = layers.Dense(16, activation='elu')(x)
x = layers.Dense(2, activation='tanh')(x)

y = x
y = BN()(layers.Dense(4, activation='elu')(y))
y = BN()(layers.Dense(8, activation='elu')(y))
y = BN()(layers.Dense(16, activation='elu')(y))
y = layers.Dense(np.prod(inputs[0].shape[1:]), activation='sigmoid')(y)
y = layers.Reshape(inputs[0].shape[1:])(y)

model = keras.Model(inputs, y)
model.summary()

It is a bit simpler if you don't sum auxiliary tensors together, but simply concat them:
x = K.concatenate(x_auxs)

However in this case the last Conv2D layer would have redundant parameters to train, since x has dim * num_classes dimensions after the K.concatenate.
If you don't want to train the input-dependent part you can freeze the layers, but I don't know why you would want to do that.
I also tested a variation of this for fun, having a constrained convolutional network but modifying the autoencoded feature based on the image's class:
inputs = [keras.Input(shape=x_train.shape[1:]), keras.Input(shape=y_train.shape[1:])]
act, dim, enc_dim, l2_reg = 'elu', 16, 2, 1e-1

x = inputs[0]
x = BN()(layers.Conv2D(8, kernel_size=3, activation=act)(x))

for _ in range(3):
    x = BN()(layers.Conv2D(dim, kernel_size=3, activation=act)(x))

x = layers.AveragePooling2D(pool_size=2)(x)

for _ in range(3):
    x = BN()(layers.Conv2D(dim, kernel_size=3, activation=act)(x))

x = layers.Flatten()(x)
for k in [6, 8, 4, 3]:
    x = BN()(layers.Dense(k, activation='elu')(x))

x = layers.Dense(enc_dim, activation='linear', use_bias=False,
                 activity_regularizer=keras.regularizers.l2(l2_reg))(x)

a = layers.Dense(enc_dim, activation='exponential', use_bias=False)(inputs[1])
b = layers.Dense(enc_dim, activation='linear')(inputs[1])

x = keras.activations.tanh(a * x + b)

y = x
for k in [3, 4, 8, 32, 128]:
    y = BN()(layers.Dense(k, activation='elu')(y))

y = layers.Dense(np.prod(inputs[0].shape[1:]), activation='sigmoid')(y)
y = layers.Reshape(inputs[0].shape[1:])(y)

model = keras.Model(inputs, y)
model.summary()

The scatter plot on the left shows the codes (aka. embeddings), and the image's class is varied on the decoded examples on the right. Higher the l2_reg is, the tighter the classes' distribution is. Although I don't know what is the utility of this network :D

Not a tensorflow expert but I may be able to offer some conceptual advice. Since you do not care to learn the filter, but instead want to fix a discrete set of possible values for a discrete set of cases, you can use tensor operations (i.e convolutions) rather than neural network layer operations. Essentially, in framework-agnostic pseudocode this would look like:
# layers with learned parameters
output1 = layers1(input)

# apply unlearned but changeable layer convolutions
kernel_val = kernel_val_selection_function(output1)
output2 = convolve_2D(output1,kernel_val)

# more layers with learned parameters
output3 = layers3(output2)

...

The function graph will treat kernel_val as a constant for purposes of backpropagation, so as long as your convolution operations are done within the framework used to create the function graph (i.e. tensorflow) you shouldn't have any problems with backprop.


text:
Additional Optimizations for Convolutional Models On Inferencing
I am aware of several ways to optimize a convolutional (or any) model after training to make inferencing quicker. I am currently implementing BatchNormalization Folding and removing Dropout layers from the network. I am also aware of post training quantization (specifically 16-bit quantization for use on GPU).
Are there other layer optimization techniques that I can use other than quantization?
My current model uses, Conv2D, Activation(relu), BatchNormalization, Dropout, Dense layers.
Basic mnist Convnet metrics for 10K images, batch size of 1: (All have 98.92% accuracy)

Original Network: 49.3s
Folded Network: 33.37s
Quantized Original: 7.449s

You could look into models pruning. There are several techniques out here, and all of them aim to reduce the amount of parameters of a model without affecting its performance metrics. Of course less parameters means less calculation and therefore faster inference time.

Thank you. I haven't looked into pruning much because it's not as straightforward as the others I've done. I'll look more into it now.


text:
Can Deep Reinforcement Learning come up with heuristics for a game it trains on and masters?
I am taking a course where we write minimax, alpha-beta pruning and interative deepening in Python for the game of Isolation.
I am supposed to write heuristics for an evaluation function of the game state.
But I wondered if training an RL agent on this game, have it master it, are there any examples of a deep RL agent that returns heuristics based on what it learns while training?

An optimal value function from reinforcement learning on a game is a perfect heuristic, allowing a single ply search for the best action.
Other than very basic games, such as tic tac toe (noughts and crosses), an RL agent will not learn a perfect value function. The issue is then that the value function is not admissible (it may over-estimate as likely as under-estimate), therefore it can mislead a search.
However, in practice, the non-admissibility of a heuristic in game playing agents is not a complete showstopper. It means the agent will be less efficient and may make some mistakes, but an approximate learned heuristic is stll pretty good.
In practice, the main issue I have found when using a neural network or other ML approximator in minimax is the number of evaluations that need to be made. For efficiency you may be able to batch them up, but for a significant depth you may still need 100s or 1000s of value function prediction runs to resolve the minimax. This can take far more CPU time than a simpler heuristic.
In my own project (playing Connect 4) where there was a strict time constraint for turns, I resolved this issue by:

First performing a deep minimax (e.g. 12-ply) with simplest heuristic (1 for winning, -1 losing, 0 for draw or unresolved) plus a flag for whether the search was to game completion (i.e. unresolved). This deep search could be highly optimised around win condition detection only.
Second, if that search did not resolve the game, I ran a much shallower minimax (3-ply) with the ML value predictions as a heuristic.

This combination performed well enough to get into top 10 positions in a coding competition where several competitors had coded perfect play. The main issue with it is that the first search could prune a branch for good reasons that the second search then preferred due to inaccuracies in the value function, causing the agent to lose to perfect play agents. There are probably ways around that, but I did not investigate further.


text:
Confusion about state-action value function notation used in Sutton-Barto RL book
Let $\pi$ be an $\epsilon-soft$ policy with state-action value function $q_{\pi}(s,a)$
and $\pi'$ be an $\epsilon-greedy$ policy with respect to $q_{\pi}$.
In Sutton-Barto RL book  (page 101, eq. 5.2), they define
\begin{align}
q_{\pi}(s, \pi'(s))=\displaystyle \sum_{a}\pi'(a|s)q_{\pi}(s,a).
\end{align}
Normally $q_{\pi}(s, a)$ means taking action $a$
at state $s$ and then following the policy $\pi$. Based on this convention, the notation
$q_{\pi}(s, \pi'(s))$ is weird because $\pi'(s)$ is not a single action like $a$.
I.e., $\pi'$ is a stochastic policy
and hence only $\pi'(a|s)$ makes sense.

Rather than writing "Confusion about..." in the title, can you put your specific question in the title?

The book presents a slight abuse of notation where $\pi'(s)$ is shorthand for a distribution of action values described by the more correct $\pi'(a|s)$. At that point there is an implied function composition of $q_{\pi}$ over this distribution, resolved on the right hand side to its expectation.
I believe it does this so that it can briefly show something familiar from the deterministic policy improvement theorem. You could almost read it as "the equivalent of this term (LHS) taken from the previous proof must now be written like this (RHS)".
It would be more correct notation to write something like this:
$$\mathbb{E}[q_{\pi}(s,A')|A' \sim \pi'] =  ...$$
or perhaps just start with the right hand side.

@ Neil Slater,  strangely authors also  set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.  I cannot see why this holds.

@DSPinfinity It holds provided the only change between $\pi$ and $\pi'$ is at the single state $s$, and that there are no cyclic paths that allow a revisit to $s$.

@ Neil Slater  $v_{\pi'}(s)$ means the value of state $s$ following $\pi'$,  $q_{\pi}(s, \pi'(s))$ means value of state state $s$  by applying $\pi'$  and then following $\pi$. So, how can they be equal? Am I missing something?

They can be equal if the only difference between $\pi$ and $\pi'$ is localised to state $s$ (plus the no loops constraint), because $\pi$ and $\pi'$ are identical for all other states that could follow $s$.

@ Neil Slater, $\pi'$ is greedily updated from $\pi$ during a stage of policy improvement. However, during that stage it is not updated for the single state $s$ but for all encountered states of the episode. Hence, it seems that the difference is not only for the single state $s$. What do you think?

In the case you describe, $v_{\pi'}(s) \neq q_{\pi}(s, \pi'(s))$ if $\pi' \neq \pi$. but I have not re-read the Sutton&Barto section to see how that relates to your comments. It is possible that in S&B, the constraints are implied or caveated correctly and we are discussing something slightly different

@ Neil Slater, when you have time, I will be happy if you can read that part (page 101, Eq.5.2) and give me a feedback as an expert.

@DSPinfinity I just re-read section 5.4 that includes that equation and related proof. I cannot see anywhere that it states $v_{\pi'}(s) = q_{\pi}(s, \pi'(s))$ nor anything similar.

@ Neil Slater,  at the beginning of page 102, he says "Thus, by the policy improvement theorem $\pi '\ge \pi$ (ie, $v_{\pi'}(s) \ge v_{\pi}(s) $" from which I concluded that he set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.


text:
In general - is Stochastic Gradient Descent a "superior" algorithm compared to Gradient Descent?
On a very informal level, if we were to compare the (classical) Gradient Descent Algorithm to the Stochastic Gradient Descent Algorithm, the first thing that comes to mind is:

Gradient Descent can be considered as a slower algorithm than Stochastic Gradient Descent, since it requires performing mathematical operations (e.g. derivatives, second derivatives) on the entire data set. On the other hand, Stochastic Gradient Descent can be considered as a faster algorithm, since it approximates the gradient using "mini batches" of observations from the data set. Logically, this makes sense : Stochastic Gradient Descent requires fewer calculations compared to Gradient Descent, therefore the same computer should take less time to perform fewer calculations compared to more calculations.

However, (in the spirit of tradeoffs) the somewhat logical counter point to the above argument is that:

Since Stochastic Gradient Descent approximates the gradient of the function whereas Gradient Descent evaluates the full gradient - one would imagine that perhaps Gradient Descent might have a better ability to find the true minimum of the function compared to Gradient Descent. However, I am not sure about this.

My Question: In the past few years, a lot of research has been done about studying the behavior (e.g. theoretical convergence properties) of Stochastic Gradient Descent (e.g. https://arxiv.org/abs/2103.14350 , https://raghumeka.github.io/CS289ML/gdnotes.pdf) which have demonstrated similar abilities of the Stochastic Gradient Descent Algorithm to converge as compared to Gradient Descent.
But are there any theoretical results which expose the above (speculated) tradeoff? At the end of the day, if Stochastic Gradient requires less computing power when compared to Gradient Descent - are there any theoretical results that suggest Gradient Descent has an inherent ability to better find the minimum of the function (e.g. perhaps less likely to get "stuck" in saddle points) since it is not relying on an approximation of the gradient? Or if Stochastic Gradient Descent is equal to Gradient Descent in this regard - could Stochastic Gradient Descent be then considered as superior to Gradient Descent?
Can someone please comment on this?
Thanks!

Don't ask a question that will lead to opinions. Ask "Is X superior to Y?" will most likely lead to opinions. Rephrase your question so that it will lead to objective answers.

Stochastic Gradient Descent empirically has shown to lead better results than classic Gradient Descent since its formulation, and today we're getting close to understand that it's not just luck, but the results of better mathematical qualities as well. SO the answer to your question is no, the theoretical results also show that SGD is better than classic GD.
Check these papers offer different explanations about why SGD tend to perform better that classic GD:

Train faster, generalize better: Stability of stochastic gradient descent
Escaping From Saddle Points –
Online Stochastic Gradient for Tensor Decomposition
An Alternative View: When Does SGD Escape Local Minima?

Quick summaries:

SGD works better cause it's less prone to generalization error, or to put it the other way around, a more stable learner compare to GD, i.e. the solutions learned with SGD are less affected by small perturbations in the training data.
SGD works better cause its randomness makes it less prone to get stuck in saddle points.
SGD works better cause it's literally GD but applied to a smoother version (convolved with noise) of the objective function we want to minimize, so again, less risk to get stuck not only in saddle points, but bad local minima as well (see image below).

Extra note:
it's not hard to understand conceptually how SGD can help escaping saddle points and local minima. When computing a single step using all gradients of all training instances like GD does, we don't leave much room for exploration of the loss function surface. So if our training instances point to a local minima that will be the solution found by GD. On the other hand, SGD compute gradients only on a batch of instances, and therefore each update is a suboptimal step toward the negative direction of the gradients. Being suboptmal, each step leave room for exploration, i.e. if an update step points to a local minima it might be that the next step point in a different direction, something that can't happen in GD since we perform a single update. Same conclusion, but different conceptualization comes from the interpretation of the last paper. They prove that SGD can be seen as GD (so again, single update step) performed on a loss function convolved with noise. Noise convolution has the effect of smoothing out the function (think as a metaphor to combine random frequencies, they are more likely to cancel each other out than boosting each other). Smoothing the loss function means precisely removing small local minima, as you can see in the last picture. The consequence is the same as in the first explanation, i.e. less chance of getting stuck in a local minima.

Thank you for your answer! I would be very interested in understanding why the added randomness and smoothness make it less likely to get stuck in saddle points!

I've updated the question

Thank you so much for your answer! If you have time, can you please take a look at this question? https://ai.stackexchange.com/questions/34389/are-bayesian-optimization-methods-better-suited-noisy-optimization-problems thank you so much for all your help!


text:
Where can I find authentic references on "categorical cross entropy" and "categorical accuracy metric"?
My Python source code uses TensorFlow and Keras to implement a neural network.
The Keras source code uses something called "categorical cross-entropy" and "categorical accuracy metric". I have searched a lot of books on NN theory, and no one talks about these two specific terms. Yes, they talk about "cross-entropy" and "accuracy metric" but there are no mentions of "categorical ...".
N.B. These terms can be found only in the so-called "Hands-on" books.
Can anyone please supply me with authentic references on these two specific terminologies?

Cross-posted: https://ai.stackexchange.com/q/34507/1794, https://cstheory.stackexchange.com/q/51092/5038.  Please do not post the same question on multiple sites.

Categorical just means that we will conduct multiclass classification. The output of the classifier is a binary vector. Each entry $x_i$ in the binary vector is a prediction whether or not the input is part of class $C_i$.
In that sense, categorical accuracy introduces nothing new: it is just the accuracy of a multiclass classifier. On the other hand, categorical cross-entropy refers to the joint entropy: $H(X_1, X_2) = - \sum{p(x_1,x_2)\log_2p(x_1, x_2)}$, where each random variable $X_i$ expresses whether or not the input is to be classified into class $C_i$. Kevin Murphy's book "Probabilistic Machine Learning: An introduction" is a great source of reference for many topics of machine learning, including cross-entropy and joint entropy.
The random variables could be mutually exclusive or not, depending on the problem.

If they are mutually exclusive, we would allow only one of the classes to have value 1 during training and use the softmax activation function.
If they are not mutually exclusive, we would allow multiple classes to have value 1 during training and use the sigmoid activation function.

what about the categorical accuracy metric?

I edited the answer to mention categorical accuracy too. Basically, categorical accuracy is not a separate concept, so you would have a hard time finding it in literature.


text:
MC control with exploring starts: do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?
Consider the following pseudocode of
MC control with exploring starts:

When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to
the randomly chosen $A_0$ as well?

When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?

No.
Under a given policy $\pi$, the action value of a state $Q_{\pi}(s,a)$ is the expected return when taking action $a$ in state $s$, and from that point on following the policy $\pi$. The action you are sampling for updates can be one that the policy would not take.
In general, when you are considering a specific action value - whether to sample it or update it - you are not concerned with how the agent got into the state and action involved. You only care what happens afterwards.
For exploring starts, you are not even concerned whether reaching any given state is possible under the policy being updated.

@ Neil Slater,  in this code (which is the Matlab implementation, they do):   "pol_pi(polInd) = unidrnd(2)-1;      % FOR EXPLORING STARTS TAKE AN INITIAL RANDOM POLICY!!!"  The link is this: https://waxworksmath.com/Authors/N_Z/Sutton/RLAI_1st_Edition/Code/Chapter_5/mc_es_bj_Script.m

@ Neil Slater, and what strange is: the  code in the above link does not work if you do not do that update. However, with update it works perfectly!

@DSPinfinity I don't know enough Matlab to tell you why they do that, but on a quick look it doesn't seem to be an implmentation of exploring starts. Instead it simulates games from the start. That still works for the blackjack environment it seems. I wouldn't recommend you use that code to understand RL.


text:
Why is Reinforcement Learning viable compared to just only using Neural Network?
I am confused as to how RL is viable compared to just using a simple NN.
I have data such as the following:
x1 | x2 | x3 | y
3  | 5  | 7  | 6
4  | 3  | 2  | 15
... and so on

where x are the input and y is the output.
Let's say I am given x1, x2 and I need to find x3 such that y is as close as 10. I can just simply do neural network on the dataset above, set the goal of y to be 10 and do a 'solver' to find the optimal x3.
My dilemma with reinforcement learning is that if you just restate this problem such that x1 x2 are now called states, x3 is now action and -abs(y-10) as the reward function, this problem seems like it can be solved using RL as well. Maybe it is just like an off-line RL? This leads to my question of how could RL be useful if just a neural network can used to handle this kind of problem, especially when
RL and NN both are identically solving for the best action or the x3, respectivley?
I am sure I am overlooking some fundamental knowledge here.
Adding on: I realized that I am missing the state transition or the 'next state' part of RL. But even for the next state, wouldn't NN be able to solve for what the best action/x3 is for that given state or the x1,x2?

Your analysis seems correct, that both approaches - searching using a neural network approximator for all three params, and using a trained reinforcement learning (RL) agent that selects the third param as an "action" - should work. The quality of the solution, given a fixed training set, is likely to be very similar if not identical. The main difference would be that training the RL agent would take longer since it is driven indirectly via the reward function and exploring alternative actions as opposed to directly from loss function on the approximation for $y$.
You note that there is no state transition (or time step) in the problem, and that does mean RL is over-specified. A related approach, called contextual bandits, would be a closer match. Again in your example, it will be slower because of trial-and-error search for solutions compared to directly approximating the function.
In general, if you have a fixed dataset and simple-to-express goal in terms of optimising a function that you know, then it can be posed as an RL problem. RL solvers will then usually work, but will most often be inefficient compared to other optimisation approaches.
RL, and learning through trial-and-error, is a very general learning mechanism. It can often be adapted and applied to scenarios where other algorithms also work. As you have done in the question, the basic approach is to map state, actions and rewards from the original problem description to frame it as RL. In your specific case, you must train the agent both offline and off-policy, because the agent cannot actively make new guesses for x3.
If this problem framing is done accurately, then RL can be applied. However, this is not always a good idea.
The more "natural" the fit of a problem is to the concepts of agent, environment, state, action and reward, then the more likely it is that RL will be a good fit to the problem. Unlike your example, it can be applied in situations where you do not have a dataset and the agent must actively collect experience.
Also unlike your example, RL can be applied in scenarios without target variables to learn, using a reward signal directly as feedback. Although sometimes explicitly learning target variables from the environment other than expected return or optimal action could help an agent (e.g. by regularising internal feature representations in its neural network if this is done using a multi-headed NN).


text:
Why does my actor-critic network always give either -1 or 1 at the output layer?
I have an actor-critic network. The state space contains continuous variables with different ranges like (0,1.57) and (-0.70, 0.70). And it also contain absolute 6D pose in the form (x,y,z,roll,pitch,yaw). The action space is continuous too in the range (0, 1.57). I apply scaling at the input layer, and scale things back before applying the action received from network. Irrespective of learning for 100 or 1000 episodes, the actor model always gives either -1 or 1. Eg: [1,1,-1,-1] which gets scaled to [1.57,1.57,-1.57,-1.57] as an action vector. Could someone give me suggestion on what's happening with the network. The learning follows DDPG algorithm.
actor_lr = 0.0001
critic_lr = 0.001
def actor(state_size, action_size):

    inputs = Input(shape=(state_size,), name="state_space")
    layer_one = Dense(300, activation="relu")(inputs)
    layer_two = Dense(400, activation="relu")(layer_one)

    outputs = Dense(action_size, activation="tanh", name="action_space")(layer_two)

    model = Model(inputs = inputs, outputs = outputs)
    model.compile(loss="mse", optimizer=Adam(lr=actor_lr), metrics=["accuracy"])
    return model

def get_critic(state_size, action_size):

    state_input = Input(shape=(state_size), name="state_space")
    state_out = Dense(64, activation="relu")(state_input)

    action_input = Input(shape=(action_size), name="actions_sapce")
    action_out = Dense(18, activation="relu")(action_input)
    concat = Concatenate()([state_out, action_out])

    critic_layer_one = Dense(300, activation="relu", kernel_regularizer=regularizers.l2(0.01))(concat)
    critic_layer_two = Dense(400, activation="relu", kernel_regularizer=regularizers.l2(0.01))(critic_layer_one)

    outputs = Dense(1, activation="linear", kernel_regularizer=regularizers.l2(0.01))(critic_layer_two)

    model = Model([state_input, action_input], outputs, name = name)
    model.compile(loss="mse", optimizer=Adam(lr=critic_lr), metrics=["accuracy"])
    return model

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ and $1$ in the output.
Let us execute the following code to get a better idea:
import tensorflow as tf
tanh_x = tf.keras.activations.tanh(-8.0).numpy()
print(tanh_x, type(tanh_x))

At least in my machine, the output will be exactly $1.0$ for tanh_x variable of type numpy.float32, which has a precision of up to $7$ decimal digits.
The value of tanh_x is in fact $-0.99999977493$, but Python only saves the first $7$ decimal digits and decides to round it up to $-1.0$. Every output value from the neural network outside the range of $[-8, 8]$  will be exactly $-1$ or $1$ after tanh activation and float32 precision. As you can see, if the network randomly outputs values they will almost surely be outside $[-8, 8]$ as this is a very short range.

Thank you for the comment and example. Does it also mean the network is not well configured(not enough layers and neurons)?

@Goldfinch Honestly, I think your network is too large with 120k parameters. You could begin by reducing dense layer sizes to 30 and 40 instead of 300 and 400.


text:
What is meant by non-linearity in Convolutional Neural Networks? And why do we focus on removing it entirely?
I'm aware of the working of ReLU that it's turns every negative value to zero and doesn't effect any positive value, but what confuses me is that: what is actually meant by Non-linearity in feature map after Convolution operation on any image? What actually is non-linearity? And why do we focus on removing it completely from the feature map?
Thank you.

Who said that "we focus on removing it completely from the feature map"?

The concept of non-linearity is not only restricted to Convolutional Networks but can be seen in RNNs, and any feed forward neural networks.
Without a non-linear activation function, two feed forward layers can be collapsed into a single feed forward layer. So, let's say you have a large neural network with 5 layers possessing weights W1 to W5.
Without a non-linear layer, passing an input x would give you the following:
y = W5(W4(W3(W2(W1(x)))))
which is equivalent to y = W(x) where W is a product of W1..W5
which is essentially a single linear layer -Hence, this would not allow the neural network to learn higher order or more complicated functions which are not linear in nature.
The presence of a non-linearity between the layers allows the weights to be adjusted in order to learn complex functions apart from linear functions. Eg. if your data might be quadratic in nature, or say some complicated mixture of logarithmic, sinusoidal functions, hyperbolic functions, etc.

You can use mathjax on this site, so I recommend that you do that.


text:
Do you think this method of creating AI is valid?

If the agent wins the game, the agent is rewarded

If the agent wins the game, the game is rewarded

Both the agent and the game are reinforcement learning agents.

When the agent wins the game, the game is rewarded, so the game creates various game winning conditions.
As the agent satisfies various game winning conditions, the reward increases, so the agent develops to win various games.
An agent that can win various games develops into a general-purpose intelligence

Your question will lead to opinions. What do you mean by "valid"? Also, why do I think that you're the same person asking the same questions again and again, always the same question about games that lead to AGI or games that are evolved? It's annoying. Stop spamming our site with these poor questions.

isn't the optimal behaviour for the game "always win" and for the agent "do whatever"?

What you are describing is a teacher-student framework in which you have one agent (the teacher) who determines the complexity of the task that another agent (the student) must accomplish. If the task is too difficult for the student to accomplish the teacher receives little to no reward and therefore changes the environment (the game) making it easier for the student to complete it.
The paper Automatic Goal Generation for Reinforcement Learning Agents is quite interesting because it applies this concept using a Generative Adversarial Network (GAN). Here the generator network is the teacher and the adversarial network is the student trying to solve the task created by the generator network. The results in the paper demonstrate that no prior knowledge of the environment is required for the student to solve it and it can even be done with sparse reward, i.e., 1 for winning the game and 0 otherwise.


text:
Why do neural network weights have to be between 0 and 1?
I've been reading about neural networks for a long time, and I saw that in each one, the weights are always between 0 and 1. Why is this? I tried programming one, but the sigmoid function just seemed like one more thing to make it more complicated, without need. Why couldn't the range of a weight or neuron value be infinite?

Having the weights between 0 and 1 helps accelerate learning. They do not have to be between 0 and 1.
Typically the weights get normalized to [-1, 1]. But it also depends on your problem.

Can you provide some evidence for the claim "Having the weights between 0 and 1 helps accelerate learning."? Have you observed this empirically? If yes, it might be a good idea to provide the link to such an experiment. If you have seen this written in a research paper or book, cite it.


text:
Dimensions of a Transformer model and purpose of masking
I'm currently studying the Transformer model (Attention is all you need) and after reading it I still have some questions for which I get conflicting answers if I google them:

What exactly are the dimensions of the input to the encoder of a transformer, from what I've seen you can input sentences with dynamic lengths but in the paper it seems like all layers expect the K/Q/V matrices to have dimensions of d_model, d_k or d_v/d_q
Same question for the decoder, what exactly are the input dimensions and how do the attention layers handle dynamic dimensions if its possible
Another question is is masking only used for training where you input the whole sentence as the input to the decoder or does it have another purpose than that (especially in inference)

Ask only one question per post, even if you have multiple questions and they are related to the same topic!. So, edit your post to leave only one specific question, and put it in the title. "Dimensions of a Transformer model and purpose of masking" is not a question and it's also quite vague.

Transformer networks are great, because they can handle variable length inputs, but they also have limitations, concerning the input size. For example BERT (a transformer based language model) only accepts $N = 512$ input tokens at most. They way transformer models can accept sequences shorter than the $N$, is by padding the input sequence with zeros and masking. When inputting a sequence with length $n \leq N$, the mask $m$ is a vector of size $N$, where $m_i = (i \leq n)$. Say we have a transformer that should translate from English to German which accepts $N = 8$ tokens at most, and we input the sentence "Today is Tuesday". Let the words have the following token values:
\begin{array} {|r|r|}
\hline \text{[SOS]} & 1 \\
\hline \text{Today} & 274  \\
\hline \text{is} & 12 \\
\hline \text{Tuesday} & 125 \\
\hline \text{[EOS]} & 2 \\
\hline  \end{array}
The first input will always be occupied by the [SOS] token and each sequence is followed by an [EOS] token.
Now the input looks like this:
\begin{bmatrix}1\\274\\12\\125\\2\\0\\0\\0\end{bmatrix}
And you also supply an attention mask (using integers representing booleans) which looks like this:
\begin{bmatrix}1\\1\\1\\1\\1\\0\\0\\0\end{bmatrix}
This means, each position can only attend to positions $i$ where $m_i = 1$. Therefore, positions where the mask is 0, do not affect the output of the model.
The vector representation of the mask above is the simplified version of the actual attention mask, describing that position at row $i$ can attend to positions at column $j$:
\begin{bmatrix}1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\\0&0&0&0&0&0&0&0\end{bmatrix}
The decoder works a bit differently during training and inference.
Training:
The decoder attention mask is built, so that each position can not attend to future positions. You can describe this by using a matrix:
\begin{bmatrix}1&0&0&0&0&0&0&0\\1&1&0&0&0&0&0&0\\1&1&1&0&0&0&0&0\\1&1&1&1&0&0&0&0\\1&1&1&1&1&0&0&0\\1&1&1&1&1&1&0&0\\1&1&1&1&1&1&1&0\\1&1&1&1&1&1&1&1\end{bmatrix}
When using this mask for the decoder, position $i$ can only attend to positions $j, j \leq i$.
For the task of translation from English to German (e.g. "Today is Tuesday" -> "Heute ist Dienstag"), you can provide the model with the entire input sentence and the entire output sentence. The model will make a prediction for each decoder input position, what the next word will be. And by using the mask, no information leaks into earlier positions (when predicting the next token for the word "ist", the model can only see "[SOS] Heute ist").
Inference:
For the inference stage, you would first provide the input sentence for the encoder and for the decoder only the [SOS] token. Now you sample a word from the transformer output (e.g. picking the most likely or sample from the output distribution) at the position of your last decoder input (in this case the position of the [SOS] token, e.g. position 0). Then you do the next inference step, by again providing the input sentence for the encoder, and for the decoder you provide the [SOS] token + the predicted word from the last step. If the previous prediction was "Heute", your decoder input now is "[SOS] Heute". You just repeat this process, each time providing the so far predicted sentence to the decoder, until the transformer predicts [EOS].

thanks so much this cleared out a lot of questions i had. One last question, does this mean that for the first step in inference you also input a (n_vocab, n_model) dimensional matrix to the decoder with a an attention mask with just a 1 at the first posiiton (for the first [SOS] token) and then for each step you expand the mask?

For inference you also just use the lower left triangular matrix as mask, just like in the training stage. This mask makes sure each prediction can't use future positions and due to the nature of it, it "expands" automatically. You could think that it would make more sense if e.g. pos 2 could also attend to pos 4 after predicting the first 5 words, but as during training this was not possible, changing this for the inference stage would not make sense. I think this is also a slight downside to transformers, as information in the decoder flows unidirectional. Glad I was able to help :)

Thanks so much, I wish I could upvote, I'll come back to this as soon as I get 15 reputation ^^

Would it be possible to generate an entire German sentence in one go, in multiple passes, by repeatedly feeding back the previous iteration of the sentence into the transformer, with a square/rectangular mask? Would that be useful? Then, the transformer would not be constrained to generate the output one word at a time.

I think what you mean would be equivalent to doing one word at a time/ that is what you are doing during the inference stage of a transformer. If I understand you right you just want to embed this process into the network architecture?


text:
Does Using the Same Background for Binary Classification Improve Model Accuracy?
I am training a CNN that detects if a there is a pot of boiling water vs if there is a pot of boiling water with pasta inside. My hypothesis is that having the same background for both a positive and negative class image will improve model accuracy because it will force it to look exclusively at the foreground for hints. Is this hypothesis reasonable?

You're not wrong, but you're also missing a couple of risky aspects that makes the idea not the best.
The part you're right are:

forcing a model to focus on specific regions of interest of an image will improve a model performance.

The points you should pay attention to are:

using the same background for every image will make your dataset biased, i.e. you won't be able to use your model in any real application cause the model will work only on images with specific backgrounds.
the power of deep learning relies in the fact that models learn features automatically. Many times these features capture correlations that are totally invisible to us, correlations that might involve part of the background as well. So before jumping to  conclusions an inspection of feature maps and maybe feature importance using some explainable ai tool is always required.
feature extraction is always a good practice (if we can help a model with some prior knowledge, why not?) but it should aim at adding information rather than removing it. So I would personally try other roads. For example testing different color spaces where the color of the pasta become more visible from the water/pot ones, or use some form of attention and let the model learn where to focus.

Your suggestions seem valid to me. I would also differentiate between training "performance" (which is somehow a vague term and might refer to the accuracy) and generalisation, which you mention implicitly in the first point. For the last point, you may also mention data augmentation.


text:
what does the OpenAI ALE/Breakout-RAM-V5 observation return
I haven't been able to understand the output that OpenAI gym return for observation from this snippet
env = gym.make('ALE/Breakout-ram-v5', render_mode='human', obs_type='grayscale')
obv = env.reset()
print(obv)

from the OpenAI documentation it suggest it return the pixel data from camera, I am guessing that it a the game state representation at a given step. (I might be wrong here, I am still new to OpenAI)
Assuming if it is the pixel data from camera, How can I retrieve information like where the paddle position, the ball velocity (incl. direction) such that it satisfy Markov Properties for Q-Learning

Your question is off-topic here. Asking about software is off-topic here. See our on-topic page: https://ai.stackexchange.com/help/on-topic.

Founded Documentation on Mapping Annotation: Mapping of RAM indexes to semantic state variables from mila-iqia
https://github.com/mila-iqia/atari-representation-learning/blob/master/atariari/benchmark/ram_annotations.py


text:
Image Recognition Method, calculate deviation from rectangular grid
I have a set-up which creates pictures of a grid that is a bit bend towards the ends, and I need some kind of program that can calculate the deviation, resp. it just needs to be some kind of indicator, a stronger deviation (strong bend) leads to higher number and something close to a perfect rectangular grid will lead to a number close to zero. What possibilities do I have to do that? I'm just asking for the Method. I can't use neural networks since I have little data, I can't generate lots of pictures. Is there any other method to implement a solution to that problem?
Here's an example picture of the grid:
I need a program that calculates the mean deviation of the red lines from the perfect rectangular grid. What methods exists for such kind of problem apart from neural networks?

why not just computing the distance ($y_{standard} - y_{curved}$) between the standard grid and the curved one?

thank you for your reply, how can I compute the distance? I don't have any units of length or anything like that I only have the picture, which I could convert into pixel data (a matrix of zeros and ones representing white and black resp. red)

So, here's an idea, not perfect but it should give you at least a starting point.

Start by converting your image into a binary grayscale one, to detect the lines of both curved and straight grid.
Detect keypoints on left and right sides of the grayscale mask (see plot below)
use the keypoints to compute distances.

The last step is the most tricky. Theoretically it should be possible to detect if 2 points belong to the straight grid by comparing their values (cause the y coordinate should be the same). In practice it's not always the case. This detection is important to understand which pairs of values to subtract (point that belong to straight grid - next point in the list, curved one). Also, if all points belong to the straight grid, you know you have 2 straight grids.
I think its possible to refine the methods by getting pairs of left/right points and computing the slope of the line they form, if the slope value is lower than a threshold (close enough to 0) then again we can safely assume the 2 point belongs to the straight grid.
I didn't implement yet this step, but if you find it hard I could try, just don't want to spend too much time on it to then find that is not useful for you
Note also that the same procedure could be then applied not only on the edges bu to all vertical lines of the image.

code so far:
from skimage.io import imread
import matplotlib.pyplot as plt
import numpy as np

def main(filename):
    img = imread(filename)

    # convert to binary grayscale for convinience
    gray = img.mean(axis=2)
    gray = gray < 255

    # get indicies of keypoints on both side of the image
    y_right = np.argwhere(gray[:, -1] != 0)[:, 0]
    y_left = np.argwhere(gray[:, 0] != 0)[:, 0]

    # get values as well
    y_right_values = np.where(gray[:, -1] != 0)[0]
    y_left_values = np.where(gray[:, 0] != 0)[0]

    # generate x coordinates [0 and width of image]
    x_right = np.array([gray.shape[1] for _ in range(len(y_right))])
    x_left = np.array([0 for _ in range(len(y_left))])

    # get points with same y coordinate (belonging to straight lines)
    distances_right = compute_distance(y_right_values, y_left_values)
    distances_left = compute_distance(y_left_values, y_right_values)
    print(
        f"Average distance cruved vs straght: {(np.mean(distances_right)+np.mean(distances_left)) / 2}"
    )

    # plot
    plt.imshow(gray)
    plt.scatter(x_right, y_right, color="r")
    plt.scatter(x_left, y_left, color="r")
    plt.show()

def compute_distance(points, points_reference):
    """Compute distances between keypoints

    Parameters
    ----------
    points :
        points for which the distances should be conputed
    points_reference : TYPE
        points on the other side of the image, used to check if a point belong to the
        straight grid
    distances : TYPE
        list filled with computed distances

    Returns
    -------
    TYPE

    """
    distances = []
    points = [p for p in points]
    ref = [p for p in points_reference]
    while points:
        # if point share value with a point to the left, it's part of the straight grid
        if points[0] in ref:
            distances.append(np.abs(points[1] - points[0]))
            # remove points from list
            points.pop(0)
            points.pop(0)
        # if first point doesn't belong to the straight grid, skip
        else:
            points.pop(0)
    return distances

if __name__ == "__main__":

    filename = "HCi6P.png"
    main(filename)

  [1]: https://i.sstatic.net/ePPQQ.png


text:
Confusing statement in Sutton-Barto's RL book in Section 8.5 ( Expected vs. Sample Updates)
In Sutton-Barto RL's book (page 174) it says:

The advantage of sample updates shown in Figure 8.7 is probably an underestimate of
the real effect. In a real problem, the values of the successor states would be estimates
that are themselves updated. By causing estimates to be more accurate sooner, sample
updates will have a second advantage in that the values backed up from the successor
states will be more accurate. These results suggest that sample updates are likely to be
superior to expected updates on problems with large stochastic branching factors and
too many states to be solved exactly.

This is very confusing because before the above paragraph they say

The values at the next states are assumed correct,...

If so, the results in Figure 8.7 for sample updates should be worse in real.

If you can, please, put your specific question in the title.

If so, the results in Figure 8.7 for sample updates should be worse in real.

Yes, the sample updates will perform worse than shown. The later paragraph is explaining that the results for expected updates may be even worse relative to sample updates. It is not claiming that sample updates are unaffected if you remove the simplifying assumption used in the comparison.
The assertion on page 174 is that the disparity between the approaches may in practice persist for longer than the idealised situation (of already having correct values to backup/bootstrap from), because it may take longer to converge the expected values from the original estimates than the sampling approach would take. This is due to the same effect impacting each time step in a longer trajectory, whilst figure 8.7 shows the impact only for a single time step.

@ Neil Slater, thanks for the answer. Sutton-Barto's explanation style is horrible in most places. Their English is also problematic in my opinion.


text:
What is the meaning of step (e) in the prioritized sweeping algorithm? Why is P calculated like that?
Following is the "Prioritized Sweeping" algorithm in Sutton-Barto's RL book (page 170).  What is the meaning of step (e) in the prioritized sweeping algorithm? More importantly, why is P calculated like that?

$P$ tells you how "off" the evaluation for $Q(S, A)$ is. If the difference between $Q(S,A)$ (current best guess) and $R + \gamma\max_a Q(S', a)$ (update-value for $Q(S, A)$ since you received $R$ and were moved to $S'$ when you did $A$) is large, then it is an indication that this is a state-action-pair you should consider to learn more about, and $(S, A)$ will be close to the front of the queue. On the other hand, if $P$ is small, there is "nothing new to learn", and no need to prioritize this state-action-pair.


text:
Factors that affect the number of iterations of value iteration
I had an assumption that value iteration will take more iterations to converge if the map size increases/environment's complexity increases.
I tried to verify this idea by running value iteration on randomly generated grid worlds of increasing sizes (from 5X5 grid world, 6x6 grid world ... to 50x50 grid world). In each grid world, there are 4 states with reward value of +1. All other states have reward value of 0, -0.03, -1.
Strangely, I noticed that although the computation time for value iteration to converge increased significantly (which is expected since the time complexity of each iteration increases as the grid world's size increases), number of iterations it took for value iteration to converge remained the same at 66 across all map sizes. This contradicts with my assumption that the number of iterations of value iteration should generally increase if the size/complexity of the environment increases.
I  am quite sure my code implementation is correct (I have checked my code and policy/values obtained from the value iterations), and I believe the stop condition that I set for the value iteration is reasonable (I stops value iterations if the maximum change in the values in the current iteration is less than 0.001).
Could anyone explain this phenomenon, and if there are factors that may affect the number of iterations of value iteration?

I think your results only make sense in a continuing (non-episodic) environment. Is that that case?

Yes, the environment has no terminating state. Thank you for the detail explanation. I have a clear understanding of the issue now.

What was your value for $\gamma$? I'm guessing at $0.9$?
What is $0.9^{66}$? It is $\approx 0.000955$ which is less than $0.001$, whilst $0.9^{65} \approx 0.001061 \gt 0.001$
What you are measuring with the 66 max iterations in your chosen environments and problems is the point at which the signal from the backed up reward signal decays below your chosen accuracy threshold. This is also showing an interesting limit for those values - in an environment with maximum reward 1 and optimal trajectories longer than 66 (or with a loop repeat length greater than 66), your chosen values for $\gamma$ and $\theta$ may not be able to find the optimal policy.
It is also showing an interesting effect in that even for small environments, value iteration will keep refining the value accuracies, even if the optimal policy was found much earlier. In the small environments (e.g. the 5x5 grid), the chances are that value iteration could be used after much fewer iterations to generate the optimal policy, but it has no stopping criteria for that - instead it keeps resolving the value calculations until they stop changing by the cutoff you have set.
This effect you noticed should disappear for value iteration when there is a maximum trajectory length less than the "signal attenuation threshold" that you have created. For example, if reaching a specific location also terminated the episode and did so whilst maximising the return. In that case value iteration should behave like a "flood fill" and should terminate once the longest possible optimal trajectory has been calculated.


text:
How to create a dataset for binary classification
I would like to classify whether a pot of water is boiling or not using a CNN.
Is it enough to take pictures of boiling water using only one pot, or should I use different pots for this to generalize well?

It depends of your application. If your aim is to have a general boiling detector, yes, if not, don't bother yourself.
I would suggest, if you can, to crop the picture of where the water is, to not be inflenced by the overall context.
I am pretty sure you will still have good results even with one pot. But you may see some accuracy drop when using different color pot during test time (if that color haven't been seem during training). Some data augmentation with color tweaking will help. That will redirect the network into texture analysing rather than color analysing.


text:
ontological tree for the concept of a word (eg "chair")
I am novice to AI, but I would like to learn the general idea that a machine understands the concepts in a text document.
I would like to ask wether there is an ontological tree of concepts, like a higher class of OBJECTS in which there belongs the FURINTURE, in which there belongs the CHAIR and the TABLE. Is such a thing in NLP? If yes, could someone provide me a way to find such relationships?

What were your findings before asking the question? This site is not a human search engine. You should have done a little bit of research on your own about your question before asking it, and share your findings with us. Please, next time, do a little bit of research and share with us the results before asking.

There have been a few notable major attempts to build language-processing systems that use directed graphs (similar to, but more general than trees) out of word associations. Variations of your idea, called Symbolic AI, have been used in NLP projects of all sorts and sizes. It is still a common approach for modern chatbots for instance, that use databases of word types, synonyms and relationships.
One well-known open-source project to do this is called ConceptNet. There are also initiatives such as CYC which attempt to encode these rules into inference engines.
A lot of recent focus has instead been on very large statistical models, e.g. GPT-3. These have performed far better at tasks such as text summarisation, text generation and translation than knowledge-encoding directly by engineers, despite the impressive sizes of CYC and ConceptNet.
However, some problems remain with the large statistical models. They often produce grammatically sound but meaningless content that defies common sense. It is not clear how this will be addressed going forward, and in the short-to-medium term at least it is possible that hybrid systems that attempt to combine the best qualities of each approach will be the most practical way to accurately process NLP tasks.


text:
Why are GAN models not heavily used for NLP?
I am wondering why there has not been more usage of GANs for NLP. I know there has been research on the subject (The Google Scholar page for the subject is here).
Are there any specific reasons why GANs do not work for NLP specifically VQGAN + CLIP variants? I do not understand why most text generated by AI is done through predicting the next letter or word in a sequence with RNNs when GANs have had so much success generating deep fakes and the such instead of say, predicting the next pixel.

A few reasons:

Transformers are amazing at text generation already (e.g. GPT-3 which almost passes the Turing-Test)
The original GAN requires a continuous data representation (e.g. images) instead of a discrete one (e.g. text), so that slight error signals can be used for learning.
Empirically speaking, GANs don't seem to work that well on non-image data. I recently applied them to regular tabular data but found auto-encoders much more useful.

P.S. Also it's my intuition that GAN's and Denoising Diffusion Models are popular in the image generation community because they both offer mechanisms that can prevent "blurriness" in the generated images (either the adversary or extra denoising iterations).

Please, remove the very arguable sentence "which essentially passes the Turing-Test". Alternatively, can you provide some real evidence for it? If it generates some meaningful text it doesn't mean passed the Turing test. I'm sure it doesn't pass the imitation game.

I edited it to include a reference, and say 'almost passes' since it still isn't great at answering nonsense questions.

I should mention however that when GPT-3 is primed with examples (via meta-learning) of calling out nonsense it does learn the skill immediately: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/


text:
How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?
Sutton-Barto's RL book (page 203)

Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a
1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and
144). The states are numbered from 1 to 1000, left to right, and all episodes begin near
the center, in state 500. State transitions are from the current state to one of the 100
neighboring states to its left, or to one of the 100 neighboring states to its right, all with
equal probability. Of course, if the current state is near an edge, then there may be fewer
than 100 neighbors on that side of it. In this case, all the probability that would have
gone into those missing neighbors goes into the probability of terminating on that side
(thus, state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance
of terminating on the right). As usual, termination on the left produces a reward of
−1, and termination on the right produces a reward of +1. All other transitions have a
reward of zero. We use this task as a running example throughout this section.

Question:  I did not understand how state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right. Could someone explain this calculation in detail?

State 1 has zero states on the left, and 100 on the right. The normal probability of getting to a state is 1/200; the remainder from 1.0 indicates termination.
So State 1: 100 x 1/200 = 1/2 = 0.5, which is the probability of reaching any of the 100 states on the right. The remaining 0.5 is thus assigned to terminating, as there are no left states that can be reached.
State 950: you have 100 states on the left (so a 0.5 probability of reaching any of them), but only 50 states on the right. So you have 50 x 1/200 = 1/4 = 0.25, which is the probability of reaching any of the 50 states on the right. This leaves a probability of 0.25 to terminate.