text:
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$
and the discounted total reward $J(\pi)$ as:
$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$
I have two questions regarding these equations.
Question 1
Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.
I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.
I have read in this question that "$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$". I have confirmed that this is true, but I don't understand why.
I tested this with a simple example:
Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.
Then:
$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$
and
$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$
So indeed if we sum them and multiply by $(1-\gamma)$ we get:
$$(1-0.9)\cdot(4+6) = 1$$
Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?
Question 2
Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.
Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?

Please, next time, ask only one question per post, even if you have related questions. If you have more than 1 question, ask each of them in their separate post.

Question 1
The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows
$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$
When you multiply by $1-\gamma$ you get
$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$
Which can be equivalently written as
$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$
Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities

Question 2
Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this
$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_
{a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$
The discounted return form is usually denoted
$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$
Where you notice the discounting terms aren't normalised which motivates cancelling it out

So from what I understand you are saying that $(1-\gamma)$ is used to normalize (i.e., get the sum of values to be equal to 1) and $(\frac{1}{1-\gamma})$ is used to cancel that normalization?

precisely that.


text:
Preparing data set for the YOLO algorithm
Hi I am working on a project which requires the You Only Look Once algorithm in order to classify and localise objects within images. I have to prepare my dataset (which has 2 classes, and predicts 6 objects per grid cell, and the 448 * 448 image is split into a 7*7 grid). What would be a viable approach to do that? I found  this code, found in this  article. However I do not understand why he has done what he has done, e.g why is he specifically checking the 24th element of the “box”, and so what element of the box would I have to check? Is there any tutorial running through that? Would it be possible for someone to explain or even adapt his approach to fit my dataset?
FYI: I am coding the YOLO algorithm from scratch

You can check this playlist for relevant information: https://www.youtube.com/playlist?list=PLKHYJbyeQ1a3tMm-Wm6YLRzfW1UmwdUIN

Ok, let go step by step.
What you are working on is YOLOv1, in this version of the YOLO algorithm, the maximum bounding boxes that the model can return is 7x7 = 49 boxes as 49 cells since the output shape is 7x7x30.
For each box, the depth of output is 30 because the number of labels of PASCAL VOCS 2012 is 20 (the author of YOLOv1 trained on this dataset) so from index 0 to index 19 will represent the label of that bounding box. From 20 to 23 are the position and size of that box.
The 24th represents 2 things, first is the confidence of that box, since this is ground truth so the confidence should be 1. Second, you know that YOLOv1 can only return 49 boxes at maximum (actually, you can edit the number of boxes by yourself) so the ground truth should only handle one per cell, hence the 24th is the binary value to make sure there are no duplicates bounding boxes in one cell. (25 to 29 is because the author predict 2 bounding boxes per cell)
In your case, the output should be 7x7x(2 + 5 x 2) = 7x7x12 with 2 classes and 2 boxes per cell.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
OpenAI Gym: Multiple actions in one step
I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the next round starts. The action for one user can be model as a gym.spaces.Discrete(5) space.
I want my RL agent to make decisions for all users. I'm wondering how to take multiple actions before progressing time and calculating the reward.
Basically, what I want is:
obs = env.reset()
user_actions = []
for each user:
    user_actions.append(agent.predict(obs))
obs, reward, done, _ = env.step(user_actions)

So the problem is that I don't immediately know the reward after getting an action since I need to collect all actions before evaluating the round.
I could of course extend actions to include actions of all users in one go. But this would be problematic if I have a really large number of users or it even changes over time, right?
I found these two (1, 2) related questions, but they didn't solve my problem.

What was the format of action space you used. Was it Discrete or Multi Discrete? and how did you map that action space to your dict for passing into your step function ?

@kikram See my question and answer below. Here, my actions were discrete and the dict, in my answer, maps agent ID to agent action.

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.
I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.
action = {}
for agent_id, agent_obs in obs.items():
    # get the policy of the current agent
    policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
    # use the policy and the agent's observations to compute its next action
    action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)

My environment's step(self, action) function expects action to be such a dict and knows how to handle it.
It applies all actions in the dict before calculating each agent's reward and progressing time in the environment.
So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.
For the multi-agent RL approach, I used PPO and the Ray RLlib framework.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
I have come across a Google paper that uses the REINFORCE algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be only one step.
When trying to replicate the experiments they propose I found that there are some problems with the stability of the method (maybe that's why it is not accepted by peer review).
Researching on my own I have found something I suspected, and that is that the problem they present could be solved as a multiarmed bandit problem in THIS link. But because of this event, I have the doubt if using methods based on trajectories (such as Policy Gradient Methods) has some mathematical problem in situations where the trajectory is a single step.
PS: I think the problem of this paper may be also that they average only after one execution of a trajectory and not over k trajectories as it is necessary for a Policy Gradient Method, so I would also like to know the opinion of more people about this issue.

You say that the paper is not accepted by peer review but I see that it's accepted to "Proceedings of the 37th International Conference on Machine Learning". There is nothing wrong with using PG methods when there is only 1 step, the validity of approach does not depend on the length of the episode, whether it is useful or not is another question.

True, I was mistaken about that. What do you think about averaging k trajectories?Recoding the problem, I face the issue that as I improve the episodes of the process, every data value converges to a value of 0.5 and I cannot figure out why.

yes, you can take a batch of samples, average policy gradients and then apply the result, seems reasonable to me

The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e
$$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\
&=E_{\tau\sim p(\tau)}[R(\tau)]
\end{align}$$
Where $\tau$ represents the probability of selecting a particular trajectory, if the trajectories all have fixed length then $\tau$ only has non-zero probability for trajectories of the specified length which in this case is 1.
The REINFORCE algorithm takes this expression and with some simplifications (causality to improve variance) and manipulation (log trick) obtains the gradient, pretty much as simple as that.

Intuition of algorithm in paper
In the algorithm they denote the $\pi_\theta$ (which is typically reserved for describing the policy) as the probability of the selection vector. By considering instead that the function $h_{\theta}$ as the policy instead I think it can be seen that they are actually averaging over multiple trajectories.
So we instead think of the data points as state-action pairs that we pass into $h_\theta$ to get the probability of selecting said action for a given state. These probabilities then dictates whether we choose the action or not. An alternative way to interpret this as if we imagine the action space for each state as binary then we can think of the "other action" as not impacting the predictor.
The gradient used for updating the parameters associated with the policy uses the log probability of $\pi_\theta$ which if we expand it, expressing it using $h_\theta$, (as done on page 4)  we can see it's the sum of the log probabilities of selecting (or not selecting) each data point.
By considering the return for each data point constant (each data point has the same loss incurred by predictor model on the validation set) and absorbing the average over batch size into the step size $\beta$ it could be interpreted as an average

Stability issues
RL is plagued with stability issues, be it selection of hyper parameters, random seeding etc. It's hard to often pinpoint why results aren't exact but as long as you get something in a similar ball park i'd say thats pretty good going


text:
What is this algorithm? Is it a variant of Monte-Carlo Tree Search?
I'm using a Neural Network as an agent in a simple car racing game. My goal is to train the network to imitate a brute-force tree search to an arbitrary depth.
My algorithm goes something like the following:

The agent starts with depth 0
Generate a bunch of test states
For each test state s:

For each action a:

s' = env.step(s, a)
for x in range(agent.depth): s' = env.step(s', agent.predict(s'))
calculate reward at state s'

Set the label for test state s as whichever action a produced the highest reward

Train the agent using the test states and labels, and increment agent.depth
Loop until desired depth

The idea is that an agent trained in this way to depth N should produce output close to a brute-force tree search to depth N...so by using it to play out N moves, it should be able to find me the best final state at that depth. In practice, I've found that it performs somewhere between N and N-1 (but of course it never reaches 100% accuracy).
My question is: what is the name of this algorithm? When I search for tree search with playout, everything talks about MCTS. But since there's no randomness here (the first step is to try ALL actions), what would this be called instead?

How are you picking your set of test states?

I iterate through the range of possible states (at regular intervals). Sometimes I augment this dataset by taking each one and playing through it a few random moves (saving each state as it goes).

I am not sure I understand. What is the "range of possible states", and do you really mean "iterate" as in you use all of them in sequence? If so, what does augmentation give you (because you already have all states in the dataset)? Did you perhaps mean that you sample a set number from all possible states, or is perhaps the iteration somehow spread out so that only part of it is completed for each generation step?

The state includes position, angle, and velocity. So, for example angle: [i * 2 * math.pi / 10 for i in range(10)]. By doing this for position and velocity as well (within reasonable limits), I get a series of states spread throughout the state space. (Why is this relevant? My question is more about the approach to tree search, not how I come up with test data.)

It is relevant because I think that you are not really doing tree search at all. Your approach is only working because of the state sampling. I want to ascertain what that is in order to be sure. So, to be clear the state space is continuous, and you are sampling points from it in a grid pattern?

Yes, continuous state space, sampled in a grid pattern. I kinda get that it's not "really" tree search (or rather, it's tree search to depth 1, followed by a NN-guided playout), but then why does it produce results similar to a real tree search? For example, if I do 3 iterations of training, its performance is somewhere between tree search of depth 2 and 3. If I go to 4 iterations, the performance is between tree search of depth 3 and 4. (The deeper I go, however, the less accurate it gets...but I would assume that's due to the network architecture/hyperparameters...)

I should note that actually, I get better results if I skew my grid pattern to have finer granularity near the points near the turns in the race course. My guess is that, when far from a turn, the correct answer is simple - just head for the turn. When closer, finer control is needed to turn/accelerate at just the right time to maximize future rewards.

Thanks for the clarifications. I'll explain why I think it works and what the limitations are likely to be in an answer soon-ish.

In short it looks like you have constructed a valid reinforcement learning method, but it does not have much in common with Monte Carlo Tree Search. It may have some weaknesses compared to more established methods, that means it will work better in some environments rather than others.
Your approach may be novel, in that you have combined ideas into a method which has not been used in this exact form before. However, it is following the principles of general policy improvement (GPI):

Estimate action values of a current policy.

Create a new policy by maximising action choices with respect to latest estimates.

Set current policy to new policy and repeat.

Your method covers exploration with a deterministic policy by sweeping through a list of state and action pairs. This resembles policy iteration, or perhaps exploring starts in Monte Carlo control. It evaluates each state/action pair by following the current policy up to a time step horizon. This has some weakness, but it may work well enough in some cases.
The depth iteration is more complex to analyse. It is not doing what you suggest - i.e. it does not make the whole algorithm equivalent somehow to a tree search. Technically the value estimates with a short horizon will be poor, biased estimates of true value in the general case, but on the plus side they may still help differentiate between good and bad action choices in some situations. It may even help as a form of curriculum learning by gathering and training on data about immediately obvious decisions first (again I expect that would be situational, not something you could rely on).
Overall, although you seem to have found a nice solution for your racing game, I think your method will work unreliably in a general case. Environments with stochastic state transitions and rewards would be a major problem, and this is not something you could fix without major changes to the algorithm.
I could suggest that you to try one or more standard published methods, such as DQN, A3C etc, and compare them with your approach on the same environments in different ways. E.g. how much time and computation it takes to train to an acceptable level, or how close to optimal each method can get to.
The main comparison with Monte Carlo Tree Search is that you evaluate each state, action pair with a kind of rollout. There is lots more going on in MCTS that you are not doing though, and the iteration with longer rollouts each time is not like anything in MCTS and does not compensate for missing parts of MCTS in your algorithm.

Wow, that's a lot to digest. Indeed, my environment is deterministic, so I'm not worried about randomness. Could you elaborate on what the weaknesses are (would they only apply to stochastic environments)? Also: agent.depth=0 is trivially tree search...agent.depth=1 is tree search (but using the NN to estimate the second branch)...it's agent.depth=2 and beyond where I'm not clear on what's happening. I am working on implementing the others you mentioned..

@Kricket: The main weakness is low depth causing the valuations you are using to not be accurate. So as well as problems with stochastic environments, those with sparse rewards could be an issue. Other GPI approaches track value functions longer-term or run to end of episodes when sampling. Your method runs truncated evaluations then throws away all the data after each policy update.


text:
Should I prefer the model with the lowest validation loss or the highest validation accuracy to deploy?
I trained a ResNet20 on Cifar10 and obtained the following learning curves.

From the figures, I see at epoch 52, my validation loss is 0.323 (the lowest), and my validation accuracy is 89.7%.
On the other hand, at the end of the training (epoch 120), my validation loss is 0.413 and my validation accuracy is 91.3% (the highest).
Say I'd like to deploy this model on some real-world application. Should I prefer the snapshotted model at epoch 52, the one with lowest validation loss, or the model obtained at the end of training, the one with highest validation accuracy?

Okay, I think it's better if we distinguish loss and accuracy first via Jeremy's answer, and I agree with him with the sentence "low or huge loss is a subjective metric".
The loss value is easy to affect by noise from data and significant increase with a few error data points. My advice in this case is to use more evaluation metrics, and understand correctly what you need from your model.
For example, with Cifar 10, and you need the more correct label the better, you can believe in accuracy. However, if you want your model to make sure its result is the correct, area under receiver operating characteristic curve (AUROC) maybe the better choice.
For example, classification problem with 3 class, correct label y = 1:

Good accuracy, bad AUROC: the output probability from softmax [0.3,0.4,0.3]
Good accuracy, good AUROC: [0.1,0.8,0.1]

And with imbalanced dataset, Precision, Recall and F1-score will be more suitable.

In highly imbalanced classification problems, the highest accuracy can often be achieved simply by assigning the majority class to all observations. This is why learning algorithms do not maximize classification accuracy but minimize a loss function.
Fundamentally, loss functions capture how much you "lose" when there is a difference between the statistic you want to estimate and the estimate itself. The appropriate loss function is not given by nature, but is provided by you.
With this in mind, it is not quite accurate to say that the highest accuracy is reached after 120 epochs: it is merely the maximum accuracy achieved by the algorithm so far. Unless you run the algorithm for longer, there is no way to say if this is even a local maximum. For example, assigning every observation to the majority class may well achieve a higher accuracy than that achieved at 120 epochs. The only significance of the 120 epochs is therefore that that is how long you ran the algorithm for.
Given these considerations, it makes far more sense to stop at around 50 epochs, when your loss function is minimized.

Hi, you are right regarding the points for imbalanced datasets but Cifar10 isn't quite imbalanced. More or less, there are same amount of samples for each class.

It doesn't matter. What matters is that learning involves minimizing an appropriately chosen loss function. Unless you choose the loss function to be 1 - accuracy, minimizing losses does not correspond to maximizing accuracy.

but your model tries to minimize the training loss, not the validation loss?

That's right. You want to minimize loss with respect to the data generating process, not with respect to the data itself. In other words: the model shouldn't just fit the data you already have, but also the data that you will get in the future. You can assess how well it does this on a test set.

What about a better metric like F1? Would you ever prefer it over validation loss to save the best model?

This answer is not true. We optimize for a loss as the accuracy is not differentiable, thus one designs a surrogate loss function capturing accuracy as good as possible while staying differentiable. Most loss functions are optimal if one achieves maximal accuracy, i.e. do not help for imbalanced data. If one has imbalanced data, one has to adapt the loss (or employ other methods) to avoid learning a majority classifier.


text:
What amount of ressources is involved in building an image recognition system?
I would like to have an order of magnitude of ressources required to build an image recognition system.
Let say you want to build a startup company which main product will have to distinguish 20 different kinds of objects (bottle, dogs, car, flowers...). Images are already tagged.

How many images are needed as a learning set ? 1k, 10k, 100k, 1
million ?
What kind of hardware and how long will the learning process take ?
How many developers, how much time ?
Does it changes a lot if the number of target output is reduced to two kinds, or increased to one thousands ?

A link to a real life paper would be perfect. Thank you

"Closed. This question needs to be more focused. It is not currently accepting answers." How stupid is that ? I ask for an order of magnitude or a real life example. Of course it is not focused !

One answer is infinite amount of time because it can always be better.
Another answer is:

10k for training set
A PC with a GPU (3~4k USD), google colab (10 USD per month), or other cloud service (probably more expensive than colab)
One developer, 1 day lol
Two kinds is easier than multiple kinds
There is no paper that seeks to answer your question the way you put it. I wouldn't even recommend a paper. In fact, for you I'd recommend an AutoML tutorial. Check these. * no offence if I've misjudged your knowledge/skill level.
Here's a paper anyway :) https://paperswithcode.com/lib/torchvision/alexnet

To conclude, please be aware that your question is super open ended, and my answer is bad (but good enough for now maybe), but a good answer doesn't really exist. It's always going to be context dependent. For instance, you never said whether you need 90% or 99% accuracy.

Your answer is perfect, the question was intentionally open, you provided a real life example.


text:
Does anybody know what would happen if I changed input shape of pytorch models?
In this https://pytorch.org/vision/stable/models.html tutorial it clearly states:

All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

Does that mean that for example if I want my model to have input size 128x128 it is or if I calculate mean and std which is unique to my dataset that it is gonna perform worse or won't work at all? I know that with tensorflow if you are loading pretrained models there is a specific argument input_shape which you can set according to your needs just like here:
tf.keras.applications.ResNet101(
include_top=True, weights='imagenet', input_tensor=None,
input_shape=None, pooling=None, classes=1000, **kwargs)

I know that I can pass any shape to those (pytorch) pretrained models and it works. What I wanna understand is can I change input shape of those models so that I don't decrease my models training performance?

Each machine learning model should be trained by constant input image shape, the bigger shape the more information that the model can extract but it also needs a heavier model.
A model's parameters will adapt with the datasets it learns on, which means it will perform well with the input shape that it learned. Therefore, to answer your question "What I wanna understand is can I change input shape of those models so that I don't decrease my models training performance?", the answer is no, it will decrease the performance.
"I know that I can pass any shape to those (pytorch) pretrained models and it works."
$\Rightarrow$ this happened because Pytorch team replace all Pooling layer with Adaptive Average Pooling 2d so you can pass any shape of the image into the model without any bugs.

Thanks. As a side question, do you why tensorflow allows to have any input shape? I mean if it is pretrained model shouldn't it as well take only the shape which it was pretrained on?

I don't have much experience with tensorflow, you can check it by load the model, then print the architecture with model.summary() if you used Keras. The layer before Dense / Fully connected layer should be Adaptive pooling which help you to assign the size of the output feature map no matter what is the input shape

Ok, I'll try it, anyways, thanks for taking time to explain these question.

In all pre-trained models, the input image has to be the same shape; the transform object resizes the image when you add it as a parameter to your dataset object
transform = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor()])
dataset = datasets.ImageNet(".", split="train", transform=transform)

T.Resize(256) changes the image shape then T.CenterCrop(224) takes a random crop to help with overfitting in the training data.
The mean and std are applied to each image before they are used as an input to the model. They were determined when the model was trained, I assume they were hyperparamters.


text:
Accuracy Not Going Above 30%
I am trying to make a big classification model using the coco2017 dataset. Here is my code:
import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import IPython.display as display
from PIL import Image, ImageSequence
import os
import pathlib
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import cv2
import datetime

gpus = tf.config.list_physical_devices('GPU')
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)

epochs = 100
steps_per_epoch = 10
batch_size = 70
IMG_HEIGHT = 200
IMG_WIDTH = 200

train_dir = "Train"
test_dir = "Val"

train_image_generator = ImageDataGenerator(rescale=1. / 255)

test_image_generator = ImageDataGenerator(rescale=1. / 255)

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='sparse')

test_data_gen = test_image_generator.flow_from_directory(batch_size=batch_size,
                                                         directory=test_dir,
                                                         shuffle=True,
                                                         target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                         class_mode='sparse')

model = Sequential([
    Conv2D(265, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(),
    Conv2D(64, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(32, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Flatten(),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(80, activation="softmax")
])

optimizer = tf.keras.optimizers.Adam(0.001)
optimizer.learning_rate.assign(0.0001)

model.compile(optimizer='adam',
              loss="sparse_categorical_crossentropy",
              metrics=['accuracy'])

model.summary()
tf.keras.utils.plot_model(model, to_file="model.png", show_shapes=True, show_layer_names=True, rankdir='TB')
checkpoint_path = "training/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1)

os.system("rm -r logs")

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
history = model.fit(train_data_gen,steps_per_epoch=steps_per_epoch,epochs=epochs,validation_data=test_data_gen,validation_steps=10,callbacks=[cp_callback, tensorboard_callback])
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
model.save('model.h5', include_optimizer=True)

test_loss, test_acc = model.evaluate(test_data_gen)
print("Tested Acc: ", test_acc)
print("Tested Acc: ", test_acc*100, "%")

I have tried different optimizers like SGD, RMSProp, and ADAM. I also tried changing the configuration of the hidden layers. I also tried to change the metrics from accuracy to sparse_categorical_accuracy with no improvement. I cannot go beyond 30% accuracy. My guess is that the MaxPooling is doing something because I just added it but don't know what it means. Can somebody explain what the MaxPooling Layer does and what is stopping my neural network from gaining accuracy?

You have two questions in one.

Is it maxpool that ruins the model?

I would say no, the maxpool is a standard operation for convolution networks, it down-samples the intermediate representation to reduce the necessary computations, improve the regularization, and adds translation invariance to some degree. Originally averaging was used to downsample over few neighbor pixels, for example, 2x2 were averaged to one pixel. Then it was discovered max-pool often performs better in practice, where you took the max value out of these 2x2 pixels. The way you applied is ok in general.

Why the accuracy is not that great?

I see two issues here - first one is COCO dataset is not a classification dataset. It's an object detection dataset and there are many objects on the same image. I.e. there is an image with a person on a bicycle and a car behind him. Which class the model should assign - a person, a bicycle, or a car? The model can't know. To check if it's the issue try top-5 accuracy - it tells if the correct answer would be among top-5 guesses of the network. I would also recommend to watch the images and try to manually guess the class for few dozens of them, that would help to build the intuition
The second thing is that your model is not that deep and 30% accuracy is not bad, i.e. the random guess would be around 1% and your model doing x30 times better. You could try models like resnet - it's still quite fast, but should be doing noticeably better.

That is the exact explanation I was looking for. I truly appreciate it!


text:
Validation Accuracy remains constant while training VGG?
I posted this question on stackoverflow and got downvoted for unmentioned reason, so I'll repost it here, hoping to get some insights
This is the plot

This is the code:
with strategy.scope():

  model2 = tf.keras.applications.VGG16(
    include_top=True,
    weights=None,
    input_tensor=None,
    input_shape=(32, 32, 3),
    pooling=None,
    classes=10,
    classifier_activation="relu",
  )

  model2.compile(optimizer='adam',
                loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                metrics=['accuracy'])

  history = model2.fit(
            train_images, train_labels,epochs=10,
            validation_data=(test_images, test_labels)
            )

I'm trying to train VGG16 from scratch, hence not importing their weights I also tried a model which I created myself, with same hyperparameters, and that worked fine
Any help is highly appreciated
Heres the full code

Ok, I solved this problem
The simple thing was that learning rate was too big
I changed the code to this
LR = batch_size/((z+1)*100000)
LR=LR/3

instead of
LR = batch_size/((z+1)*1000)
LR=LR/3

and it seems to work well

The question still remains that why does it show decent accuracy in case of training and not in validation when the learning rate is too high


text:
What is the advantage of using MCTS with value based methods over value based methods only?
I have been trying to understand why MCTS is very important to the performance of RL agents, and the best description I found was from the paper Bootstrapping from Game Tree Search stating:

Deterministic, two-player games such as chess provide an ideal
test-bed for search bootstrapping. The intricate tactics require a
significant level of search to provide an accurate position
evaluation; learning without search has produced little success in
these domains.

I however don't understand why this is the case, and why value based methods are unable to achieve similar performance.
So my question would be:

What are the main advantages of incorporating search based algorithms with value based methods?

In short and simplified terms, MCTS is like being able to look into the future based on experience and choosing the best action :-)

@DavidIreland yes, but value based methods approximate the expected return of the future as well, so why would looking at the future while executing be more accurate than our value estimate?

we can only estimate our values using function approximation, so our estimates will never be true (because we have far more states than weights). If we can look at the (approximated) value of states we take in, say, 5 actions time, it is better to make a decision based on these estimations, taking into account the true rewards observed after the 5 actions. Further, MCTS also allows more implicit exploration as when choosing the actions to expand the tree we are potentially choosing lots of non-greedy actions that lead to better future returns.

@DavidIreland That comment looks like the basis of a good answer already

Assuming a continuous/uncountable state space, we can only estimate our value function using function approximation, so our estimates will never be true for all states simultaneously (because, loosely speaking, we have far more states than weights). If we can look at the (approximated) value of states we take in, say, 5 actions time, it is better to make a decision based on these estimations, taking into account the true rewards observed after the 5 actions.
Further, MCTS also allows more implicit exploration as when choosing the actions to expand the tree we are potentially choosing lots of non-greedy actions that lead to better future returns.

I would note that technically, "because we have far more states than weights" is not the full story as to why our estimates may not be 100% correct. We could probably come up with some weird MDP where there are many different states, each represented by a single variable, and even linear function approximator with just a single weight is sufficient to perfectly represent the optimal value function. In general, in "real-world" / "interesting" problems, it's roughly true-ish though :)

@DennisSoemers I’ve added a disclaimer for you. :)

@DavidIreland I think your point is that using value based methods only is like having perfect confidence in what would happen in the future, while MCTS simulates the future, so it should have better approximation of the value, am I right?

Yes, that sounds correct @Hossam. MCTS essentially allows the agent to search(/plan) and make decisions using real transitions that are simulated in the environment to better help with the decision making.


text:
Do RNNs/LSTMs really need to be sequential?
There are many articles comparing RNNs/LSTMs and the Attention mechanism. One of the disadvantages of RNNs that is often mentioned is that while Attention can be computed in parallel, RNNs are highly sequential. That is, the computation of the next tokens depends on the result of previous tokens, thus, RNNs are losing to Attention in terms of speed.
Even though I fully agree that RNNs are sequential as stated above, I think they are still parallelizable by splitting the mini-batch into sub-batches and each of these sub-batches is processed independently by a dedicated thread. For example, a training batch of size 32 can be split into 4 sub-batches of size 8; 4 threads process 4 sub-batches independently. That way, RNNs/LSTMs are parallelizable and this is not a disadvantage compared to Attention.
Is my thought correct?

You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
Can RNNs/LSTMs do that? No.
Can Attention (i.e. Transformers) do it? Yes.
The reason is simple Attention is effectively an affinity matrix between each and every input that goes into a network. So, it is able to do that. We have a huge memory overload but hey, we want the performance.
In case of RNNs/LSTMs, the cells have to do this heavy-lifting, there is only a set amount of information that can be contained in them. That's why you have to forget gate to control information retained.
Nevertheless, your thought is correct but that's not the reason for Attention to be in vogue. But, your thought has negative ramifications when we see how to implement it. Also, nevertheless the computation will be still sequential since you can't process input (n + 1) without input n. Local parallelization is possible but not global.

Thank you, Abhishek, for your answer! Regarding the implementation of parallelization, could you specify in a bit more detail why it would be more complex to do so in batch-dimension (in the case of RNNs/LSTMs) compared to how parallelization is done in the case of Attention?

https://leimao.github.io/blog/Data-Parallelism-vs-Model-Paralelism/

Hi Abhishek, thanks for the link but I cannot fully understand your point. I don't know if Model parallelism is often used for Attention or not, but data parallelism seems simpler than model parallelism to me. On huggingface pre-trained models, they also make use of data parallelism and support sending sub-batches to each GPU/CPU device.

That's exactly the point. What you were talking about is model parallelism which is hard because it is difficult to consolidate back the weights calculated on different workers.

hmm, please correct me if I am wrong, I think I was talking about using RNNs/LSTMs with data parallelism, not model parallelism, isn't it?

In data parallelism, the model is replicated. You are not talking about model replication. You are saying you want to break the model computations, that is model parallelism. Breaking model into various workers.


text:
What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?
I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. What does 'clock rate' mean in this context?

The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with "John went to..." and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model.
As mentioned in the paper, the most common method to combat this (at the time) was using an LSTM that stored long term data in it's cell state, or as put in the paper:

[an LSTM] uses a specialized architecture that allows information to be stored in a linear unit called a constant error carousel (CEC) indefinitely

However, this requires a whole heap of extra parameters in order to work (input gate, output gate and forget gate which all require parameters). So proposed was the CW-RNN.
The fundamental idea behind a clockwork RNN is to have "modules" computed periodically. First to clear things up, a timestep is 1 input into the model. So in the case of our example, if we're inputting character by character, timestep 1 is the character "J", timestep 2 is "h" and so on; "o", "n", " ", "w"...
So what is the clock-rate? Well, it's simply how often each module of the CW-RNN is computed. Let's say the hidden layer of the clockwork RNN is split into 8 modules, which I will reference as $M_1, M_2, M_3 ... M_8$, and the associated clock-rates for each of these modules are the powers of 2, so: $1, 2, 4, 8, 16, 32, 64$ and $128$.
Before I continue I want to note a potential discrepancy, if you consider the first input to be timestep 0, then it changes how this executed (all modules would be activated at the first timestep), however if you consider it to be 1 (like I do for this example), only module 1 would be executed (again, in this example).
So we're at timestep 1, ie "J", so we check the timestep against the clock-rates of each module to determine which ones will be executed in the computation of the hidden state and output for this timestep. To do this, we take the mod of the timestep against the time-rate, so: 1 mod 1, 1 mod 2, 1 mod 4 ... 1 mod 128 and if it equals 0 (basically, is the timestep a multiple of the time-rate) then that module will be executed at this timestep. So in this case $M_1$ will be executed. When we input "h" at timestep 2, $M_1$ and $M_2$ will both be executed, and will equally contribute to the hidden state (ie, they will be added together) and the output.
By performing calculations this way, each module can all simultaneously be responsible for information over different time periods, for example $M_8$ which is only executed every 128 timesteps will be responsible for very long term dependencies, but $M_1$ will cover short term dependencies.
So in basic terms, clock-rate refers to how often (what timestep interval) a given module of a Clockwork RNN is computed

Thanks for the answer. So does a CW-RNN actually produce superior results to the LSTM (for this purpose of 'long term memory')?

According to the paper, yes it does "CW-RNNs beat both RNN and LSTM networks by a considerable margin of 8-20% on average" for classification and " CW-RNNs give by far the best results, with the
smallest one being roughly on par with the second-biggest
LSTM network" for sequence generation

It seems this paper defines a "clock period ${T}_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T_i$ is smaller than period $T_j$.".
This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a "clock period" (or "clock rate") on other, more self-similar, RNNs. However "time t" is something that is defined to express the amount of times the RNN has recursed, "t times".

Thanks for the answer. So what exactly does "clock rate" mean in the context of the linked paper?

@ThePointer "clock rate" refers on how modules process data. The idea of the paper is to compute information at different "rates", for this, it uses modules, the modules are connected to others according to the module's clock period.  This creates a network where information is processed by each module at different rates. Quote: "Each module $i$ is evaluated only every $T_i$-th time step", this would be a 'rate' of computation of incoming data.


text:
How is a ResNet-50 used for deep feature extraction?
I'm trying to implement the vehicle re-identification model described in https://arxiv.org/pdf/2004.06271.pdf.
My question focuses on Section 3.2 of the paper, which uses a ResNet-50 for deep feature extraction in order to generate discriminative features which can be used to compare images of vehicles by Euclidean distance for re-identification. It takes a 256x256x3 image as input.
My understanding of ResNet-50 is that its output is of the shape N, where N is the number of classes which an input image could be, and ground truth labels take the form of a one-hot encoding where the '1' value represents the node in the output layer which is associated with the given class.
I am therefore confused by the usage of ResNet-50 in a re-identification task in which the goal is to generate an array of discriminative features which can be compared by Euclidean distance. There is no discrete set of N classes, as the model should work on any of the infinite number of vehicles in the world.
What is the ground truth label in a ResNet-50 in the context of a re-identification task?

The authors use so-called embeddings, it's a form to represent the images in some meaningful vector form.
The procedure to get embedding as follows. First, keep in mind most of the popular convolutional net architectures starts with convolutional layers and then have few fully connected layers. Then do the following.

Train the full network with one-hot encoded labels as usual
Take away the last fully connected layer and use the values on the previous layer as a representation.

In the case of resnet50, you will get a 2048-value float vector. The property of a neural network is that semantically close images usually have close representation on the last layers and you could use euclidian distance to measure the similarity of images in some sense (not really, but it's another long discussion)
I don't know the paper, but I glimpsed through it. You could see in formula 4 the $x$ is embedding representation. Loss at formula 5 is cross-entropy after applying the last linear layer ($Wx + b$) and then softmax.
As a side note, you could skip the first step completely and use pretrained weights as a starting point. I.e. in pytorch you could take pretrained on ImageNet classification weights as follows
import torchvision.models as models
resnet = models.resnet50(pretrained=True)


text:
Expected behavior of adversarial attacks on deep NN?
I am trying adversarial attack (AA) for a simple CNNs. Instead of the clean image, my simple CNN is trained with attacked images as suggested by some papers. As the training goes on, I am not sure if the training is well going or something is wrong.
Here is what I observed:
When the epsilon value is large, the classification performance of the model from the adversarial training is low. I understand if the attacked image is given to the model, then the performance is poor. Although the model is from the adversarial training, because the epsilon is large, the model is poorly perform. However, when an clean image is given, the performance of the model is still low. Performance on the clean images are higher than the performance of the attacked images, but not as high as the baseline model without adversarial training.
So, I wonder if the adversarial training also degrades the performance of the model on the clean images. When I read papers, I only see the results on the adversarial Images, not clean images. If you have any experience, it will be very helpful to check if my training code is working well or not.
When the epsilon is very large, the accuracy of the model on clean image is around 15%. The model without the adversarial training is around 81%.
Some details.
I use PGD attack with 5-iterations and epsilon is one of eps = [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.03, 0.05, 0.07]. Step size is eps/3. Only one epsilon is selected and the adversarial training is conducted. So there are 8 different models trained with different epsilons.  I use natural image Dataset.

This seems to be a known problem, and intuitively seems reasonable. You might be interested in the paper Adversarial Training Can Hurt Generalization.
The authors suggest that this might be because training on the perturbed data requires the model to learn more robust features, which means more samples are required to obtain performance comparable to a model that is not adversarially trained.
You could try collecting or generating additional samples to see if this leads to an improvement. They also mention that in their experiments on the MNIST dataset, using Xavier initialisation led to a significant benefit, so you could experiment with that too.


text:
Object Detection: Can I modify this script to support larger images (Scaled YOLOv4)?
I am looking at training the Scaled YOLOv4 on TensorFlow 2.x, as can be found at this link. I plan to collect the imagery, annotate the objects within the image in VOC format, and then use these images/annotations to train the large-scale model. If you look at the multi-scale training commands, they are as follows:
python train.py --use-pretrain True --model-type p5 --dataset-type voc --dataset dataset/pothole_voc --num-classes 1 --class-names pothole.names --voc-train-set dataset_1,train --voc-val-set dataset_1,val  --epochs 200 --batch-size 4 --multi-scale 320,352,384,416,448,480,512 --augment ssd_random_crop

As we know that Scaled YOLOv4 (and any YOLO algorithm at that) likes image dimensions divisible by 32, I have plans to use larger images of 1024x1024. Is it possible to modify the --multi-scale commands to include larger dimensions such as 1024, and have the algorithm run successfully?
Here is what it would look like when modified:
--multi-scale 320,352,384,416,448,480,512,544,576,608,640,672,704,736,768,800,832,864,896,928,960,992,1024

Yes, the functionality should is there. But, don't you think you are overdoing the scales. You have at least 18 scales mentioned here. Too much of anything is bad. There is a reason it likes things divisible by 32 because at that increase in size something more meaningful will show up in the image. Spamming sizes like this won't help you at all, it would rather waste your time.

thank you, that makes sense. So should I do something like start at 1024 and list 6 or 7 sizes down from it; would that work?

I think you can start with 128, 256, 512, 1024.


text:
Extracting Descriptors and feature points for 3d mesh
I'm programming my work with python, and I have a mesh and I want to extract 3d descriptors and feature points from it( trying to work on multi-scale strategy) , to visualize them later on the mesh,
What I'm asking about, is references, guidelines, anything which could benefit me with this situation
The main work I'm trying to do, is to reach matching stage, where I could find one-to-one correspondence.

The work was done through the following:

Extract feature points using a Detector
Extract the descriptor for these feature points
Make matching using similarity measure between two descriptors from two meshes.


text:
How to explain that a same DNN model have radically different behaviours with each new initialization and training?
I'm trying to predict the continuous values of a variable $y$ using a Fully Connected Neural Network while providing it with data from a $(3300, 13)$ matrix $X$ where $X[i, :]=[0,...,1,...,0,x_{i}]$. So the first $12$ elements of a data vector are all zeros except for one element which is equal to $1$ to denote  the belonging of this data to a category. I'd like to add that my $X$ data is normalized with regard to the $13$-th column and that both $X$ and $y$ are shuffled in the same manner. Please find below my code for my model:
model = Sequential()
model.add(Input(shape=(13,)))
model.add(Dense(6, activation = 'relu'))
model.add(Dense(2, activation = 'relu'))
model.add(Dense(1))

model.compile(loss = 'mean_squared_error',
              optimizer = 'adam',
              metrics = ['RootMeanSquaredError'])

history = model.fit(X, y, validation_split = 0.1, epochs=64)

When trying to plot the learning curve using:
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('rmse')
plt.xlabel('epoch')
plt.legend(['train', 'val'], loc='upper left')
plt.show()

I get these curves:

There's already an "unusual" element to point here; I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner but we don't notice that on the training learning curve.  For example the last four values of the loss are: $2.5176$, $3.4718$, $3.0704$ and it settles down on $3.8177$. I've also noticed that the losses provided by history.history are different than those shown during training, I suspect ones are computed before the epoch and ones after but I'm not sure.
I've tried to predict on the $275$ first elements of the training data. Most of the predictions took the value $4.2138872e+00$ but there are other predictions who took lesser values. I've computed the maximum of the predictions on the whole training set and it is $4.2138872e+00$.
I've also tried to train on the whole training set without a validation set to see what'll happen. I've made sure to rerun the cells of the model so that it doesn't take the weights it already found. I've noticed the same behaviour for the loss during training, but this time there is no constant predicted value that comes up as a maximum limit for the predictions.
I've already asked this question here and a user suggested to me that I should ask this question separately while providing the whole code. I ran the same code I was running and that was giving me the same predictions for no matter for my input vectors.
I think, as the user @Kostya that answered my previous question pointed out, what's happening here is called "dying ReLus". It's the same code that I'm running over and over but gives different predictions and the only random parameters are the weights and the biases. I'm sure the biases are initially initialized to zero but I don't know how the weights are handled. I suppose they're randomly generated by a centered and reduced normal distribution.
I have last came to this question: does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ? I came to think that because if we had a large number of weights, their values are likely to fill that interval where the majority of the probability mass is concentrated. And since we have a small number of weights, we can get some "outlier" weights which lead to dyind ReLus.

I'm sure the biases are initially initialized to zero but I don't know how the weights are handled.

Looking at the Dense layer docs: by default Dense layers biases are initialized with zeros (bias_initializer='zeros') and weights are initialized with Glorot uniform (kernel_initializer='glorot_uniform').

... "unusual" element to point here;  I've noticed that throughout the training the loss decreases but sometimes in an oscillating manner.

There's nothing unusual about the oscillations. Quite on the contrary - your curves are suspiciously too smooth.

I've also noticed that the losses provided by history.history are different

Yes, so this is a little gotcha in the keras implementation. For training loss, keras does a running average over the batches throughout an epoch, while for the validation loss it computes it after the epoch finishes. (link)
That also explains why your validation loss starts slightly lower than the training one - it actually lags behind the training loss by $\sim1/2$  of an epoch.
The fact that both losses stay almost equal suggests that your model don't really rely on training data for prediction - it got saturated ("dying ReLus") almost instantly.
Couple of suggestions that I can make:

I don't see you setting up a learning rate. Try making it smaller (like, much smaller). And see if there's any difference.

MSE is a nasty loss - especially at large values. Have you tried standardizing the target values?

Does the number of neurons, hence the number of weights influence the phenomena of "dying ReLus" ?

Yes, the less neurons you have the higher the chance that all of the neurons will die out. You can get some intuition about it on https://playground.tensorflow.org/ - following the link, I've got a relu network that trains reasonably well. Try increasing the learning rate and observe saturation in the neurons. Reduce the number of neurons and see how the whole net gets stuck.

thank you for your answer and sorry for my late reply. You're right what's suspicious is the smoothness of the curve, normally we must notice the oscillations we notice in the logs. I've tried both making the learning rate smaller ($0.0001$) and standardizing the target values and it seems it's working quite well this time. Both training and validation losses decrease over the epochs and when they stabilize there is some difference between the two values (about $0.1$).

Can you tell me please how did you came up with the solutions ? I don't understand how decreasing the learning rate can help against the "dying ReLus". A big learning rate can make us miss the minima of the loss function and it can shift the weights in a direction that can cause our ReLus to die but it can also "drive them away" from that direction am I wrong ? And I don't see how MSE helps us avoid that issue as well. Thank you in advance.

@Daviiid So, I think the lesson for you is that you always have to do hyperparameter tuning. That's something that is usually not stressed enough in guides and tutorials.

@Daviid Just playing around with various parameters on the playground site will give you some intuition on what various parameters do. One advise is - don't be afraid of adding more layers and neurons.

@Daviiid It is hard to give extended answers in the comments (that's not how this site is supposed to work anyway). Some of your questions merit separate posts. I'd also suggest that you formulate you questions by talking about your particular predicament, but in more general context. "How decreasing the learning rate can help against the dying ReLus" is a good start.

I see, I'll be sure to remember that advice everytime.Thank you again a lot for all the help and explanations !


text:
Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent
new to RL here.
As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and Control, which is maximizing the reward or the value function. what if i have a heuristic agent that performs almost acceptable performance in an environment but i want to find a policy that tends to be the optimal policy, is there a way to cut the first half of the task by teaching the agent ? will be a side by side buffer of the (states, actions) be sufficient ?

Not sure I fully understand your question, are you asking:

if you can skip the policy evaluation part?
if you can speedup the training of your agent by mimicking the behaviour of your heuristic agent first and then optimise from there?

If the first, I am not sure exactly what you are asking as most RL algorithm doesn't have an explicit policy evaluation phase and even for ones which does (policy iteration for example) you would be using the wrong policy to evaluate.
If you are asking about the second one (warm start your agent) that's very much possible. You would do it differently based on the algorithm:

With DQN, you have your buffer which contains (state, action, next_state, reward) tuples. You could just let loose your heuristic agent in your environment and record these tuples and put them into the buffer. Then you could train your DQN agent offline (without interacting with the environment) for a while until it performs roughly as well as your heuristic one and then you could let it loose in your environment so it can get better.
With something like an Actor Critic you might not have a buffer so the process is slightly different. You also have two components, the policy function and the value function. You can train the policy function to mimic what your heuristic agent does by gathering lot of experience from the heuristic agent and basically do supervised learning on that experience. You would also need to train your value function but that would be very similar to it's normal training.

Hope this helps, let me know if anything is unclear.

Thanks Kristof, this is so clear. the buffer concept is exactly what I need.

You're welcome. Please accept my answer :)

Done, yet typical of stackexchange users to give the question -ve rating xD


text:
Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?
While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the

"The board is oriented to the perspective of the current player."

I was wondering why this is the case if there are two agents (black and white). Is it because there is only one central DCNN network used for board and move evaluation (i.e. there aren't two separate networks/policies used for the respective players - black and white) in the algorithm AlphaZero uses to generate moves?
If I were to implement a black move policy and a white move policy for the respective agents in my environment, would reflecting the board to match the perspective of the current player be necessary since theoretically the black agent should learn black's perspective of moves while the white agent should learn white's perspective of moves?

There is a single neural network that guides self-plays in the Monte Carlo Tree Search algorithm. The neural network gets the current state of the board $s$ as an input and outputs current policy $\pi(a|s)$ and value $v(s)$.
The action probabilities are encoded in a (8,8,73) tensor. First two dimensions encode the coordinates of the figure to "pick" from the board. The third dimension encode where to move this figure: check out this question for a discussion on how all possible moves are encoded in a 73 dimensional vector.
Similarly, the inputs of the network are organized in the (8, 8, 14 * 8 + 7 = 119) tensor. The first two 8 x 8 dimensions, again, encode the positions on the board. Then the positions of the figures one plane per 6 figure types: first 6 planes for player's figures, next 6 planes for opponent's figures and two repetition planes. The 14 planes are repeated 8 times supplying predecessor positions to the network. Finally, there are 7 extra planes encoding as a single uniform value over the board - castling rights (4 planes), total move count (2 planes) and the current player color (1 plane).
Note that the positions of player's figures and opponent's figures are encoded in fixed layers of the state tensor. If you don't flip the board to the perspective of the player then the network will have very different training inputs for black and white states. It also will have to figure out which direction the pawns can move depending on the current player color. None of that it is impossible, of course - but that unnecessarily complicates something that is already a very hard problem for the DNN to learn.
You can go further and completely split the training for white and black players, as you've described. But that'll essentially double the work you'll have to do train your nets (and, I suspect, there would be some stability troubles typical for adversarial training).
To summarize - you are generally right - there is no fundamental need to flip the board. All the above details in state encoding are done to simplify the learning task for the deep neural network.

Thanks for your explanation! (And I was the person who asked the question you linked haha, thanks again for all your help.)

I am not an expert in RL. I have been playing Go for some years.
Let's quote from AlphaZero's paper first:

Aside from
komi, the rules of Go are also invariant to colour transposition; this knowledge is
exploited by representing the board from the perspective of the current player (see
Neural network architecture).

In the Game of Go, the difference between Black and White except the board representation is the komi (the amount of points that Black has to compensate White in the final count for playing first). Except the presence of komi, there should be no difference in strategy under the same position if colours exchanged. In other words, given a state $s$ of black stones and white stones on the board, if the optimal policy of Black playing first is $\pi$, then if colours of stones on the board exchanged and it is White's turn, the optimal policy for White should be the same as $\pi$.
With this in consideration, there are at least 2 advantages of using a network that represents the board in the perspective of Self/Opponent rather than Black/White.
The first is that it prevents the network from the possibility of giving inconsistent strategies under two representations of the same state. Consider a network $f_\theta$ that accepts the board representation in the order of $(B,W)$, and a state $s = (X_t,Y_t)$ in which $X_t$ is a feature map for black stones and $Y_t$ is a feature map for white stones and it is black's turn. Now consider a state $s' = (Y_t,X_t)$ (i.e. colours flipped) and it is white's turn. $s$ and $s'$ are essentially representation of the same state (except Komi which does not affect optimal policy). There could be a possibility that the network $f_\theta$ gives different policies for these two representations. However, if $f_\theta$ accepts the state as $(Self,Opponent)$, the input to the network would be the same (except the komi feature).
Therefore, this representation would significantly reduce number of states represented by the features vector $(X_t,Y_t)$, which would be the second advantage to training the neural network. If we consider that in Go, the same local position could appear in exchanged colour in another position, the network could by this implementation, recognize them as the same position. A decrease in the number of states could mean a significant drop in parameters and power needed from the network.
The same principle of making use of different representations of the same state is followed in AlphaGo's other training implementations as well, such as augmenting its training data to include rotations and reflections of the same board position.
However, in the game of Chess, this would be a different case. For a chess position, if the pieces' colours are exchanged and it becomes opponent's turn, it would be a different state because the positions of the KING and the QUEEN are not the same for the two colours.

Thank you! I was just curious as to whether it was necessary or not, but now I realize the problems that are fixed when you flip the board.

In board games, whose turn is it play makes a significant difference in the outcome of the game. Many times its a difference of loss or a win.
So this information has to be communicated to the neural network.
There are two ways to communicate that information:

To add extra input nodes which say whose turn it is to make move.

To simply flip the signs of the pieces.

Method 1 works but it increases the number of nodes hence making the neural network more complex and larger. So method 2 is preferred to keep the neural network simple without losing the needed information. This process is known as canonical form of the board.
Flipping signs wouldn't work for games that have more than two players.

Note that the AlphaZero paper has both of these: the board is flipped to current player's orientation AND the current color is passed in.


text:
What does the "number of channels" correspond to in U-Net?
I'm studying the U-Net CNN architecture. I'm new to CNNs and am confused regarding the "number of channels".
Referring to the U-Net diagram, the input image is convolved with a 3x3 mask which generates a 570x570 output. This output image is then convolved again by a 3x3 mask to produce a 568 x 568 signal. However, what do the 64's correspond to?
The U-net says something about a multi-channel feature map. But how does convolving an image by a 3x3 mask results in a "64".

This related question might help you get a better understanding of the structure and function of (multichannel) CNNs.

@legammler So how I'm currently interpreting the U-net diagram. The architecture declares a set of 64 convolution masks that we're attempting to tune. Thus, convolving a single gray scale image with 64 masks would produce 64 new signals. However, the second 64 confuses me. If we were to repeat this operation, why do we still have 64 channels? Wouldn't we get 64 x 64 channels?

In this example you have a gray scale image of size 572x572 and 1 (gray) channel. The first convolution operation consists of 64 filters of size 3x3 and 1 channel per filter. The channel of the filters always fits the channel size of the previous layer (here: the Input).
In the second convolution step of this explicit architecture, you again use 64 filters of size 3x3. In this case, each of these filters consists of 64 channels according to the previous output (64 feature maps/channels). The output of the second convolution consists of 64 feature maps according to the amount of 64 filters in the second convolution. This video from Andrew Ng visualizes it perfectly.

I think I understand. The first convolution features 64, 3 x 3 x 1 filters to generate an output that is 570 x 570 x 64. The second convolution features 64, 3 x 3 x 64 filters resulting in a 568 x 568 x 64 output. I understand the arithmetic now, it definitely was not obvious while reading.


text:
PPO agent for vehicle control does not learn to stop at traffic lights
I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from stable_baselines3 without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.
I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again un-learns stopping at red lights.
I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?

The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.
It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.

I agree. If the algorithm works, then its failure in a different environment is probably because of complexity or rewards.

Thanks, I tried normalizing my observations between -1 and 1 and this caused a major improvement. I might also try this for the rewards and actions next.


text:
How to properly use Flatten layer?
Context
I'm trying to create net that will be able to recognize printed-like digits. Something like MNIST, but only for standard printing font.
Images are of the size 40x40 and I'd like to put them into feedforward net since ConvNet seems too powerful for this task.
Question
How should I use Flatten layer in this task?
Code
My current net:
X, test_X, y, test_y = train_test_split(X, y, test_size=0.25, random_state=42)

self.model = Sequential()
self.model.add(Flatten())
self.model.add(Dense(64, activation='relu', input_shape=X.shape[1:]))
self.model.add(Dense(no_classes, activation='softmax'))
self.model.compile(loss="categorical_crossentropy",
                   optimizer="rmsprop",
                   metrics=['accuracy'])

self.history = self.model.fit(X, y, batch_size=256, epochs=20, validation_data=(test_X, test_y))
print(self.model.summary())

Example images

Current results

Does this help? https://stackoverflow.com/q/44176982/9506872

The Flatten layer is used for collapsing an ND tensor into a 1D tensor. In your case, the inputs appear to be $28\times28$ images, so Flatten will convert that into a tensor with shape $1\times768$. Note that no information is lost. Flatten layers are usually used where you have a convolutional layer with dimensions $N\times M \times C$ (where $N$,$M$ are the feature map sizes and $C$ is the number of channels) and want to fully connect with a Dense layer or another layer that only accepts 1D inputs. Flatten can also be used when the network is meant to output a feature vector from a final convolutional layer for image classification purposes using a different technique.

Do You have an idea how I should choose net architecture to get decent results? I'm trying with 1 layer of size up to 1024 and two layers with different sizes, but not larger than 1024 and all the time I get results like on the plots. Pics are 40x40.


text:
Is there a clear distinction between Artificial Intelligence and running a sequential program?
Artificial Intelligence (AI) is often defined as a machine that is intelligent, or one that can think rationally.
From a high-level perspective, things like self-driving car or Alpha-Go can easily be classified as an AI system, while things like a washing machine that follows a strict sequential program is not considered as AI.
However, what confused me is that when looking at the definition from a low-level perspective, there does not seem to be a clear distinction between AI and non-AI.
For example, consider an Artificial Neural Network from Deep Learning. Fundamentally, it is just a complex non-linear function. Why is this considered AI while a washing machine is not considered that?
Is it because of the learning involved? But then path-finding will not considered as AI too.
Is it because of the calculations? But then traditional calculators will be considered as AI.
Is there even a clear distinction between AI and a sequential program? Or is it just a vague term that is only valid when viewed from a high-level perspective?

That question doesn't really make sense. AI is a field of study, and a sequential program is a list of instructions.

What I am asking is how can I draw the line between AI and non-AI? Is there a definition that can distinctly separate these two?

There is no clear-cut line, partly because there is no clear standard definition of AI. For example, many people would call a washing machine with a fuzzy logic chip AI; but is it really?

No. As of now, there is no clear distinction between AI and a sequential program. The biggest reason is that there is no officially agreed definition of AI. Since AI researchers themselves hold different opinions of AI, the field itself is constantly redefined regularly, there is no clear-cut distinction between AI and non-AI.

Our prof used to say that AI is cool things that computers cannot do.

Whether or not the academic community can agree on a formal basic definition, AI will continue to function!  (I wouldn't be surprised if AI is fully able to define humanity before scholars can agree on something so simple as "an artefact that makes a decision".


text:
What language is the GPT-3 engine written in?
I know that the API is python based, but what's the gpt-3 engine written in mostly? C? C++? I'm having some trouble finding this info.

The GPT-3 source code hasn’t been released but the creators say it uses the “same model and architecture as GPT-2” (source) with some exceptions.
The GPT-2 source code is written in 100% Python. The model is based on Tensorflow and NumPy which are written using C and C++. My best guess is that GPT-3 is also written in Python using libraries based on C.


text:
Data augmentation for very small image datasets
I am looking for techniques for augmenting very small image datasets. I have a classification problem with 3 classes. Each class consists of 20 different shapes. The shapes are similar between the classes, but the task is to identify which class the shapes belong to. Per shape, I have between 1 and 35 training examples. For two classes, I have 25 training examples per shape, but the number of examples per shape for the third classes is usually around 5. Now, what data augmentation schemes do you recommend? Geometric / affine transformations seem like a good place to start. However, I have also thought of applying Fast Fourier Transform (do the forward transform, add some noise, do the inverse transform). GANs seem infeasible, right? Not enough data, I suspect. In any case, I am grateful for your advice.

In my opinions, you have around $25\times 20 \times 2 + 5 \times 20 = 1100$ samples, so the list of problems is:

Lack of data
Imbalance between class 1,2 and class 3

With the simple task, the model with the low capacity (small number of parameters) is more suitable for your task and to avoid overfitting because of the lack of data. Since the augmentation is nothing but regularization, based on EfficientNetV2 paper, with the small model you don't need too much regularization, select some basic affine transformation and elastic transformation then apply Rand Augment with small magnitude mix with 0.1 dropout should be enough.
The second problem should be solved with the weight class for weight, which means set the scale factor for the third class, or duplicated the samples of the third class then apply the strong affine transformation to it.


text:
Reward interpolation between MDPs. Will an optimal policy on both ends stay optimal inside the interval?
Say I've got two Markov Decision Processes (MDPs):
$$\mathcal{M_0} = (\mathcal{S}, \mathcal{A}, P, R_0),\quad\text{and}\quad\mathcal{M}_1 = (\mathcal{S}, \mathcal{A}, P, R_1)$$
Both have the same set of states and actions, and the transition probabilities are also the same. The only difference is in the reward functions $R_0$ and $R_1$. Suppose that we've found an optimal deterministic policy $\pi^*_0$ for the problem $\mathcal{M}_0$ and we've checked that this policy is also optimal for $\mathcal{M}_1$
$$\pi_0^*(s) = \arg\max\limits_a Q^*_0(s,a)\qquad  Q_1^*(s,\pi_0^*(s)) = \max\limits_a Q^*_1(s,a)$$
Now, given the two MDPs one can build a whole family of MDPs interpolating between them:
$$\mathcal{M}_\alpha = (\mathcal{S}, \mathcal{A}, P, \alpha R_0 + (1-\alpha) R_1)$$
Where $\alpha\in[0,1]$ is the interpolation parameter between the two problems - the rewards are linearly changing from $R_0$ to $R_1$ with this parameter. My question is - in general. will $\pi_0^*$ be optimal for all MDPs in the middle of interpolation interval?
$$Q_\alpha(s,\pi_0^*(s))\stackrel{?}{=}\max\limits_aQ^*_\alpha(s,a),\; \forall\alpha\in[0,1]$$
I feel like this could be generally true due to linearity of the dependence and convexity of the optimization problem. But I cannot neither prove it, nor find a counterexample.

You may want to have a look at this paper if you haven't yet.

I believe the claim is true. Here is my attempt at a proof.
Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$.
The value $V_\alpha^*(s)$ is the expected sum of discounted rewards under an optimal policy $\pi_\alpha^*$, i.e.,
\begin{equation}
      V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[\sum\limits_{t=0}^{\infty}\gamma^t\left( \alpha R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)R_1(s_t, \pi_\alpha^*(s_t)) \right)\middle| s_0 = s, \right],
\end{equation}
with the expectation taken with respect to the steady state distribution $\rho_\alpha$ of states under $\pi_\alpha^*$.
In the following, I drop the condition $s_0=s$ for conciseness, but you can assume it's in each expectation.
Now, break up the sum:
\begin{equation}
     V_\alpha^*(s) = \mathbb{E}_{\rho_\alpha}\left[ \alpha\sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) + (1-\alpha)\sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Then, by linearity of expectation:
\begin{equation}
      V_\alpha^*(s) = \alpha\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_\alpha^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_\alpha}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_\alpha^*(s_t)) \right].
\end{equation}
Note that the first expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_0$, and the second expectation term is the value of $\pi_\alpha^*$ in $\mathcal{M}_1$.
We already know that $\pi_0^*(s)$ is optimal in $\mathcal{M}_0$ with reward function $R_0$, and $\pi_1^*(s)$ is likewise optimal in $\mathcal{M}_1$ with $R_1$. Further, as per your assumption, $\pi_0^*(s) = \pi_1^*(s)$.
So $\pi_\alpha^*$ can be at most as good as $\pi_0^*$ with reward function $R_0$ (resp., with $R_1$):
\begin{equation}
      V_\alpha^*(s) \leq \alpha\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_0(s_t,\pi_0^*(s_t)) \right] + (1-\alpha)\mathbb{E}_{\rho_0}\left[ \sum\limits_{t=0}^{\infty}\gamma^t R_1(s_t,\pi_0^*(s_t)) \right].
\end{equation}
Note that we know take the expectation under the steady state distribution $\rho_0$ of $\pi_0^*$ instead.
Thus, we have shown that $V_\alpha^*(s) \leq \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
Now it remains to argue that the case with a strict less than relation is not possible.
Suppose this were the case, and we would have $V_\alpha^*(s) < \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$.
But then $\pi_0^*$ would attain a higher value than $\pi_\alpha^*$ in $\mathcal{M}_\alpha$, which is a contradiction (because we assumed that $\pi_\alpha^*$ is an optimal policy for $\mathcal{M}_\alpha$).
Thus, $V_\alpha^*(s) = \alpha V_0^*(s) + (1-\alpha)V_1^*(s)$ and furthermore, acting according to $\pi_0^*$ is optimal also in $\mathcal{M}_\alpha$.

Essentially you've proven $\forall \pi. V_\alpha^\pi = \alpha V_0^\pi + ( 1 - \alpha )V_1^\pi$ and then $\forall \pi. V_\alpha^\pi \leq \alpha V_0^{\pi_0^} + ( 1 - \alpha )V_1^{\pi_0^} = V_\alpha^{\pi_0^*}$. This makes sense. Thank you for looking into it.

@Kostya That's a nice way to phrase it, the decomposition of the value function holds for any policy. I am curious how you came about asking this question - I could not think of any practical examples where one might have the property you start with. Any comment?

Imagine you need to be able to quickly solve MDPs that differ only by $R$. E.g. a user set up his $R$ and wants the optimal policy. The statement shows the convexity of this kind of task: inputs inside the convex hull of previous equal solutions evaluated instantly to the same solution.


text:
What approach would work well for predicting earthquake intensity based on historical data?
My problem: I own warning system where I collect data from institutions and send them over through various ways to users. I would like to hear your advice on what approach I can use for solving my problem with earthquake intensity far from epicenter. Since seismogical institutions mostly issue info about intenstiy of an earthquake for the epicenter, I would need to predict and classify what intensity the earthquake can have for places distant of several km/miles from the epicenter.
As an input/training set, I can use data of historical earthquakes and their magnitudes in an epicenter. Then I would need to fill mostly "by hand" an information about intensity based on seismological records, historical testimonies, chronicles atc.
What I need from AI: I need "something" that would predict earthquake intensity based on dataset of historical earthquakes.
Example/TLDR: There is an earthquake with magnitude 3.8, distant 80 km with depth 6 km. Based on dataset of historical earthquakes (with same type of information + witnessed and collected intensity), and output, I would need prediction of intensity of an eartquake 80 km from the epicenter.

You can do prediction like this with regression. Take a look at scikit-learn models for regression: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model
Here's an example:
# This is made up data!
#        epicenter magnitude, depth_m, distance_km, local_magnitude
data = [[3.5,                 6000,    61.3,        2.1],
        [9.1,                 16000,   261.0,       6.0],
        [9.1,                 16000,   96.5,        8.2],
        [3.8,                 6000,    80.0,        2.6],
        [2.5,                 4000,    101.3,       0.4],
        [7.0,                 9000,    81.9,        6.1],
        [5.1,                 7000,    21.3,        4.1]]

from sklearn.linear_model import LinearRegression
import numpy as np

data = np.asarray(data) # to numpy array
X=data[:,:3] # the input variables
y=data[:,3] # the output variable (local magnitude)
reg = LinearRegression().fit(X, y) # fit the model based on the data
reg.predict(np.array([[6.6, 8000, 27]])) # Get a prediction based on new inputs

>>> 6.1511027


text:
What is an "input embedding" in the context of NLP?
When reading about NLP, I saw it said that "input embeddings" are a main element of encoder-decoder learning frameworks for sequence modelling. What is an "input embedding" in the context of NLP?

An embedding is a vector that semantically represents an object/input, which, in the context of NLP, can be a character, word, or sentence, depending on the task. The main property of embeddings is that they are close/similar to each other (for some notion of similarity, e.g. the cosine similarity) if the corresponding original objects/inputs also have similar meanings or are contextually related. So, for example, the words "boy" and "man" are expected to be mapped to vectors/embeddings that are close to each other. There are different ways to create these embeddings. An example is word2vec. Note that the notion of an embedding also applies to other contexts, e.g. geometric deep learning. You can read more about this topic in chapter 6 of this book.

Ahh, yes, I'm in the early chapters of the Jurafsky and Martin textbook. Thanks for the clarification.

It's worth adding where the similarity comes from: usually embeddings are based on the surrounding context of a word, and both 'man' and 'boy' tend to be used in similar environments, at least more than 'man' and 'quickly'. The more the usage of two words overlaps, the more similar they tend to be.


text:
How to formulate discounted return in cartpole?
I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return:

finite horizon undiscounted return (used for episodic problems)

$G = \sum_{t=0}^T R_t$

infinite horizon discounted return (used for non-episodic problems).

$G = \sum_{t=0}^\infty \gamma^t R_t$
However, I'm confusing that "Is Cartpole episodic task?". Ideally, the simulation lasts forever.  This is my final objective (prolonging the lifetime). But it still has some termination states. Should I introduce the termination state and use it with a discounted return like:
$G = \sum_{t=0}^T \gamma^t R_t$

It's not true that finite horizon MDP can't use a discount factor. Any discount factor $\gamma \leq 1$ is fine for finite horizon.
Whether or not cartpole is a finite or infinite horizon depends on the environment implementation. The default environment in openai gym uses $T=200$ as the horizon. This is customizable, and most papers I see which use cartpole as a test environment use $T=1000$ instead. The task is considered "solved" when the agent can reach $T$ timesteps consistently (e.g., the last 100 episodes have all reached $T$ timesteps).
If you want to treat cartpole as an infinite horizon task, that's fine too. Clearly you need $\gamma < 1$ here, and you can just $T$ to some arbitrary large number.


text:
What is the difference (if any) between semantic segmentation and multi-class, mutually exclusive classification?
Multi-class classification is simply assigning all data points into one of up to any finite number of mutually exclusive labels.  I am new to the field(s) of AI/ML and I keep hearing people use the term "semantic segmentation."
I want to "translate" this AI/ML jargon into something more familiar to me.  The best video I have found so far to explain what it is made me wonder, what is the difference between semantic segmentation and classification?
NOTE:  I am specifically not referring to so-called multi-label "classification" which allows a data point to have more than one label at a time.  In my experience, that sort of labeling is not classification at all, which is a division into mutually exclusive sets (no overlap).

Image/object classification (or recognition)
(Multi-class) image/object classification (or recognition) typically refers to the task of assigning one label to an image, so we typically assume that there's only one main object in the image. The multi-class only refers to the fact that we have more than 2 possible classes or labels (if we had only 2, this would be binary classification), but note that this does not mean that we have more than one main object in each image. So, in this task, we are not interested in labelling each pixel, but to label the whole image, so, in this sense, this is a sparse classification task. An example of a dataset that is used for image classification is MNIST, where there's only one object (a number) per image. Here's a picture that shows 3 MNIST images, each of them has only one associated label (below), which corresponds to the number in the image.

Semantic segmentation
Semantic segmentation is the task of classifying each pixel in an image (or at least groups of pixels), so that objects of different classes have their pixels labelled differently. Instance segmentation is a similar task, but we additionally want to differentiate between different objects of the same class, so we assume that there could be more than one object in the image and there could even be more objects of the same type/label. Given that we label pixels, this is a dense classification task.
Here's an example of an image that has been segmented, i.e. pixels associated with the same object (e.g. the umbrella) have the same label (color).

Object detection
So, in this way, semantic (or instance) segmentation is more similar to object detection, which is both a classification and regression task, because we want both to classify one or more objects in the image, but we also want to draw a bounding box around them (and this is often solved as a regression problem). The reason why we draw a bounding box around each object is that, as opposed to image/object classification, there can be more than one object in the image, so we need a way to identify the locations of the objects. As opposed to semantic/instance segmentation, this is also a sparse classification task.
Here's an image to which object detection has been applied.

I was hoping for a straightforward answer, which is lacking.

@brethvoice If you ignore the "object detection" section (which was not part of your question, but I decided to add it because I think it can be useful) and the images, my answer is about 20 lines long. I don't know how to convey the definitions of these tasks in a shorter but still understandable way.

@brethvoice Maybe this description is what you are looking for: Object classification = sparse classification + typically only one main object in the image + each image is labelled with the only main object in the image. Image segmentation: dense classification (you classify pixels) + multiple objects in the image. In any case, the images above were supposed to intuitively convey the meaning of these tasks.

Here is what I am getting at: "objects of different classes have their pixels labelled differently" does not preclude a pixel from having more than one label simultaneously, does it?

@brethvoice As far as I know, yes, it generally precludes a pixel from having more than one label, because, otherwise, it would be a multi-label classification problem. Of course, it may also be possible to formulate image segmentation so that you associate multiple labels to the same pixel, if that makes sense for your problem (not sure when that would be needed, though).

Both things are similar. But, I think there is a bit of a difference in interpretation.
If what you are solving is a multi-class classification problem in an image, a proper measure of performance of an algorithm would be the accuracy of the prediction for each pixel.
While one of the most used measures of performance for semantic segmentation is the
IOU (intersection over union) for each class. Which, makes sense if your objective is to create a segmentation (a mask) for each class.

Why are you asking about 3 unrelated items? Multi-class and mutually exclusive are related. Semantic segmentation is an approach to object recognition in images, that has nothing to do with the other 2.
The different between multi-class and mutually exclusive is simple: a spam engine is an example of mutually exclusive, each item either is spam or is not. An algorithm that would potentially guess what grade an essay would get is an example of multi-class: there are many potential classifications, not a simple binarism.