text:
Do other online/incremental algorithms not suffer from catastrophic forgetting?
All the literature I read seems to indicate catastrophic forgetting affects only neural networks. Do other online/incremental algorithms not suffer from catastrophic forgetting (for example, SGDClassifier)? Why would that be the case?

To enlighten this, you need to understand the cause of the catastrophic forgetting. Fundamentally, the cause is an overlap in representations of different aspects of data in the learning model [Using Semi-Distributed Representations to Overcome
Catastrophic Forgetting in Connectionlst Networks]. This can be explained easily in neural networks based on the representations of the data in hidden layers. However, in learning models such as SGD Classifier, we cannot see any explicit representation for input data, and we merely have a decision function to classify it. Hence, we cannot see any serious discussion about catastrophic forgetting in machine learning literature around other learning methods than neural networks.
By the way, this forgetting can happen in another form of the problem called "concept drift" [Understanding Concept Drifts]. If the data distribution will change during training a model (any learning model such as SGD Classifier), the model is likely to forget the first part of the data as it is fitted to the new distribution. The cause of such an issue is the capacity of the model for training different distributions of data. So, you can find many discussions in the literature on "Concept Drift" for all types of learning methods.

thank you for the reply. Just to make sure, is it that catastrophic forgetting can take place even when there is no concept drift? It would greatly help if you can illustrate an example at a very broad level for the case where catastrophic forgetting can take place but there is no concept drift.


text:
What exactly is a grid-like topology according to the book Deep Learning?
I am reading this book called "Deep Learning" (by Goodfellow, Bengio and Courville).
On page 326, in the first paragraph, it says:

CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixel

Considering an image as a grid is completely intuitive. And, similarly, we can extend the logic to a 1-D time series.
But then what cannot be considered as having a grid-like structure?

Graphs.
In graphs, nodes and edges do not usually have a specified position/orientation: for example, we cannot say that node $A$ is to the right/left of node $B$ because edges do not typically have an orientation. In grid-like structures, like images, we could say that about pixels. In a graph, all we know is that nodes are connected to other nodes via some edges, which may have weights, but, again, no orientation.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can neural networks have continuous inputs and outputs, or do they have to be discrete?
In general, can ANNs have continuous inputs and outputs, or do they have to be discrete?
So, basically, I would like to have a mapping of continuous inputs to continuous outputs. Is this possible? Does this depend on the type of ANN?
More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not (a more detailed description of my problem can be found here).

Neural networks normally work in continuous spaces. A typical neural network function could be written as $f(\mathbf{x}, \mathbf{\theta}): \mathbb{R}^N \rightarrow \mathbb{R}^M$. That is, a function of some $N$ dimensional input vector of real numbers $\mathbf{x}$ that outputs some $M$ dimensional output vector of real numbers (which you could call $\mathbf{y}$) and that is parametrised by a vector of weights and biases $\mathbf{\theta}$.
When you train a neural network with some data $\mathbf{X}, \mathbf{Y}$ then you are trying to adjust $\mathbf{\theta}$ so that $f(\mathbf{x}, \mathbf{\theta})$ approximates some assumed mapping function $g(\mathbf{x}): \mathcal{X} \rightarrow \mathcal{Y}$ that provides a "true" mapping between all possible inputs $\mathcal{X}$ and the matching correct values from all possible outputs $\mathcal{Y}$. In a neural network, then $\mathcal{X}$ and $\mathcal{Y}$ must be sets of vectors of real numbers.
So, yes all neural networks map between continuous values by default*. When NNs are used to work with discrete values, then the discrete variables need to be converted to real-valued variables in order to be usable with a NN. So discrete inputs might be mapped to a small set of different real values or a vector of $\{0, 1 \}$ (called "one hot" encoding). Discrete outputs are often mapped to a probability vector giving the confidence that the learned function assigns to each possible discrete output - this is the usual case in classifiers.
You have a specific use case in mind:

More specifically, I would like to use neural networks for learning the Q-function in Reinforcement Learning. My problem has basically a continuous state and action space, and I would like to know whether I have to discretize it or not

For an action value, or Q function in reinforcement learning, then you do not need to be concerned that state space or action space are continuous. However, there are some details that may be important:

Neural networks learn most efficiently when input elements are within a unit-like distribution such as $\mathcal{N}(0,1)$ (normal distribution with mean $0$, standard deviation $1$). This does not have to be precise, but you should take care to scale the input values so that each element has a typical magnitude around $1$.

The output values you want to learn need to be within the domain of the output layer's activation function. It is common to use linear (or "no" activation function) in the output layer for regression problems for this reason, and learning the action value Q associated with a state, action pair is a regression problem. So unless you have good reason to do otherwise, use a linear activation function on the output layer.

Control problems in continuous action spaces cannot be solved using only a learned action value function. That is because, in order to derive a greedy policy from a Q function, you need to solve $\pi(s) = \text{argmax}_aQ(s,a)$ and finding the maximum value in a continuous action space is not practical. If your Q network is part of an agent solving a control problem - i.e. finding the optimal policy - then you will need some other way to generate and improve the policy. This is usually done using a policy gradient method.

Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN.

* Caveats:

There are a few architectures which really do use discrete inputs, weights and/or outputs. These are either historical variants or specialist though, you will not find any mainstream libraries that do this by default.

Computers model real numbers as floating point which are technically discrete due to practical limitations of CPUs. Sometimes it is important to know this, but it will not impact modelling a Q function.

Thanks Neil for your great answer (I had alreay upvoted and accepted it). One follow up question to your comment "Due to this last point, it is sometimes advisable to discretise the action space if you want to use a simpler value-based RL method, such as DQN."--> Basically I do want to use DQN (maybe also SARSA afterwards) and i would like to know whether it is also advisable to discretize the state space and not only the action space. My state space is quite large. I have about 10 variables that are all continious from min 0 to max 10,000.

@PeterBe You do not need to discretise the state space for DQN. Do remember to scale the state values for use with a neural network (e.g. scale each value from -1.5 to 1.5 is reasonable)

Thanks Neil for your anwer. I appreciate it. By scaling you mean to normalize or standardize each variable, right?

@PeterBe Yes, you should scale to fit each vraiable into a standard range

Hi Neil, thanks for your answers. I appreciate it. Basically I don't really understand why I should not discretize the state space but the action space. The ANN just maps inputs to outputs. When used in a RL environment the ANN maps state/action pairs to a reward as far as I understood. Surely, if you have less state/action pairs (which you will have if you dicretize the state space) the mapping of the ANN is not as challenging which will most propably lead to better results for the Q-learning algorithm

@PeterBe: My answer spells that out. The discretisation is required to support max and argmax operations over the action space. You don't need to do that over the state space. Nothing stops you if you want to though.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Given an input of shape $(3, 32, 32)$, which is convolved with a $(3 \times 3)$ kernel, how do I calculate the FLOPS?
I have an input tensor of shape $\mathbf{(3, 32, 32)}$ consisting of 3 channels, 16 rows, and 16 columns. I want to convolve the input tensor using $\mathbf{(3 \times 3)}$ kernel/filter. How can I calculate the required FLOPs?

FLOPs required to compute it

First, do you know what a FLOP is?

Floating point operations

And do you know how to calculate the output? If you know how to calculate the output, you should be able to count the FLOPs in the calculation.

I don't know how to. That's why I asked the question

How many output channels?

It will be the same as the input. Three

Each output pixel channel is a 3x3x3 filter, so 27 inputs which get multiplied by 27 weights and then added together. This is 27 FMA (fused-multiply-add) operations, or 27 multiply operations and 26 additions. I believe all modern devices implement FMA.
The number of output pixel channels is 30x30x3 = 2700 (as a 3x3 kernel shaves off one pixel on each edge) and each one takes 27 operations to calculate. So that's 72900 operations in total.


text:
How does the learning rate $\alpha$ vary in stationary and non-stationary environments?
In Sutton and Barto's book (Chapter 6: TD learning, 2nd edition), he mentions two ways of updating value function:

Monte Carlo method: $V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)]$.
TD(0) method: $V(S_t) \leftarrow V(S_t) + \alpha[R_{t+1} + \gamma V(S_{t+1}) - V(S_t)]$.

I understand that $\alpha$ acts like a learning rate where it take some proportion of MC/TD error and update value function.
From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time. Hence, one should supposedly use $\alpha-$decay to update value functions. On the other hand, since distributions change with time in non-stationary environments, $\alpha$ should be kept constant so as to keep updating the value function with recent TD/MC errors (in other words, history doesn't matter).
What's been bothering me is that in Example 6.2, 6.5, and 6.7, probability and reward distribution doesn't change. So why is constant-$\alpha$ being used?
Question: How does $\alpha$ vary in stationary and non-stationary environments?

So why is constant-$\alpha$ being used?

This is because control scenarios are inherently non-stationary with respect to value functions. Decaying alpha comes with a risk that improvements to the policy will occur progressively more slowly, because the impact to changing the policy will be learned slowly.

From my understanding, in stationary environments, transition probability distribution and reward distribution don't vary with time.

This is correct when considering immediate reward and transitions from any give $(s,a)$ pair. However you are forgetting that the policy function $\pi(a|s)$ does vary with time whilst the agent is discovering the optimal policy, and this affects trajetories and expected values that are being estimated.

Question: How does $\alpha$ vary in stationary and non-stationary environments?

For non-stationary environments, you will want to maintain some minimum learning rate $\alpha$ and also some mininum exploration ($\epsilon$ if you are using $\epsilon$-greedy exploration).
In stationary environments, then a learning rate schedule is still valid and may be useful. You will see it discussed in passing when referencing proofs of convergence for the basic algorithms.
For example in the second edition of Reinforcement Learning: An Introduction it says this regarding convergence of Q learning:

Under this assumption and a variant of the usual stochastic approximation conditions on
the sequence of step-size parameters, Q has been shown to converge with probability 1 to
$q_*$.

The "usual stochastic approximation conditions on the sequence of step-size parameters" part is a reference to decaying the learning rate.
However, due to the added complexity of handling exploration vs exploitation, the inherently non-stationary nature of value predictions in control scenarios, and the difficulty of getting function approximation to work, discussions of learning rate decay is a minor detail in texts like Sutton & Barto.

I don't follow how "this affects trajectories and expected values that are being estimated" relate to non-stationary environment. Regardless of what policy I follow, for a particular state-action pair,  won't $R(s, a, s')$ be sampled from the same distribution (albeit, varying rewards)?

@user529295 Yes, but the total return is dependent on the whole trajectory, which in turn depends on the policy, which is changing. Hence non-stationary expected returns (not individual expected rewards) even if the environment is itself stationary

Ah! Makes sense.


text:
Is there an entry level textbook on Bayesian Inference that is a nice blend of theory and applications?
I am looking for a textbook that is a nice entry level to Bayesian Inference. I was hoping that there is a nice blend of theory and applications (data sets) on how concepts are applied. Programming techniques presented are welcome.
Just for perspective, I feel that Christopher Bishop's PRML is a theoretical treatment. It is very good theoretically, but I find myself not understanding how to apply it given a data set.
I have tried jumping from one book to another and this has just confused me. Is there any authoritative book with these requirements?

Are you asking about inference in Bayesian graph?

Sorry, not about Bayesian graphs, but Bayesian inference in general.

Using as a best reference accordingly my own google research, find the best post about best introductory Bayesian statistics book and summarize the answers. I find this post in stats.stackexchange about bayesian statistics books maybe this is the best recomendation for you. I read the post weeks ago and some books are stunning.
This is my TOP 3 books from the answer from stats this books repeat themselves in answers below (all the books have applications)

Doing Bayesian Data Analysis: A Tutorial with R and BUGS.
Bayesian Data Analysis 3rd Andrew Gelman
Statistical Rethinking A Bayesian Course with Examples in R and Stan Second Edition Richard McElreath

I start reading Bayesian Data Analysis 3rd Andrew Gelman days ago I am in the first chapter about Bayesian Inference you can check the first chapter:
Part I: Fundamentals of Bayesian Inference 1
1 Probability and inference 3
1.1 The three steps of Bayesian data analysis 3
1.2 General notation for statistical inference 4
1.3 Bayesian inference 6
1.4 Discrete probability examples: genetics and spell checking 8
1.5 Probability as a measure of uncertainty 11
1.6 Example of probability assignment: football point spreads 13
1.7 Example: estimating the accuracy of record linkage 16
1.8 Some useful results from probability theory 19
1.9 Computation and software 22
1.10 Bayesian inference in applied statistics
24 1.11 Bibliographic note 25
1.12 Exercises 27

and also the book have the Chapter 10 Introduction to Bayesian computation..
Some people prefer some book than others for understand Bayesian Statistics so you have to choose which one fits you, for example statistical rethinking is a different than other books is another way to explain bayesian statistics some people not understand. but is the best 3 from the post.
Just take a look the chapter difference between Gelman and Statistical Rethinking


text:
How is this statement from a TensorFlow implementation of a certain KL-divergence formula related to the corresponding formula?
I am trying to understand a certain KL-divergence formula (which can be found on page 6 of the paper Evidential Deep Learning to Quantify Classification Uncertainty) and found a TensorFlow implementation for it. I understand most parts of the formula and put colored frames around them. Unfortunately, there is one term in the implementation (underlined red) that I can't tell how it fits in the formula.

Is this a mistake in the implementation? I don't understand how the red part is necessary.

It should remain from a general code that has been refactored. By the way, the red code phrase is always zero. Because, beta is a vector of 1, and $\log(\Gamma(1)) = \log(1) = 0$, i.e., tf.math.lgamma(beta). So, sum of zeros will be zero.
As you said, the other parts of the code are clear and completely matched with the definition.

Thank you, you are right. I was worried because I found this code multiple times on the internet, but it seems everyone copied from the same source.


text:
Can future information be included in a control problem with Reinforcement Learning?
I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour (more details can be seen here:Reinforcement learning applicable to a scheduling problem?). Although the problem is basically a scheduling problem, I want to implement it like a control problem for every time step.
Now, I have 2 questions:

Is it possible to somehow consider future values (e.g. of the electricity price) while during a control action for every time slot? E.g. when the agent knows that in 2 hours the price will fall significantly, then it should tend to consume electricity in 2 hours to get closer to the optimal solution.

Related to 1: Is it possible to get the reward just at the end of the day instead of every hour (although the control actions are every hour)? If you get the reward at every hour, this might lead to a greedy behaviour, which often results in bad results.

Typically in a control problem, it is OK to include data about a future event, if it can be reliably predicted at the time that a decision is required.
This would include known rules such as a pricing schedule. You could even use some feature engineering to help the agent by making a state feature that counted down to changes, or presented it as a relative change to current price. E.g. it would be fine to have a couple of features that pre-calculated "in 10 minutes, prices should change by -0.003 per kWH" - whether or not that helps your agent in practice I could not say.
What you should not include is any resolution of random variables in advance. It would be OK to predict these using a model, or give the agent details about the distribution, but it is not OK to work with all data from all time steps already resolved, when the agent will in reality be required to make a decision without that data. An example of this kind of data is the measured internal or external temperature on future time steps. (Predicted temperature from a weather forecast would be OK though)
This constraint, of not showing the controller the whole future when it is making a decision, should also be applied for other controller optimisations as well as reinforcement learning - no real world controller has perfect knowledge of how stochastic results will turn out, or the ability to rewind time and make the correct decision in the past based on observations that happened after it made that decision. However, to repeat the point made at the start, it is OK to make those decisions based on predictions of what will happen, where it is reasonable to have the knowledge in advance, and those predictions could be very accurate in some environments. A pricing schedule that is agreed in advance with your power provider should be one of those types of knowledge that it is fine to use.

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Implementing Multiple NNs in one DQN model?
I'm trying to build a DQN Agent to take a set of 10 best actions simultaneously (integer values from 1 to 100) as outputs per episode. The input is a float. The goal is to find the optimal combination of (10) actions per episode.
Currently, the set up is having a single NN output 10 actions w/ the highest q-valules for each episode. But in the Memory Replay process, each individual Set (of 10 fixed actions obtained from the exploration phase) is being treated as a single action. Because the target network also takes the output of the list of 10-action from the main NN. Hence I can see the agent repeatedly trying certain Set (with a fixed 10 actions) in the replay/retrain part, whereas our goal is to find the optimal combination of 10 actions per episode, Not the optimal Set of fixed combinations. So in essence, I would like the agent to pick out and mix up the actions from the Sets with higher Q-values (known from the exploration phase) to form new optimal "Sets" in the Replay process.
I was thinking maybe instead of using a single NN with 10 outputs I could do 10 NNs with single outputs for each episode so that each action is treated separately. And I suppose I will have 10 q-networks and target networks as well, then I could combine the results by the end of each episodes. But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.
Alternatively, I think the problem could be treated as a multi-armed bandit problem, except each arm here has "sub-arms" too so to speak, but that could require some changes to the custom environment I am working with and I don't want to touch that unless necessary.
Maybe there is a clever manipulation within the retrain process given my current setup that I am not seeing. Here is a snippet of the code for some more clarity.
class DQNAgent():

    def __init__(self,optimizer):
        # Initialize atributes
        self._state_size = 1
        self._action_size = 76
        self._optimizer = optimizer

        self.experience_replay = deque(maxlen=2000)

        # # Initialize discount and exploration rate
        # self.gamma = 0.6
        # self.epsilon = 0.5
        self.gamma = 0.95
        self.epsilon = 1.0
        self.epsilon_min = 0.01
        self.epsilon_decay = 0.95
        self.learning_rate = 0.01

        # Build networks
        self.q_network = self._build_compile_model()
        self.target_network = self._build_compile_model()

    def store(self, state, action, reward, next_state, terminated):
        self.experience_replay.append((state, action, reward, next_state, terminated))

    def _build_compile_model(self):

        model = Sequential()
        model.add(InputLayer(input_shape=(self._state_size,)))
        model.add(Dense(100, activation='relu'))
        model.add(Dense(100, activation='relu'))
        model.add(Dense(self._action_size, activation='linear'))
        model.compile(loss='mse', optimizer=self._optimizer)
        return model

    def alighn_target_model(self):
        self.target_network.set_weights(self.q_network.get_weights())

    def retrain(self, batch_size):
        if len(self.expirience_replay) < batch_size:
            return
        minibatch = random.sample(self.expirience_replay, batch_size)

        for state, action, reward, next_state, terminated in minibatch:

            target = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
            print('target size :', np.shape(target))

            if terminated:
                target[0][action] = reward
            else:
                t = self.target_network.predict(np.reshape(np.array(next_state), (-1,1)))
                target[0][action] = reward + self.gamma * np.amax(t)

            self.q_network.fit(np.reshape(np.array(state), (-1,1)), target, epochs=1, verbose=0)

    def act(self,state):
        self.epsilon *= self.epsilon_decay
        self.epsilon = max(self.epsilon_min, self.epsilon)
        action_space = [ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
       52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68,
       69, 70, 71, 72, 73, 74, 75, 76] #all 76 available nodes
        if np.random.rand() <= self.epsilon:
            return np.array(random.sample(action_space,10))-1 #-1 to match control's index

        q_values = self.q_network.predict(np.reshape(np.array(state), (-1,1)))
        print("q_vals shape",np.shape(q_values))
        print('q_vals type',type(q_values))

        top_actions_idx = q_values[0].argsort()[-10:][::-1]

But, I am not sure if that is necessarily the best way to fix the problem of having repetitive sets of fixed action in the replay process.

The best way to handle this is using a Multi Discrete Action Space. You don't need a neural network for every action on its own.

Hi @tnfru, thanks for your suggestions. I'm not really familiar with this concept. I am under the impression that for each action taken, other actions in the action space would be impacted in someway. Could I simultaneously sample actions from several different action space for every single episode? And if I were to use Multi Discrete Action Space, would it be necessary to adopt methods like actor-critic instead?

Yes @Rkz , DQN will work on discrete actions and Multi Discrete allows you to select from a set of values per action dimension and thus your action space is discrete. You don't need to switch methods per se, but it is very unual to use DQN for Multi Discrete so I would recommend the switch yes.


text:
Is it really hard to learn in a stochastic environment?
I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, right?).
For example, the frozen lake environment is a stochastic environment. Sometimes you want to move in one direction and the agent slips and moves in another direction. Unlike an environment with multiple agents that the probability of the actions of the other agents is changing because they keep learning (a non-stationary environment).
Why is it difficult to learn in a stochastic environment, if, for example, Q-learning can solve the frozen lake environment? In what cases would it be difficult to learn in a stochastic environment?
I have found some articles that address that issue, but I don't understand why it would be difficult if Q-learning can solve it (for discrete states/actions).


text:
Can I always interpret features as random variables in machine learning safely?
Consider the following statements from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

Machine learning tasks are usually described in terms of how the
machine learning system should process an example. An example is a
collection of features that have been quantitatively measured from
some object or event that we want the machine learning system to
process. We typically represent an example as a vector $\mathbf{x} \in
 \mathbb{R}^n$ where each entry $x_i$ of the vector is another feature. For example,
the features of an image are usually the values of the pixels in the image.

Here, an example is described as a collection of features, which are real numbers. In probability theory, a random variable is also a real-valued function.
Can I always interpret features in machine learning as random variables or are there any exceptions for this interpretation?

In general terms yes. Because what the ML algorithms do in general is to learn the hidden probability density function of the target examples (cats, dogs..). And that is done by learning the conditional probability function between inputs, $X$, and target outputs, $y$, for discriminative models or by learning the joint probability function for generative models.
More on discriminative VS generative models and their relation with probabilities in this nice article from where I took the picture.

It also helps to think that given a set of different examples (say the image of cats). What the ML algorithm really learns is the invariant features between all the examples, i.e. the cats. So the ML algorithm learns to discriminate all the variability (backgrounds, occluding objects) and learn what is invariant: the representation of a cat.
This representation, since images are pixels, can be described as a probability density function of values that would represent a cat. It would be like the ML algorithm said: "If those pixels values behave close to the probability density function I learned, then those pixels must represent a cat!"

You say "... hidden probability density function", but maybe you mean "... unknown probability density function"? Because $p(y \mid x)$ is typically unknown. Hidden refers to something different. For example, the hidden space in VAEs or hidden variable in Hidden Markov models.

You are right, thinking it through I think we could say: “unknown but existent”, like although it is unknown, we know it is there


text:
What does it mean there is no rollout in AlphaZero's training?
According to a blog post by DeepMind, AlphaZero doesn't have a real rollout.

AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will win from the current board position. Instead, it relies on its high quality neural networks to evaluate positions.

Instead, I assume it just interprets the winner at a given state by the NN values head result. This replaces the rollout. So the computation time saved could be used for many expansions instead. Evaluating a state from a root node would then be the best action derived from the visit count in MCTS, which is only based on the predictions of the NN value heads. (no current score, no policy?)
With policy, I mean the NN's policy head (softmax).
This would mean that the NN policy is only used in the loss calculation and nowhere else?

You have lot's of questions and interpretations there. So I'll do my best to explain it below.

TL;DR: Alpha Zero removed rollout altogether from MCTS and just used current DNN estimates instead.

The single Deep Neural Network has 2 heads:

A Value Head (which assigns a score to each state).
And a Policy Head (which predicts the score for all possible moves).

Instead of doing rollouts to determine the outcome, it uses the DNN estimations, so it don't need to explore too deep.
By relying on the DNN, the MCTS gets even simpler:

Probability of each action is a simple normalization of previously obtained Policy Head.
Selection chooses move with “low count, high move
probability, and high value”
Expansion is done by DNN, outputing Value and Policy.
Simulation with rollouts: no longer needed.
Back-propagation updates nodes using DNN's value (instead of a rollout outcome).

Source


text:
In deep reinforcement learning, what is this model with state as input and value as output?
I was looking at this implementation for creating an agent for playing Tetris using DeepRL.
This model uses "a state based on the statistics of the board after a potential action. All predictions would be compared but the action with the best state would be used".
So at each iteration, it's feeding a set of future states, computed based on the current state (future state made up of statistics of the game like nb of holes in the board, cleared rows, total height...)
to a neural network and outputs one "value" per future state.
So at every step, you predict N "values" from the neural network for all N possible future states for the one you are currently in and choose the greatest one as your future state and thus associated action.
Now, my issue: the implementation says it's "deep Q-learning", but I do not see it that way. The action, nor some sort of current state is given as input of the network.
Since it is feeding the "future states", for me, it looks more like a value iteration algorithm with a neural network or at least something where you know the transition model?
Did I miss something and it is actually DQN?
If not, do you have any references for this kind of RL model? Does this have a name?

This is a variant of RL value-based approach using afterstate values. These are similar to action values, but have the following properties:

Afterstates treat an action as "choosing a next state". This works well for deterministic environments, or in games where setting a board state is at least deterministic before any random factors might apply (such as an opponent's turn).

Afterstates can result in efficient functions (and efficient learning) when there are multiple paths to a given state and where the details of how that state was achieved do not matter much.

An afterstate value function is pretty similar to an action value function, provided the agent can predict all the state features that will result. This is not necessarily the same as needing a full model, as it does not need to account for changes after the state is selected due the environment - or opponent - but before the next time step. However, it is often more than a simple action selection, for example in tetris you may have actions which are rotate, shift, drop, but the afterstate selection will be a model of what happens to state for each allowed action selection. So some functions are needed to:

Convert allowed actions to afterstates before passing them to an afterstate selection algorithm.

Convert selected next states to actions before passing back to the environment.

Environments can also be written to present valid next states and only accept a valid next state as a selection from the agent. It is possible to code Tic Tac Toe agents to do this for example - the environment generates all possible next states, and the agent can select one. The environment may also check that the selected next state is a valid change from the current state. In that case you do not need any conversion between actions and afterstates, everything is handled by passing states between agent and environment. I would not expect a Tetris game and agent to be written so directly like this though.
Due to the similarity of afterstate values with action values, it is straightforward to adjust other value-based methods like Q-learning or SARSA to use afterstates.


text:
How to define an action space when an agent can take multiple sub-actions in a step?
I'm attempting to design an action space in OpenAI's gym and hitting the following roadblock. I've looked at this post which is closely related but subtly different.
The environment I'm writing needs to allow an agent to make between $1$ and $n$ sub-actions in each step. Leaving it up to the agent to decide how many sub-actions it wants to take. So, something like (sub-action-category, sub-action-id, action) where the agent can specify between $1$ and $n$ such tuples.
It doesn't seem possible to define a Box space without specifying bounds on the shape which is what I need here. I'm trying to avoid defining an action space where each sub-action is explicitly enumerated by the environment like (action) tuple with n entries for each sub-action.
Are there any other spaces I could use to dynamically scale the space?

Is the last element of the tuple (sub-action-category, sub-action-id, action) a discrete or continuous value? What is its dimension? gym offers the class MultiDiscrete for a similar use case. You could also write your own action type if you wanted to, but imo the straightest way to deal with this is to have actions to be instances of some suitably defined Python class you can document etc. and deal with the details of the execution to the Environment subclass.

One way to handle an arbitrarily large sequence is by adding a STOP signal as one possible token in the sequence, just like LSTM.
So you could divide your game in turns:

What you now call a single action (composed by multiple sub-actions) would become a turn.
Now, you can have as many actions you'd like inside a turn.
Each action is simply a list accumulated inside the environment, but won't evaluate the game yet.
When the player is satisfied with their actions, they can call the action: "End Turn".
When turn ends, you can concatenate every sub-actions, evaluate the game and proceed as usual.


text:
What is the difference between a vision transformer and image-based relational learning?
I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.

I'm on top of the vision transformer but not the RL. But I know enough to tell you that these are very unrelated papers (within deep learning). And transformers are used across many types of tasks (vision, NLP, RL, audio).
Imagine making an analogy from DL to all vehicles. In this context you can't exactly say a car is like a plane because they both have pistons. Same reason I wouldn't say these papers are particularly related by the fact that they both use transformers.

An Image is Worth 16X16 Words:
TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE
A Transformer consists of alternating layers of multiheaded self-attention.
The Transformer Paper adapts a NLP architecture for making Image Classification. For that, it first need to tokenize the image (like a piece of text). The tokenization is done by splitting the image into fixed-size patches and then embedding it.
In other words, they treat a picture as a set of sub-images, just like a phrase is a set of words.
Relational Deep Reinforcement Learning
The Reinforcement Learning paper uses self-attention to iteratively reason about the relations between entities in a scene and to guide a model-free policy.
Focused on game playing, they decide to model a non-local concept of entities (players, objects) and the relationship between them.
So instead of tokenizing the image, they first extract entities from it, using a more standard approach, which is convolution. So the multi-head attention layer receives a set of entities, the output is normalized so it's once again formatted as the entity set, so it can proceed with an iterative process.

TL;DR:
Same base (very promising) technique, but applied to 2 drastically different models.


text:
What is the borderline between unsupervised learning and regular algorithms?
Unsupervised learning using neural networks is clearly machine learning since it is utilising neural nets.
However, some algorithms, k-means clustering, for example, are considered unsupervised learning, while they look just regular algorithms (non-ML).
What should be the borderline (criteria) to differentiate between unsupervised learning and a non-ML algorithm?

Any algorithm that uses data (in some form) to improve some performance measure (aka objective function), or to find some function, can be considered a machine learning algorithm. See this answer for more complete definitions of ML.
k-means does that. It uses the data to find some division of the data itself into groups, in order to maximize some objective function. So, k-means is a machine learning algorithm.
The use of neural networks is not a requirement for something to be called a machine learning approach. In fact, there are many other machine learning approaches/algorithms that do not use them, such as tabular Q-learning, support vector machines or hidden Markov models.

tks, data+improvement is the point

data here are possibly statistical data to be specific, because some optimisation algorithms in graph theory are also optimising for better values after a number of iterations on the graph data

@datdinhquoc I don't know what definition of "statistical data" you're using, but by "data" I mean any data, including graphs. So, if that algorithm you're talking about uses data, such as graphs, and optimizes some objective function given more data, then it could be considered an ML algorithm too. In fact, there's an area in ML called "geometric deep learning", which is concerned with the application of deep learning-like ideas to graphs. If you provide the name of the specific algorithm, I may look at it later and tell you more what I think of it.


text:
How does randomization avoid entering infinite loops in the vacuum cleaner problem?
Suppose we have a vacuum cleaner operating in a $1 \times 2$ rectangle consisting of locations $A$ and $B$. The cleaner's actions are Suck, Left, and Right and it can't go out of the rectangle and the squares are either empty or dirty. I know this is an amateur question but how does randomization (for instance flipping a fair coin) avoid entering the infinite loop? Aren't we entering such a loop If the result of the toss is heads in odd tosses and tails in even tosses?
This is the text from the book "Artificial Intelligence: A Modern Approach" by Russell and Norvig

We can see a similar problem arising in the vacuum world. Suppose that a simple reflex vacuum agent is deprived of its location sensor and has only a dirt sensor. Such an agent has just two possible percepts: [Dirty] and [Clean]. It can Suck in response to [Dirty]; what should it do in response to [Clean]? Moving Left fails (forever) if it happens to start in square
A, and moving Right fails (forever) if it happens to start in square B. Infinite loops are often unavoidable for simple reflex agents operating in partially observable environments. Escape from infinite loops is possible if the agent can randomize its actions. For example, if the vacuum agent perceives [Clean], it might flip a coin to choose between Right and Left. It is easy to show that the agent will reach the other square in an average of two steps. Then, if that square is dirty, the agent will clean it and the task will be complete. Hence, a randomized simple reflex agent might outperform a deterministic simple reflex agent.

And this is the agent program from the same source:
function REFLEX-VACUUM-AGENT([location,status]) returns an action
 if status = Dirty then return Suck
 else if location = A then return Right
 else if location = B then return Left

how does randomization (for instance flipping a fair coin) avoid entering the infinite loop?

The coin is flipped on each occasion that a decision is required (as opposed to once in order to define the actions that will be taken from that point onwards). Which means the agent can make a different decision each time it encounters the same (faulty detected) state.
This prevents tight loops in a single location that the quote is discussing. The "infinite loop" that is being broken is repeatedly and deterministically making a movement decision that results in no effective movement, thus leaving the agent in exactly the same state as before for all remaining time steps.

Aren't we entering such a loop If the result of the toss is heads in odd tosses and tails in even tosses?

It is not important that the agent might repeat ineffective movements multiple times. Eventually the agent will make a correct decision - this is very likely after only a few failed attempts at most. It is guaranteed in the limit of infinite time steps, unlike the situation where no randomness is added.


text:
What is the intuition behind comparing action values to state values in the policy improvement theorem?
Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.
What is the intuition behind this comparison?
It seems more natural to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference but perhaps conceptually it does?


text:
Can an AI create another better AI, which in turn creates another better AI, and so on?
I have no specific knowledge of the AI field, but I heard that AI systems get better the longer they learn.
So, I was wondering: could it be possible that AIs will learn how to create better AIs (Or assists humans to create a better AI), and then those better AIs will learn to create an even better/faster AI, and so on? Wouldn't this mean that the AI would get exponentially better/faster because, after each successive generation, a slightly faster AI will do the job?
I also heard that "Google is using AI to design processors that run AI more efficiently". Wouldn't this be the same? AI designs faster CPU => AI get's faster and can design an even better CPU to run on.
Is something like that possible? Would this mean that at some point there will be a breakthrough in AI that will significantly increase the speed of AIs because of those loops?

No. Your premise that AI systems get better over time (without any other input) is not accurate.

The learning process isn't well addressed in this question, but from my understanding the premise is only that an AI (or some AI) can learn (without caring how/from what data it learns).
So the premise is indeed flawed in the sense that an AI can't learn indefinitely (there is a limit to how much it can learn, and the auteur should understand it) but, IMHO, it isn't enough to totally discard the question as it could (at least theoretically) learn enough to create a "better" version of itself.

Just want to say that I think your question is justified and I believe it's possible in the future. If we could create AGI, and if it has more combined info than all organizations or any single person in the future, it could theoretically have the ingredients to construct something novel by synthesis of an enormous amount of disparate phenomena and ideas that no AI researcher can on their own. No AI researcher has studied every field under the sun. If you have an AGI that had done that, it could pull inspiration and remix like nobody's business.

The first thing to consider is that (as of today) there is no general AI (AGI) and most if not all research don't try to create an AGI. An AGI can be described as an AI able to perform well (at least at a human-like level) on a wide range of tasks. If you are interested in the question I recommand you to do some research about the term AGI as there is no consensual definition, but I believe that the one I gave you will be enough here.
This means that an AI can actually do just one (or a few) task and so, if we created an AI creating new AIs it would just create new AIs better to either do something else (so there is no loop of improvement and it's not what you are interested in) or better to create a better version of themselves which in itself is useless as this better version can't do anything useful to us (except maybe to show that we can do it).
I don't know if there are examples of research creating an AI improving themselves, but there are examples of AI improving other AI such as this one where an evolutionary algorithm is used to design a better neural network.
Concerning your example about AI improving the hardware, it could indeed lead to the improvement of a wide range of AI, but it's, in my opinion, different than an AI improving themself as the loop take several years before being closed (from this source it take several years to produce a new cpu) where an AI directly training an AI could do so in a matter of days (or weeks or hours depending on the technology, but anyway it's way faster)
But, if we had an AGI able to create better, more or equally general AI, it could indeed lead to a breakthrough in AI*, or this is at least the opinion of some people including Nick Bostrom which wrote the famous book "superintelligence" in which he described how an AGI could very rapidly improve itself and described it as a possible singularity (meaning that we are unable to foresee what this AGI could become and how fast it could improve itself).
*at the condition that a better AGI is able to further improve the next version, which is not so obvious (thanks Neil Slater for this remark). Adding this condition it means that your metrics for "better" must include the ability to create a better AGI and that you can effectively always (or for long enough to obtain a breakthrough) improve this ability (As far as I know, we don't know if it's indeed the case)

Another thing worth considering: An AGI that can improve another AGI does not necessarily lead to that second AGI being able to further improve the next generation (as if the "capability to improve" is just some real number that can be manipulated through wishful thinking). The idea of exponential improvements to AGI is based on some very fuzzy understanding of what an AGI is, let alone an "improved AGI" or some kind of measure that could grow generation to generation.

Indeed, it's not so obvious that an improved AGI can further improve the next iteration, I added a point about this.
As for the idea of exponential improvement, I wrote that it's the opinion of some people, not that there is a consensus about it. As far as I know this is an idea that I would judge as plausible, and I see no argument against it. From what you say this is due to a fuzzy understanding of what an AGI is, could you be more precise on what misunderstanding of an AGI could revoke this idea ?

The misunderstanding is about measuring capabilties. In terms of D&D-like stats, lets call the stats INT (for general intelligence), CraftINT (for ability to create another object with a specific INT). The sigularity starts with an AGI with INT 10 and CraftINT of 11, then goes on to assume that when the INT 11 AGI is built it will itself have CraftINT even higher. However, we have no theory that can measure or predict anything for CraftINT - or anything similar. We could link INT to something like IQ, but what we know so far is that no human, whatever their IQ, has any CraftINT ability.

. . .  unless you include biological reproduction, in which case human CraftINT is essentially the same as their INT in the short term, and evolution may cause both to drift up or down depending on the environment. Either way, we are presupposing a relationship between INT and CraftINT when we cannot define or describe the latter in anything other than the most general terms. Applying any kind of maths to it is extreme conjecture.


text:
Is it normal that the values of the LogSoftmax function are very large negative numbers?
I have trained a classification network with PyTorch lightning where my training step looks like below:
def training_step(self, batch, batch_idx):
    x, y = batch
    y_hat = self(x)
    loss = F.cross_entropy(y_hat, y)
    self.log("train_loss", loss, on_epoch=True)
    return

When I look at the output logits, almost all of them are very large negative numbers, with one that is usually 0. Is this normal or might be something wrong with my training?
I am just using nn.LogSoftmax() on the outputs and taking the max to make my predictions, but my network is not doing so good when I am running on unseen data, and I want to make sure the problem is just me overfitting.

Kind of yes... It's log of softmax. Now softmax will always be < 1 and > 0. Apart from the top class, rest would be close to 0. So log of that will be large negative number.

Sounds like it worked to me.
nn.LogSoftmax returns the log of the softmax (duh). The outputs from softmax add up to 1, and form a probability distribution.
0 is the log of 1, meaning that class was predicted at a level of nearly 100%. And the other classes with large negative logs are a rounding error.


text:
Is it possible to apply 2D convolution to 1D data?
Suppose that I have a 1D dataset with 6 features.
Can I apply a 2D convolutional neural net to this dataset?

This depends on the engine you use, but in general, yes, of course.
For example, in the TensorFlow height and width are separate variables, so nothing in your way to set one of them to 1 to have 1D data in it.

You can certainly reshape the data to make it fit a 2D network. You could set the width or height to 1 as suggested by DamirTenishev, but you could also set the features/channels to 1 and treat the features as a height, if you wanted to convolve that way (would be a bit strange).


text:
When should we use CNN instead of MLP?
Is CNN only applicable to time-series data or image data?
When should we use CNN instead of MLP?

A simple answer is when you need a convolution like applying filters in image processing or auto-regression in time series.

CNN applies the same filters to every "chunk" of the input data. It's applicable when you think every chunk should be processed the same way.
For example, we think a face in the top-left of the image should be recognized just as well as a face in the bottom-right. So we do expect to process each part of the image the same way, and a CNN is good for this.
If we did not use a CNN, the network would need to learn separately at each position. It would learn to recognize faces in the top-left based on training examples where faces were in the top-left, and it would learn to recognize faces in the bottom-right based on training examples where faces were in the bottom-right. Because a CNN necessarily uses the same weights on each part of the image, seeing a face anywhere helps it update the shared weights which helps it recognize faces in every part of the image. Therefore, less training data may be needed.
A CNN might not be so good at classifying MNIST digits, for example, for the same reason. A /-shaped line in the top-left probably indicates a 6, but a /-shaped line in the bottom-right is more likely to indicate a 9, so we actually don't want the same weights there. It could still learn to sort this out in its final dense layer, of course. But if you wanted to find digits in a larger image you'd use a CNN.


text:
What problem does the neural network really solve?
In the image below taken from a Youtube video, the author explains that the neural network can be used to fit a relational graph for a set of data points shown by the green line. And that this is accomplished by using weights, biases and activation functions.
My slight confusion is that, initially, the weights and biases and randomized, and they are re-adjusted by backpropagation. This means that, at the end of the output layer, we must have the actual values of the target function anyway.
So what problem does the neural network really solve?
So, for example, we want to find the target function for dosage and efficacy, we are given the data points shown in blue. If we initially choose randomized values for the weights, biases and activation function, then, at the output layer, we determine an output value for efficacy, but there is no way to know whether this value is in fact correct or not. So, we need the actual values to determine the difference.
What about when we choose a value of dosage which has not been observed, for example, 0.25? Doesn't this rely upon a best-fit relation graph that has already been fitted to the data prior to adjusting the neural network?

Hello. Your questions are all legitimate, but, please, next time, ask only one question per post. If you have multiple questions, ask one for each post.

Note that real neural networks typically have lots of input dimensions. With only one input dimension, we can usually just interpolate the known data points.

This means that, at the end of the output layer, we must have the actual values of the target function anyway.

Yes, this is necessary for supervised learning. You will often see this called a labelled dataset, where the "label" is an output value that you know is associated with each input. A set of labels associated with some inputs, that you have collected for training may also the called the "ground truth".
We do not need all possible values though, but enough examples that the neural network can interpolate between them. How many examples that is depends on the complexity of the function we want to learn.

So what problem does the neural network really solve?

There are three main things it solves, and these are shared with most other machine learning approaches:

The neural network learns a function from examples of input and output.

The neural network will learn an expected value (or for classifiers, a probability distribution) when trained using noisy or stochastic data.

The neural network makes few assumptions about the relationship between input and output, and can learn successfully even for quite complex relationships.

These traits are all useful when you do not have a strong sense of what the correct function should be, in terms of writing an equation, but do have many examples of it.

What about when we choose a value of dosage which has not been observed, for example, 0.25?

The neural network will still produce an output. In a very simple scenario, where you had trained with example inputs at e.g. 0.2 and 0.3, then the output will likely be somewhere between the outputs for those two values. For a neural network, this in-between value can be much more sophisticated than a simple mean of the nearest examples. ML that uses the nearest values exists, that is called k-nearest neighbours.
If this process has worked well, and the trained neural network produces useful, accurate predictions from unseen inputs, it is said to generalise well. Very often, that is the goal for training a neural network or other machine learning system.
It is worth mentioning some additional facts and features of training neural networks (and ML in general):

When generalisation is the goal (and it often is), then you need to test for it. This is done by keeping some example data back, not using it to train, but instead using it to check results on unseen inputs. In fact, this is so important, the data is often split into three sets - a training set, a cross-validation (aka development) set, and a test set.

The lack of assumptions in the basic model can lead to needing many training examples. If you know something specific or useful about the function being learned, you can pre-process the inputs to help - this is called feature engineering.

Machine learning can be good at interpolation, i.e. calculating outputs for inputs that are not in the original training data, when the unseen inputs are in-between or close to the training examples. Even when good at interpolation, it will still be bad at extrapolating to new unseen inputs that are outside of the ranges of the training examples. That is because it has used a very general/flexible system to fit some line or curve to examples, it has not learned an analytical function.

Exceptions to all of these points exist. They are the norm, but it will depend on the details of what you are trying to do.

Thank you! appreciate the efforts you've put into your post! Will upvoted :)
Do you have any recommendations for good books, articles? Ive made a note of the ones you provied alreayd

I learned the basics from Coursera courses. https://www.coursera.org/learn/machine-learning is still very good because it doesn't dive into theory too heavily and teaches good practice (test/train split) from the beginning


text:
Is it possible to apply a particular exploration policy for the on-policy RL agents?
Is it possible to use any particular strategy to explore (e.g. metaheuristics) in on-policy algorithms (e.g. in PPO) or is it only possible to define particular policies to explore in off-policy algorithms (e.g. TD3)?

In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C.
However, if you are willing to consider the added exploration strategy as part of the current target policy, and can express it mathematically, you should be able to add an exploration term to on-policy approaches:

For value-based on-policy methods like SARSA, then there is no requirement to base the current policy on the learned value function. However, you will probably want to reduce the influence of the exploration heuristic over time, otherwise the algorithm may not converge. A simple way to do this would be to weight the heuristic for each action, and add it to the current value estimates when deciding the greedy action, slowly decaying the weight of the heuristic down to zero over time.

For policy gradient methods, the adjustment is harder. Your heuristic needs to be introduced under control of a parameter of the policy function, and should be differentiable. You might be able to do this simply and directly, but it will depend on details. For some exploration functions it not be possible at all.

Perhaps a more tractable approach to work with on-policy policy gradient methods would be to pre-train the policy network to approximate the heuristic function. The policy would then start like the heuristic then evolve towards optimal control. This works provided your heuristic outputs a probability distribution that is only dependent on current state, and without additional information such as number of times the same action was already taken.
If you want to explore using a heuristic without decaying it, or losing the exploration in the longer term, but still target an optimal policy, then you must use an off-policy method. Most off-policy methods are extensions of on-policy versions that have been adjusted/extended to deal with a split between behaviour and target policies. They are necessarily more complex as a result.
If you want to use a custom exploration function with policy gradients, then you may have some luck adjusting Deep Deterministic Policy Gradient (DDPG) where the exploration function is already a separate component and could be replaced - there are already a couple of variants in use.

Could you explain a bit more on "reduce the influence of the exploration heuristic over time" (mentioned in the SARSA paragraph)? Is it like decaying the epsilon over time in epsilon-greedy policy?

@Cloudy: Yes I am suggesting some kind of decay or multiplying the heuristc by an "envelope function" that drops from 1 towards 0 over time, e.g. $\frac{1}{\sqrt{t}}$


text:
What, exactly, do mlp(64,64) and mlp(64,128,1024) mean in PointNet, and how many input neurons does 1 (x,y,z) point have?
I couldn't find out how to interpret the multilayer perceptron notation given in PointNet. Specifically, I am looking to find out what the numbers inside the parentheses of mlp(64,64) and mlp(64,128,1024) actually mean.
(I also have a 2nd question about PointNet MLP architecture, which I ask towards the end.)
Here's what I found online, which I believe applies:

https://towardsdatascience.com/deep-learning-on-point-clouds-implementing-pointnet-in-google-colab-1fd65cd3a263
There's a paragraph here that says

In this case MLP with shared weights is just 1-dim convolution with a kernel of size 1.

Here, a link is provided to explain more about the 1-dim convolution...
https://jdhao.github.io/2017/09/29/1by1-convolution-in-cnn/
...and I follow this pretty well.

There's also this Matlab example...
https://www.mathworks.com/help/vision/ug/point-cloud-classification-using-pointnet-deep-learning.html
...which tells me to

set the classifier model input size to 64 and the hidden channel size to 512 and 256 and use the initalizeClassifier helper function...to initialize the model parameters.

inputChannelSize = 64; hiddenChannelSize = [512,256];

Then there's this link: https://www.researchgate.net/figure/Network-architecture-The-numbers-in-parentheses-indicate-number-of-MLP-layers-and_fig2_327068615
...in which they say,

The numbers in parentheses indicate number of MLP
layers

but this is, in my opinion, not written very well. Do they mean,

The notation mlp(64,64,128,256) means that the MLP has 4 layers, and each layer produces an output with 64, 64, 128, and 256 channels, respectively?

Here are my 2 questions about PointNet MLP notation / architecture:

What do each of the numbers in something like mlp(64,64,128,256) actually mean, and what do their positions mean? Are these numbers ONLY referring to the hidden layers, which includes the output layer? Also, are they referring to the number of channels, akin to the depth-wise feature layers of a CNN?

Finally, if your input is nx3 (as in, n (x,y,z) points), does this mean that the PointNet MLP takes an input of 1x3, meaning 1 input neuron, or 3 input neurons?

Hello. This post was created more than 1 week ago, but it may be a good idea to split this post into 2, one for each of your 2 main questions/issues, so that people can focus on one problem at a time, but, in this case, this may not be strictly necessary, as the questions are very related, i.e. about the notation used in the context of PointNets. It's your choice.

The following link satisfied my inquiries:
https://www.mdpi.com/1999-4907/12/2/131/htm

I hope this is useful for someone else!
Justin

useful diagram, had the same question myself. +1


text:
How to find "relationships" between two data representations?
I am a researcher in a field, and new to the whole of AI and machine learning techniques.  May the following question is trivial or not framed in the ML language but I try my best.
I have two sets of representations (I can extract feature vectors, etc., from the datasets) from vastly different domains. I want to find, if any, a relationship exists between these two sets.  In other words, I want an algorithm (the idea of an algorithm) to learn both representations and find the connections and convert one representation to another.
There is neither apparent one-to-one correspondence nor both need to be the same lengths.
Any suggestion on how to approach this problem is appreciated.
I thought of one method; write an encoder-decoder for each of these presentations separately and swap the decoders.  I am not sure whether it works or not, and besides I may not have any idea what's going on there.
I prefer a general approach if it exists.

Have you tried a simple design where you input one representation to an NN and make it predict the other?

You say "There is neither apparent one-to-one correspondence nor both need to be the same lengths", but then I don't understand why you want to convert one representation to the other. What's your ultimate goal or why do you want to do this? Could you also provide an example of 2 vastly different feature vectors that you would want the algorithm to convert from/to, so that we have a better idea of what you really want to achieve?

Yes, I am sorry that the message was kind of cryptic.  I am comparing chemical vibrations to musical notes.   I have a dataset of chemicals, let's say proteins, polymers, crystals etc. and another dataset of musical notes, ragas etc.
I need to find the correspondence or any hidden structural similarities between these two domains.
I can assign one amino acid to one note manually and create some kind of music but that's not what I want.
In the end, the algorithm needs to convert chemicals to music or music to chemicals vice versa.

Well, I suppose one can use some kind of contrastive learning in this case.
A famous example of the establishment of relation between two different representations is the CLIP  - Contrastive Language–Image Pre-training, where model gets a huge corpus of image captions and images and the image caption is passed through the language model, and the image itself through convolutional or ViT backbone, and the model learns to make the image embedding (output of the visual model) to be similar to the output embedding (output of text model) and dissimilar (in the sense of cosine distance) to the embedding of captions, that belong to other images in the training dataset.

In case your two data representations allow for the notion of similar and dissimilar, I expect you can apply a similar procedure.


text:
How to represent "terminate episode" for Knapsack problem with a Pointer Network?
I am currently implementing a Pointer Network to solve a simple Knapsack Problem. However, I am a bit puzzled over the correct (or common, or "best") way to give the agent the option to stop taking the item (terminate episode). Currently, I have done it in 2 ways, adding raw dummy features or adding encoded dummy features (dummy features are all zeros). If the agent selects the dummy item, then the agent will stop taking the item anymore and the episode is terminated.
I trained both methods for 500K episodes and evaluated their performance on a single predefined test case in each episode, after adding the gradient. I found that concatenating dummy features with the encoded features yielded a higher score earlier, but also scored 0 very often. On the other hand, adding the dummy features to the raw features learned to maximize the score very slowly. Therefore, my questions are:

Is adding the raw dummy features make learning slower because of additional encoding layer learning?

What is the most correct (or common or arguably best) way to give the agent the option to terminate the episode (in this case stop taking item)?

I do not think there is one standard way to do this, it will depend too much on context. Ultimately you want the agent to output a stop action that is different from a continue action.
That stop/continue choice could either be part of the existing action encoding, additional data in parallel with the action sequence, or an entirely separate action choice on time steps alternating with the main action choices.
I do not know anything about pointer networks, so not sure what your action coding options are. Your "dummy" object selection seems like a reasonable choice though, it is part of existing action encoding that the neural network can already output. In that sense it seems a lot like an <END> token that a seq2seq model might output for e.g. translation or summarisation language tasks.
An alternative might be to add a separate head to the pointer network, that output a stop classifier alongside each object choice. The combined action would be [selection, stop]. If you are feeding previous choice back into the RNN as input (it is not clear to me, but sampling seq2seq networks do this), you have free choice as to whether to use the combined action (with the stop flag as a raw confidence in $[0,1]$), or just continue with previous selection as only feedback.
Finally you could use a different NN for deciding whether to stop or continue and feed it the same sequence.
Which is better? I cannot say, but your dummy object selection appears to work already to some degree, so I would stick experimenting with that, and the simple all zeroes token. The common 0 total rewards may be an issue with your RL agent exploring, or maybe with a difficult reward metric. E.g. can the agent score less than zero if it overfills the knapsack? If so, not trying anything at all may sometimes look good to the agent, and it will need more training so it can properly predict when this is not the case.

Pointer network is very similar with seq2seq but is designed for problems which input does not have any concise/meaningful order e.g. items in knapsack problems, city coordinates in tsp, nodes' coordinates in convex hull. Therefore, the inputs does not have to be encoded by RNN, but only by simple single/multiple layer perceptron. The choice then are based on the attention on the inputs' encoded features.
Yes, I was hoping to replicate END symbol in seq2seq. I'm interested in the additional head alternative that you mentioned. I will google it first. Thank you very much!

You can find information similar to exposed by Neil, but with more theoretical detail, in the book Deep Learning (Goodfellow et al., 2016) in the chapter 10 (Recurrent networks), more specifically in 10.2.3 Recurrent Networks as Directed Graphical Models and other subchapters.
Additional, related with pointer networks there are people changing the LSTM with Transformer (Learning Heuristics for the TSP by PolicyGradient, 2018)

Transformer for TSP, there are some variants of it, I was just started reading the one proposed by Kool, thank you for the book reference!


text:
Why does $I_N + D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ have eigenvalues in the range [0, 2]?
In Semi-supervised classification with Graph Convolutional Networks, I am unable to understand a few things.
Given an undirected graph having

adjacency matrix $A$,
degree matrix $D_{ii} = \sum_j A_{ij}$,
normalized graph laplacian $L = I_N + D^{-\frac{1}{2}}AD^{-\frac{1}{2}} = U \Lambda U^T$, where $\lambda_{max} \approx 2$ (see page 3, 2nd paragraph, not sure which matrix they are talking about)

Then, $I_N + D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ has eigenvalues in the range [0, 2]. How?

The proof of the first statement can be found in these Lecture Notes.
Have a look of the Proof of the Claim 1.
Concerning the renormalization trick I do not see easy way to justify this statement. The paper claims:

i.e. adding self-loops to the graph, improves accuracy, and we
demonstrate that this method effectively shrinks the graph spectral
domain, resulting in a low-pass-type filter when applied to SGC.


text:
What type of neural network do you need if you want to detect an action or dynamic pattern instead of a static pattern?
Let's say that you want to detect if a man is running, walking, or dancing instead of just detecting a man still. What type of neural networks will you use for this purpose?

It all depends on the kind of data you have and the type of output you expect.
Input:
For example, if you have a series of numbers (heart rate at every minute, or accelerometer data at every second), you can use some time-series approach, like LSTM.
Output / Label
If you want to classify a given segment, you should look for classifiers.

What Bob was doing from 1PM to 2PM?

If you want to find and extract a sub-sample inside a longer period, that's a very different approach.

Find the start and finish time for all running sessions Bob took last week.

In any case, keep in mind the expected prediction format should match the training data format. And some data are more easy to find.


text:
Does it make sense to use BLEU or ROUGE for any machine translation task?
Many machine translation metrics such as BLEU or ROUGE are used to evaluate sequence to sequence models where, usually, the sequences are pieces of natural language.
Is it possible to use these metrics when the dataset is not constituted of natural language sequences? For instance, if the sequences are source code (in some programming language), does it still make sense to use BLEU or ROUGE? How "good" are these metrics in general?

Yes - and no. The important distinction is whether your data contains proper word boundaries and rigorous translation references.
BLEU and ROGUE both work by comparing a candidate (ie, model output) to reference text (ie, training data). In a translation task (what these metrics are typically used for) this works quite well, as you can normally assume the translation will use a small set of words common among all references, so naturally the candidate should be using these words in some order. The more references you have the more allowance the model has to interpret the translation it's own way and still achieve a relatively high score.
However, this does not always work. Both algorithms require word boundaries for them to work properly. You can modify your dataset so that words are properly separated for the algorithms to use, but using your example of source code, this becomes very tricky with indentation and more. Additionally, it's very hard to capture the possible outputs. Not only can a variable be reasonably named hundreds of different ways (making it unlikely an exhaustive reference list exists that would capture your models output), the amount of potentially correct code grows extremely quickly the larger the task becomes, meaning the references become increasingly less useful. If you keep the tasks very short, you could potentially find some use from these algorithms, but better metrics exist.
It's probably best to use these algorithms for what they're meant for - translation. However, there is absolutely no harm in using them in a task anyway, they're very simple algorithms and could still potentially give you some insight.
(For source code evaluation, you would probably be better actually just running the code and seeing if it completes the task, maybe making your own scoring system. Or you could draw inspiration from something such as OpenAI's Codex HumanEval)


text:
Bag of Tricks: n-grams as additional features?
I've been playing with PyTorch's nn.EmbeddingBag for sentence classification for about a month. I've been doing some feature engineering, playing with different tokenizers, etc. I'm just trying to get the best performance out of this simple model as I can. I'm new to NLP, so I figured I should start small.
Today, by chance, I stumbled on this paper Bag of Tricks for Efficient Text Classification, which very well may be the inspiration for nn.EmbeddingBag. Regardless, I read the paper and saw that they increased performance through using "n-grams as additional features to capture some partial information about the local word order"
So by the wording of this sentence, specifically "additional features", I take it to mean that they made n-grams as part of their vocabulary. For example "abc news" is treated as a single word in the vocabulary, and then appended to the training data that is being embedded like so:
dataset = TextFromPandas(tweet_df)
label, sentence, ngrams = dataset[0]
label, sentence, ngrams

# out:

(1,
 'quake our deeds are the reason of this # earthquake may allah forgive us all',
 ['quake our',
  'our deeds',
  'deeds are',
  'are the',
  'the reason',
  'reason of',
  'of this',
  'this #',
  '# earthquake',
  'earthquake may',
  'may allah',
  'allah forgive',
  'forgive us',
  'us all'])

I just wanted to check my assumption, because the paper is not very explicit. I already tried to string n-grams together as a new sentence in place of the old, but performance dropped significantly.
I will continue to experiment, but I was wondering if anyone knows the specific mechanism?

Yes, N-grams is about joining $n$ words as one single token.
Keep in mind it will greatly increase your features size:
If you originally have 1000 unique words, notice you could get up to 1000² 2-gram (usually you don't get ALL the combinations, but notice the number of features can potentially grow huge!)
If your dataset contains between thousands or millions samples, it could be enough to train a simple bag-of-words. But when you use a bi-gram, you'll probably need at least a million samples and lot more training steps. Besides, you'll probably have to tweak the hyperparameters.
That's a common Machine Learning trade-off. A simple model is not very accurate. But a more complex model requires more data, more training and can overfit more easily.

It seems I am constantly facing a lack of data to train the kind of models I'd like to learn.

Welcome to machine learning!


text:
How do CNNs handle inputs of different sizes and shapes?
I am new to deep learning so feel free to correct me where I am wrong.
Imagine this scenario where we have a 7 * 7 input. We want to slide a 3 * 3 filter with a stride of 3 and padding of zero over this input. As you know, it is not possible to do this.
Also, CNNs have a fixed input shape(Correct me if I am wrong) or at least the input should be of the multiples of the CNN's intended input shape(e.g., 112 * 112, 224 * 224, etc)(Although the situation that this may work is rare.)
According to this PyTorch page, ResNet (for example) accepts images of any size as long as they are bigger than 224.
So my question is, how does it handle images of different sizes? Does it dynamically tweak parts of the structure (e.g., kernels, strides, paddings) based on the input? If yes, wouldn't that change the network architecture? Or it changes the input sizes to the intended size automatically?
Also, this does not answer my question.


text:
Why do some techniques use random augmentations during convolution processes
While going over PyTorch image augmentations, https://pytorch.org/vision/stable/transforms.html, I see that some augmentations can be applied with a certain probability. What is the purpose of applying stochastic augmentations rather than consistently applying a certain augmentation?

Stochastic augmentations are used to sample from all possible augmentations, when the dataset size of all possible augmentations would be too large.
This is usually done on-demand, and as a result each training epoch should result in slighty different inputs. This approach can be beneficial in preventing overfitting when the variability in possible inputs is far larger than you could ever properly sample from (e.g. natural images).

Thank you for the answer. I am curious, aren't augmentations anyways used to tackle overfitting. It is quite surprising for me when I see that after augmenting my RL input, it seems to lose its generalization capabilities. Would stochastic augmentations be able to mitigate that?

@desert_ranger Yes, augmentations are a general way of trying to tackle overfitting. Parametric augmentations with on-demand random variations take the idea a bit further. Usually RL does not have the same problem, because it has a built-in generation method (constantly making new observations). That may depend on what you are using RL for, and there are related issues in RL, such as catastrophic forgetting


text:
Is there a benchmark for multi-objective evolutionary algorithms?
I'm working on a project for an evolutionary algorithms course, and the problem we're trying to solve is multi-objective. We'll use NSGA-II but we also wanted to compare with some other MOEAs, however, we haven't been able to find good comparisons/benchmarks of these algorithms, so we don't really know how to decide.
Any insights will be appreciated.

There are like a whole lot of it, try ZDT, WFG, some also have "complicated" Pareto set (I think it's from MOEA/D-DE paper),  look at newer MOEA papers and see the references of the benchmark functions they use.

@Sanyou If you have an answer to the question (which adds something to the existing answers), you should formally write it below.

The DEAP library (a Python library for EAs) contains some benchmarks. In particular, you may want to look at the following functions

Function(s)
Reference/paper

kursawe
?

schaffer_mo
[1]

dtlz1, dtlz2, dtlz3, dtlz4, dtlz5, dtlz6, dtlz7
[2]

fonseca
[3]

poloni
[4]

dent
[5]

Besides the DEAP library repository, you might want to look at several well established MOEA frameworks such as PyMoo (Python) and PlatEMO (MatLab). Both have implementations of well known MOEAs and benchmark functions. You can look through their collection of benchmark functions for inspiration, and also implement your algorithm with those framework so that you can easily test your method's performance on their benchmark functions.
PlatEMO even has a GUI for experimental study, where you can choose the algorithms and the benchmark functions to test, followed by a Wilcoxon’s rank sum test to see how your algorithm really perform compared with other algorithms. For PyMoo you can create a new test file by following the existing examples, but, as far as I know, it doesn't have an experimental study platform similar to PlatEMO yet.
You may also be interested in the paper Scalable and Customizable Benchmark Problems for Many-Objective Optimization (2020).

+1 for mentioning the DEAP library, knowing that there are several well established Multi Objective Optimization libraries, I think the OP can start browsing benchmark function provided in the libraries, besides the obvious benefit of using the library as the platform to implement, evaluate, and compare the OP proposed algorithm with other algorithms.


text:
Do we need automatic hyper-parameter tuning when we have a large enough dataset?
Hyperparameter tuning is the process of selecting the optimal hyperparameters for an ANN.
Now, my guess is that, if we have sufficient data (say, 1.4 million for, say, 6 features), the model can be optimally trained and we don't need a hyperparameter tuner (like Keras-Tuner), because, while training, the data itself will optimize the model.
Do we need a hyperparameter tuner if we have a sufficient number of random data for training our ANN model?

Unfortunately, even with large amounts of training data, hyperparameter choices can strongly influence the performance of a trained model.
What you can usually drop when you have large amounts of training data is regularisation. If your training examples cover the function space you are learning really well, then it is harder to overfit the training data. Regularisation choices are also hyperparameters, so you can save some search space and time by ignoring them.

Do we need a hyperparameter tuner if we have a sufficient number of random data for training our ANN model?

Having lots of data may mean that you can use simpler "brute force" architectures and designs, and that the end result is robust over wider range of hyperparameter choices.
You may still want to tune hyperparameters at least a little though. This tuning can be tedious to drive by manual edits and re-tries, which is where an automated tuner can help.

You don't NEED a hyperparameter tuner, but it can help in various situations.  For example, if your model is not training well, perhaps using a tuner can help.
It's hard to say in which hyperparameters you would be turning over in your specific model, but for some specific hyperparameters if you choose a bad value your model won't learn or diverge.  Take for example the learning rate, if you pick a value too high it will overshoot minimums and the error might constantly grow (divergence), or if you pick a value too low it will get stuck in a local minimum and not be able to continue learning. You can have the world's largest dataset, but if you don't pick the correct range for your learning rate, the model will not properly learn.
In general, if you're not confident about specific hyperparameters ranges, then hyperparameter tuning can be a helpful tool, regardless of your dataset size


text:
How do I choose the hyper-parameters for a model to detect different guitar chords?
I need to build a hand detector that recognizes the chord played by a hand on a guitar.
I read this article Static Hand Gesture Recognition using Convolutional Neural Network with Data Augmentation that looks like what I need (hand gesture recognition).
I think my task is (from my point of view) a little more difficult than that in the paper, because I think it is more difficult to distinguish between two chords than between a punch and a palm.
What I don't understand clearly is how to choose the best parameters for this more complex task: is it better to have more/less convolutional layers? A higher or lower number of poolings? Max or avg pooling?
The input will be more or less like this one:

There will be a first net (MobileNetV2 trained on EgoHands) that will find the bounding box, crops the image and then passes the saturated blending between the original one and Frei&Chen edges to the second net (unfortunately I don't have a processed picture yet, I will post an example as soon as I get it)

there is no way to compute analytically what the best parameters are. It is also not trivial to conclude that the task is more complex simply cause for "us" is more difficult to distinguish between 2 chords than palm and punch. It all depends on the data you have. Any concrete picture to share with? Or idea on how you would collect the data in terms of camera, position, lighting, angle?

The ideal hyperparameters is usually dependent on your dataset and will differ on a case by case basis. Go for trial and error to determine the hyperparameters that works best for you.
Few research papers similar to your use case is listed below.

CNN transfer learning for visual guitar chord classification
A Study of Left Fingering Detection Using CNN for Guitar Learning
A 3D Guitar Fingering Assessing System Based on CNN-Hand
Pose Estimation and SVR-Assessment
Applying Deep Learning Techniques to Estimate Patterns of Musical Gesture : forearm gestures on violin playing


text:
When to use Value Iteration vs. Policy Iteration
Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms.  However, they differ in the mechanics of their updates.  Policy Iteration seeks to first find a completed value function for a policy, then derive the Q function from this and improve the policy greedily from this Q.  Meanwhile, Value Iteration uses a truncated V function to then obtain Q updates, only returning the policy once V has converged.
What are the inherent advantages of using one over the other in a practical setting?

I think there's a typo here "V function to then obtain Q updates". What do you mean by "Q updates"? Maybe you mean "to obtain policy updates".

Value Iteration Vs Policy Iteration
Below is the list of differences & similarities between value iteration and policy iteration
Differences

Value Iteration
Policy Iteration

Architecture
VI Architecture reference
 PI Architecture Reference

Execution starts with a
random value function
random policy

Algorithm
simpler
complex

Computation costs
more expensive
cheaper

Execution time
Slower
Faster

No of Iterations to converge
significantly more
takes fewer iteration to converge

Guaranteed to converge
Yes
Yes

Similarities

both are dynamic programming algorithms
both employ variations of Bellman updates
Both exploit one-step look-ahead
Both algorithms are guaranteed to converge to an optimal policy in the end

Policy iteration is reported to conclude faster than value iteration
USAGE PREFERENCE
As mentioned earlier in the difference, the main advantage for using Policy iteration over value iteration is its ability to conclude faster with fewer iterations thereby reducing its computation costs and execution time.
REFERENCES

Research papers

Value iteration and policy iteration
algorithms
for Markov decision problem
On the Complexity of Policy Iteration

Book

Artificial Intelligence: A Modern Approach, by Peter Norvig and Stuart J. Russell Chapter 17 Making Complex decisions

Architecture

VI Architecture reference
PI Architecture Reference

The presentation/readability of this answer is good, but this does not answer the question "When to use one over the other?". Moreover, you do not explain 1. why value iteration is slower and more expensive than policy iteration and 2. who claimed that (i.e. cite something reliable). In any case, I don't understand why you say that policy iteration is "complex". Both algorithms are simple enough.

I added the notice because you're not really citing anything that supports the claim e.g. that policy iteration "takes fewer iteration to converge". Provide inline references/citations to research papers or books that support those claims (e.g. you can use this mathjax style [\[1\][1]] and it will be shown as [1]. Moreover, your last edit actually made things worse. For example, the diagram for value iteration is wrong. You're confused. Please, do not provide wrong information to the site. If you're not reasonably familiar with the concepts, don't provide an answer just by guessing.

@nbro: The claim that policy iteration is "complex" is made, though not explained, in this tutorial, on which this answer appears to be very closely based.

@Scortchi-ReinstateMonica Yes, thanks. It looks like this answer is based on that tutorial. The initial table provided by the OP is really analogous to the one given in that tutorial. This answer is now more misleading than originally. In the last edit, the OP provided the diagram of "Value Iteration Network", which is not "value iteration". We may need to delete this answer to avoid spreading misinformation, although some info in this answer may be correct. More importantly, this does not even answer the question "when to use one over the other?". Not sure why it was accepted.


text:
Is there any recommended way to perform pooling in this context?
Suppose I have three batches of feature maps, each of size $180 \times 100 \times 100$. I want to concatenate all these feature maps channel-wise, and then resize them into a single feature map. The batch size is equal to 10.
Consider the following code in PyTorch
import torch
from torch import nn

x1 = torch.randn(10, 180, 100, 100)
x2 = torch.randn(10, 180, 100, 100)
x3 = torch.randn(10, 180, 100, 100)

pool1 = nn.AvgPool3d(kernel_size = (361, 1, 1), stride= 1)
pool2 = nn.AvgPool3d(kernel_size = 1, stride= (3, 1, 1))

final_1_x = pool1(torch.cat((x1, x2, x3), 1))
final_2_x = pool2(torch.cat((x1, x2, x3), 1))

print(final_1_x.shape)
print(final_2_x.shape)

and its output is
torch.Size([10, 180, 100, 100])
torch.Size([10, 180, 100, 100])

You can observe that both types of polling I did are able to give a feature map of the desired size. But the first one takes a large amount of time with unsatisfactory results and the second one ignores many values in the input feature maps. I don't know whether it is okay to ignore or not.
I want to know the recommended way to perform polling in order to get the desired size of feature maps. Is there any such recommended way to perform pooling?

Why do you want to pool over channels? That's an unusual choice, because channels do not usually have any sequence or metric that would make the operation correspond to something useful

@NeilSlater That is the reason for asking this question. I want to achieve that without some parametric model, but need to capture information.

This one is a bit crazy:
pool1 = nn.AvgPool3d(kernel_size = (361, 1, 1), stride= 1)

because it averages large numbers of the features at once. Very little information about individual features will remain after doing that.
The most obvious one you have not tried is this:
pool3 = nn.AvgPool3d(kernel_size = (3, 1, 1), stride= (3, 1, 1))

which includes all the feature data, and does not try to average over large amounts of it at a time. I expect pool3 to perform better than pool1 in terms of speed, and better than pool2 in terms of metrics for the trained CNN.
If your goal is to reduce number of feature maps from 540 to 180, then pooling is not usually a good choice of operation. The motivation behind pooling assumes that there is some consistent metric space that it is pooling over - and this is used to decide values of size and stride. The sequence of channels will not usually have such a metric, it is usually arbitrary result of learning.
Instead, the usual way to reduce number of channels between layers is to add a new convolution layer, with the desired number of output channels. In this scenario it is also common to use a kernel size of 1. This adds learnable parameters to the CNN, and those will learn an optimal compression of the channels for your problem.


text:
Does the ANN's training data include the proper output for every neuron?
I was designing an Artificial Neural Network a while back, but hit a bump when I got to the backpropagation. I was having trouble making the script choose whether to add or subtract from the weights, when I had a thought.
Does the ANN's training data include the proper output of all the neurons, or just the input and output layers? If it's the latter, could somebody please explain how backpropagation works in a simple way?

watch the 3blue1brown vids on what is neural net, personally it as simple and as intuitive as it can be.

Does the ANN's training data include the proper output for every neuron?

The short answer is: no (not usually or directly).
The long answer is that you can train neural networks in different ways. There's supervised, unsupervised, reinforcement learning/training, or even other ways (e.g. online learning).
The most common way of training neural networks is probably in a supervised (and offline) way. In this case, you typically have a dataset of input-output pairs of the form $D = \{(x_1, y_1), \dots, (x_N, y_N) \}$, where we assume that there's an unknown function $f$ such that $f(x_i) = y_i$, so $x_i$ and $y_i$ are the input and corresponding output of $f$. In this case, we want our neural network, which we can denote by $g_{\theta}$ (where $\theta$ are its weights/parameters), to approximate $f$, i.e. $g_\theta \approx f$, for every input and output of $f$. Unfortunately, $D$ does not usually contain all input-output pairs of $f$ (if we had access to all input-output pairs, we wouldn't even need machine learning), so that's why we can only approximate $f$ with our neural network $g_\theta$ (there's also the aspect that $D$ could be noisy, but you can ignore this for now).
So, in this supervised learning context, $y_i$ (also known as the labels or targets) are the supervisory signals to teach the neural network what it's supposed to produce for $x_i$. So, let's say the neural network produces $\hat{y}_i$ rather than $y_i$ when given $x_i$, i.e. $g_\theta(x_i) = \hat{y}_i$. In that case, we have an "error" of $\hat{y}_i - y_i$, but you could also compute the error in other forms other than the difference (depending on the nature of the targets/labels), for example,$|\hat{y}_i - y_i |$ or $(\hat{y}_i - y_i)^2$.
So, once we know the error, we should change the weights of the neural network, i.e. $\theta$, so that the error is smaller the next time we feed $x_i$ as input to $g_\theta$. The way we usually do that is with back-propagation.
You can only understand back-propagation if you understand derivatives and, in particular, how to take derivatives of functions with respect to multiple parameters. Essentially, back-propagation is really just an automatic way of computing derivatives. So, to understand it, you will first need to study a little bit of calculus. This answer is already quite long, so I will not provide more details here, but there are many videos online that explain (even just intuitively) how back-propagation works. So, you should watch them. If you're already familiar with derivatives, maybe this article may be useful (I read it when I was first learning about the topic, but I had already a good knowledge of basic calculus before doing that).