text:
How does MAML inner loop optimization works?
I started to learn meta-learning, reading the MAML paper https://arxiv.org/pdf/1703.03400.pdf

In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of inner SGD.
I will calculate adapted parameters after two or more gradient steps ($\theta{'}$, $\theta^{''}$ , $\theta^{n}$), then using testing parameters, I will have a loss in respect to the original $\theta$ (If I understand the derivation correctly). Now I am supposed to backpropagate through the gradient. Unfortunately, I am not sure how it is done...

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Page 3.
 "The MAML meta-gradient update involves a gradient
through a gradient"

To do this, I understand that I have to store each $\theta^{n}$, but don't know how the loss from $\theta^{n}$ to $\theta^{n-1}$ is transferred up to the original  $\theta$.
I guess that the for the last series of parameters ($\theta^{n}$) loss is calculated in the standard way with the testing set, but then I somehow need to pass information saying how much the previous set of parameters was wrong... (Gradient of the gradient?)
I see Hessian and vector products popping up everywhere on the internet, but I cannot imagine how that works, and have no idea how it is calculated and passed/implemented...
Can someone explain to me how the inner loop [back-propagation trough meta-gradient] is working - how the derivations go and how loss is transferred/updated?

I see too many questions in this post. Edit your post to leave only one specific question.

Hi, sorry for the confusion, I have clarified the question, and I hope that it is now more readable.

According to: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" it is initialized in Class MAML:
class MAML:
    def __init__(self, dim_input=1, dim_output=1, test_num_updates=5):
        """ must call construct_model() after initializing MAML! """
        self.dim_input = dim_input
        self.dim_output = dim_output
        self.update_lr = FLAGS.update_lr
        self.meta_lr = tf.placeholder_with_default(FLAGS.meta_lr, ())
        self.classification = False
        self.test_num_updates = test_num_updates
        if FLAGS.datasource == 'sinusoid':
            self.dim_hidden = [40, 40]
            self.loss_func = mse
            self.forward = self.forward_fc
            self.construct_weights = self.construct_fc_weights
        elif FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
            self.loss_func = xent
            self.classification = True
            if FLAGS.conv:
                self.dim_hidden = FLAGS.num_filters
                self.forward = self.forward_conv
                self.construct_weights = self.construct_conv_weights
            else:
                self.dim_hidden = [256, 128, 64, 64]
                self.forward=self.forward_fc
                self.construct_weights = self.construct_fc_weights
            if FLAGS.datasource == 'miniimagenet':
                self.channels = 3
            else:
                self.channels = 1
            self.img_size = int(np.sqrt(self.dim_input/self.channels))
        else:
            raise ValueError('Unrecognized data source.')

"How does MAML inner loop optimization work?"

Where Rectified Linear Unit (ReLU) neural networks are locally almost linear (Goodfellow et al.,
2015), and second derivatives close
to zero, using a first-order approximation removes the need for computing Hessian-vector products in an additional backward pass.
It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in "Understanding the difficulty of training deep feedforward neural networks", Xavier Glorot and Yoshua Bengio, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.
A method to alleviate the risk of overfitting for gradient-based meta-learning is explained in: "Regularizing Meta-Learning via Gradient Dropout" - April 13 2020, by Tseng, Chen, Tsai, Liu, Lin, and Yang:

"[they] aim to find model parameters that
are sensitive to changes in the task, such that small changes
in the parameters will produce large improvements on the
loss function of any task drawn from $p(T)$, when altered in
the direction of the gradient of that loss.".

Combined, those methods shorten the time taken to complete each pass.
In the line:
FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
            self.loss_func = xent

See pages 6 and 7:

"A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient
at the post-update parameter values $θ^
0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the
improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
$$\theta \leftarrow \theta \; - \beta \nabla_\theta \sum_{\mathcal{T}_i\thicksim p (\mathcal{T})} \mathcal{L_{T_i}}(\mathcal{f}_{\theta^{{'}}_i}) \tag{1}
$$

Hi, thank you for the answer, unfortunately, I am not sure that I understand. In the code, you have sent me I can see that loss for each set of parameters in the inner gradient is passed to the outer loop however I still don't understand how it is backpropagated to the original $\theta$ . Derivations are handled by TensorFlow (I guess that it uses some clever tricks to avoid the Hessian) but I don't understand the principle...

@GrumpyC, that is described separately for omniglot and miniimagenet on pages 6&7 of the paper you cited; using a first-order approximation.

A different paper than the one you mentioned: "ES-MAML: Simple Hessian-Free Meta Learning" is clearer, backpropagation isn't used.

The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi-layer neural networks. Actually, back-propagation refers only to one method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient.
— Page 204, Deep Learning, 2016.
It would be fair to say that a neural network is trained or learns using Stochastic Gradient Descent as a shorthand, as it is assumed that the back-propagation algorithm is used to calculate gradients as part of the optimization procedure.

For a simplified explanation of the activation function see here: https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/ and https://en.wikipedia.org/wiki/Activation_function


text:
Why did MYCIN fail?
I've been reading about expert systems and started reading about MYCIN.
I was astonished to find that MYCIN diagnosed patients better than the infectious diseases physicians.
http://www.aaaipress.org/Classic/Buchanan/Buchanan33.pdf
Since, it had such a good success rate, why did it fail?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the difference between an episode, a trajectory and a rollout?
I often see the terms episode, trajectory, and rollout to refer to basically the same thing, a list of (state, action, rewards). Are there any concrete differences between the terms or can they be used interchangeably?
In the following paragraphs, I'll summarize my current slightly vague understanding of the terms. Please point any inaccuracy or missing details in my definitions.
I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s, a, r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.
With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.
I think rollout is somewhere in between since I commonly see it used to refer to a sampled sequence of $(s, a, r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.

I don't really think there are fixed, different definitions for all those terms that everyone agrees upon. In most contexts, they're going to be quite interchangeable, and if anyone is really using them in a context where they are supposed to have crucially important, different meanings, they should probably precisely define them right there.

I think episode has a more specific definition in that it begins with an initial state and finishes with a terminal state, where the definition of whether or not a state is initial or terminal is given by the definition of the MDP. Also, I understand an episode as a sequence of $(s,a,r)$ sampled by interacting with the environment following a particular policy, so it should have a non-zero probability of occurring in the exact same order.

Agreed with this.

With trajectory, the meaning is not as clear to me, but I believe a trajectory could represent only part of an episode and maybe the tuples could also be in an arbitrary order; even if getting such sequence by interacting with the environment has zero probability, it'd be ok, because we could say that such trajectory has zero probability of occurring.

I can't really think of cases where it's sensible to talk about trajectories with tuples shuffled into an arbitrary order. I'd still think of trajectories as having to be in the "correct" order in which they were experienced. But I do agree that trajectories can be little samples (for instance, little sequences of experience that we store in an experience replay buffer). So, every full episode would be a (long) trajectory, but not every trajectory is a full episode (a trajectory can just be a small part of an episode).

I think rollout is somewhere in between since I commonly see it used to refer to a sampled sequence of $(s, a,r)$ from interacting with the environment under a given policy, but it might be only a segment of the episode, or even a segment of a continuing task, where it doesn't even make sense to talk about episodes.

I'd say that... often a rollout should have a "terminal" state as ending, but maybe not a true "initial" state of an episode as the start. We might be in the middle of an episode, and then say that we "roll out", which to me implies that we keep going until the end of an episode. I don't think this term is as common as the other two in Reinforcement Learning, but more common in search/planning literature (in particular, Monte Caro Tree Search).
That said, when I'm working with MCTS I often like to put a limit on my rollouts where I cut them off if no terminal state was reached yet... so that isn't exactly a crisp definition either.
Due to how commonly-used this term is specifically in MCTS, and other Monte-Carlo-based algorithms, I also associate a greater degree of randomness with the term "rollout". When I hear "episode" or "trajectory", I can envision a highly sophisticated, "intelligent" policy being used to select actions, but when I hear "rollout" I am inclined to think of a greater degree of randomness being incorporated in the action selection (maybe uniformly random, or maybe with some cheap-to-compute, simple policy for biasing away from uniformity). Again, that's really just an association I have in my mind with the term and not a crisp definition.

So is it okay to say that all three of them represent a sequence of (s, a, r) tuples, where an episode is generally from an initial state to a terminal state, a rollout is from an initial or intermediate state to a terminal state, and a trajectory is between any two states?

@nikhilweee That's how I (and I guess many people familiar with RL literature) would generally interpret them if no further clarification were given, yes.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Implementation of PPO - Value Loss not converging, return plateauing
Copy from my Reddit post:
(Sorry if this does not fit here, please tell me and I delete it)
Help regarding
I'm working on an implementation of PPO, which I plan to use in my (Bachelors) Thesis. To test whether my implementation works, I want to use the LunarLanderContinuous-v2 Environment. Now my implementation seems to work just fine, but plateaus much too early - At an average reward of ~ -1.8 reward per timestep, where the goal should be somewhere around ~ +2,5 reward per timestep. As the implementation generally learns I am somewhat confused, as to why it then plateaus so early.
Some details regarding my implementation, also here is the GitHub repo:

I use parallelized environments via openAi's subproc_vecenv

I use the Actor Critic Version of PPO

I use Generalized Advantage Estimation as my Advantage term

I only use finished runs (every run used in training has reached a terminal state)

Even though Critic loss in the graphic below looks small it is actually rather large, as the rewards are normalized and therefore the value targets are actually rather small

The Critic seemingly predicts a value independent of the state it is fed - that is it predicts for every state just the average over all the values. That seems like harsh underfitting, which is weird as the network is already rather large for the problem in my opinion. But this seems to be the most likely cause for the problem in my opinion.
Edit1: Added image

Interesting question. Did you succeed? Is the code available somewhere? I have similar issues...


text:
Train, Validation and Test Split for Reporting Accuracy of Neural Model and BOW
I need to report the accuracies of my neural model in a conference paper as compared to various baselines. What are the accepted standards for reporting accuracies in a fair manner?
Neural Model:
To be specific, I'm using 60% as train set, 20% as validation set and 20% as test set to report the accuracy of my neural model.

Should I take an average or highest of 3 runs accuracy where in each
run I randomly sample 60% as training data from the total 80% train + validation set.
My neural model is computationally intensive and therefore it is not feasible to perform k-fold cross-validation. Will my accuracy results be accepted by the academic community without k-fold cross-validation? Since my data set is large, I assume using 20% of it used solely for testing should be a fair indicator of accuracy.

Bag-of-words (BOW):

How do I report the accuracy of this model so as to perform a fair comparison?

Should I train BOW on only 60% of data (same as which my neural
model is being trained on) or should I train BOW on 80% of data
(train + validation for my neural model)? Which is the accepted way?
I then test BOW on the same remaining 20% of data (as in neural
model) in either of the above case.

The other approach is to perform k-fold cross validation but the test set will not be the same 20% as the test set on which my neural model is being evaluated. Is this approach recommended though?

Any other information on how to report accuracy results in a research paper comparing neural models (train, val, test) with linear models-BOW, SVM (train, test) is welcome.


text:
Agent exploration which leads to a negative state where actions are limited
I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation that allows the agent to explore for one year. I then examine the results of the agent's performance exploiting its optimal policy for the following year. The agent can perform the following actions (available actions depend on the state of the environment):

Turn the electrical heating element on.
Turn the electrical heating element off.
Turn gas heating on.
Turn gas heating off.
Do nothing.

The goal of the agent reach the target temperature (50 deg C) when hot water is scheduled. The agent is rewarded for choosing actions that produce the lowest CO2 emissions (the CO2 emissions produced from electricity vary over time).
One of the issues I have noticed is that during the exploration phase, the agent tries a lot of weighted random actions which causes the water heater to overheat (>80 deg C). When the water heater overheats, it is not possible for the agent to perform further actions other than switching off the heating and doing nothing. The agent is also punished for reaching the overheating tank state. The tank may remain in the overheated state for some time. It seems as if the tendency to overheat the tank during exploration is negatively impacting how the agent learns its policy as it reduces the number of experiences in other states.
Is there a term for this kind of situation during exploration in reinforcement learning? During exploration, the agent uses a chooses a softmax weighted random action. Are there alternative ways of choosing actions that may still allow for exploration while not reaching the overheating state?

what is that you want the agent to learn? isn't switching off/doing nothing when the desired temperature is attained the end goal?

@okkhoy I want the agent to learn how to perform a set of actions which leads to the tank being at the target temperature when hot water is required. The problem is that during the exploration phase, the agent spends far more time in the negative overheating state. Once the water heater overheats, it can take some time for it to cool down to a point where the agent can perform other actions/begin to reach the target temperature.
I've updated my post to clarify the goal of the agent.

@Darkphenom Your project concept scenario does not require an agent,it's just require you to re-invent the wheel...in this case you don't require an agent..I did that project in c++[embedded systems programming]

@Darkphenom the way I see it, during the exploration phase, learning to do nothing when it has over heated is also a part of the learning process. if the tank got over heated, then the agent should know it should allow it to cool down. this way not doing anything is also useful, no? is there some detail missing which I am not grasping?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/3.0/


text:
RL to generate sentences
I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence.
Eg:

Input: capital, Paris, France

output: Paris is the capital of France

Input: cute, cat

output: cats are cute

The system adds the missing words such as 'is', 'as', 'are', 'the', 'of', etc.
How can I build a system like this? My gut feeling is it can be done through Reinforcement Learning by training on a huge corpus like Wikipedia.
So the states are the individual input words. The reward would be 1 when the sentence is correct and 0 when not. The actions available are taking an individual word available from the input and attaching it to the connecting word (is, of, the..). Then in the second step take the generated word and pick another word from the input and connect it and so on. Stop when all the input words have been used. It's a win when the final sentence is grammatically correct, else fail.
Ultimately, what I imagine is there will be a knowledge graph. The user asks some questions. Navigating through the knowledge graph, the system will generate some keywords. Then the RL system will take those keywords and construct a human-like sentence.
I'm totally new to RL. I just finished watching David Silver's 10 part course on RL on YouTube. Any guidance on this topic is much appreciated.

DeepCoder (https://openreview.net/pdf?id=ByldLrqlx) uses RL to generate computer code. Its different from natural language, I agree, but it should give you a headstart.

@SouradeepNanda: Where does DeepCoder user RL? I can see it uses a deep NN and SMT solver . . . but nothing about any RL technique. The term  q(a | E), appears, but I don't think that is an action value?

Sorry, I confused with this one. My bad. https://arxiv.org/abs/1706.05125


text:
How to feed a variable size sequences into a CNN?
If I want to train a convoluted NN on time series but I cannot decide where to split the data.
I see that other people use a jumping window over the input. so the feed say 20 sec of observation as 1 sequence into a CNN.
I cannot do that as splitting the observation by fixed size will most definitely break important patterns - ie first part of a pattern will go into the end of the current seq and the rest into the beginning of the next seq.
I can however find a sensible solution by preprocessing data and finding places of much smaller significance and making seq cutting in the middle. but then the seq length will vary greatly.
Can I still use CNN? or this idea is silly?
How else do people extract features from time series using CNNs?


text:
Can neural networks be used to study (elementary) number theoretic problems?
Can neural networks be used to study (elementary) number theoretic problems? What are examples where this has been done in the past? Or is there on the contrary an understanding that neural networks are not helpful for such problems?
To make the question more concrete, let me give an example of the kind of number-theoretic problem I'm thinking of: given two natural numbers a and b I may want to compute the rounded down quotient int(a/b). Naively I would restrict to 64-bit unsigned numbers and build a neural network that has 128 neurons in the input layer and 64 neurons in the output layer representing the binary expansion of the numbers. Assuming I laid out the network properly and trained it well, would I be expected to get useful output? In particular, would I be able to interpret the output as a number and would it often be the right answer?
Note: the reason why I think of this as a "number theoretic" problem is because I want to compute int(a/b) rather than the rational number a/b. This is essentially a step in Euclid's algorithm. So the non-linear behaviour int(4/3) = 1, int(5/3) = 1, int(6/3) = 2 is crucial and would need to be recognizable in the output.

Hi, did you find an answer? I'm interested as well

No, I didn't find a formal answer. Heuristically my impression is that neural networks would perform rather badly.

I'd bet that you can gain nothing as number theory is rather abstract and you'd need a system working with formulas in order to gain any useful knowledge. However, your integer divisor problem can definitely be learned as it's a final optimization problem. What's unclear is the number of neurons needed - you probably don't want to use 2**128 of them. AFAIK no CPU has a single-step 64:64 bit divider, so you most probably have to use some feedback network, too.


text:
Can a CNN or MLP discover similar but untrained-on patterns?
I've been experimenting with a simple tic-tac-toe game to learn neural network programming (MLP and CNNs) with good results. I train the networks on board positions and the best moves and the network is able to learn and correctly predict the best moves to make when it encounters those board positions.
But the network is unable to "discover" newer patterns/features from existing ones. For example -
Let's say that the board position is below and the move is for the X player (AI)
O  _  _

_  O  _

_  _  _

The recommended move would be 8 (0 based indices) so that the opponent doesn't win, the resulting board would be -
O  _  _

_  O  _

_  _  X

If I train the network on the above enough times, the AI (MLP or CNN based) learns to play 8 when it encounters the above situation.
But it doesn't recognize the below as variations (rotated and shifted, respectively but slanted straight lines in general) of the same pattern and is not able to correctly pick 6 and 0, respectively -
_  _  O            _  _  _

_  O  _     or     _  O  _   etc

_  _  _            _  _  O

My question is - Should I expect CNNs to be able to discover new previously untrained patterns/features such as above?

How do you train the NNs?

In this case, training is done on board states as inputs and the best moves (as 9x1 one-hot encoded matrix) for those board states as outputs

And then tf.reduce_mean on tf.nn.softmax_cross_entropy_with_logits optimized with Adam. Works great with known board states, not so much on similar but not-trained on ones.

Not being an expert, just my 2 cents. It should be possible if you gave the NN enough patterns, for it lo learn the few missing ones. A suitable CNN should be better because of the board being 2D.

I've run it through hundreds of thousands (different combinations of layers and depth etc) but it never seems to be able to learn the missing ones so far. I'm beginning to suspect that this problem, counterintuitively, is too simple for it to be able to solve but I'm curious if it has worked for someone.

You may have to increase the training set/test set ratio (if I have guessed correctly what you are trying to do - train the network on existing patterns and expect some new patterns to be learned). I have no idea if a classical MLP is able to do this, but a CNN should (though CNN is a general term), because you basically have image pattern recognition task. I do not think there is such thing as problem too simple to be solved.


text:
Could we have a phone control itself?
A phone can capture an image that lies on the front of the screen. It is also possible to manipulate the input of the touchscreen using various programs and external devices.
If we combine these two elements of technology together, it's possible to have the phone aware of what's happening on the screen and do dictated commands.
This could be extremely useful to those who want to play a mobile game without the grinding effort, or those who do menial tasks on the phone but don't want to spend the time navigating across the phone.
For example, a phone could automatically open up an app on its own, press buttons that consistently appear on a screen for the daily log-in bonuses, auto-mode, etc, and collect the rewards for playing that app, without a human wasting time.
Such behavior could even be recognized by the phone's memory, and perhaps linked to AI (like Google Now) so that the behaviors could be remotely activated on command, or on a timer.
Now my question is - do we have apps that are able to recognize what happens on screen and provide self-input from the background?


text:
Is there a measure of AI relative strength, modified by resources?
For instance, Strength/Size$\times$Speed, where size and speed refer to memory and processing.
We now have very strong, narrow AI, but they tend to run on fast hardware without volume restrictions.
To understand why I'm asking, this article on BBC may provide some insight: "Which life form dominates Earth?"  (If I was a betting man, I'd put money on tardigrades outlasting humans, and the secret of their success is that they require minimal resources and processing power, unlike higher-order automata.)

Can you just give us a glimpse of what that AI is? is it a game,a living creature?

@quintumnia They're currently in the very early stage of development, but the scope of the first stage automata is described here: Epimetheus. The game complexity will ultimately range from < Chess to > Go, where the immediate, practical need is for local, constrained, weak, general AI. (The model is a fundamental economic "metagame", based on a small set of combinatorial mechanics with an alarming range of extensions, i.e. the product.)  Right now I'm just trying to get a gauge on similar initiatives, since I'm still new to the field.


text:
Number of input variables for a cellular automaton (was: Squares or hexagonal?)
A cellular automaton is a state machine that is controlled by external input. The input is given by geometrical space around a cell. In a square matrix, each automaton gets input from 4 surrounding cells, while a hexagon grid has 6 neighbor cells that can be used as automaton input. For example, a 4-cells input may be the string “1011”. This string specifies a state of the cellular automaton. The automaton will switch to a different state according to the lookup table. I want to know if increasing the number of input cells in a hexagon automaton will make the resulting computer more powerful.
original message
I'd like to learn more about the differences between related automata which can be based on hexagonal cells instead of squares (rule 34/2), like in CoDi model which uses spiking neural network (SNN).
Is using a plane tiled with regular hexagons more efficient and reliable than using square cells? What is the difference and how do I know which one to use in which scenario?

In other words, the more efficiently flexible that it grows, the more difficult scenarios it can be used for (for me, hexagonal implicates more possibilities, because it can send/share the signal with/to more tiles). Or maybe one is more modern than the other, or they're both on the same level? In general, I'd like to learn the differences between them to know when I should use one over the other.

This seems more suited to Data Science, though I could be convinced otherwise. It's being discussed on meta.

No, I think this question would be closed on Data Science. It might be on topic in CSTheory, and I think it could be on topic here if we wanted it to be. I would be surprised if you could find anyone who could answer this question in more detail than "there are more sides". I think this is the kind of thing that has long since been abandoned in research.
Cellular Automata are to the best of my knowledge not used anywhere for a practical application, so measuring their efficiency is not a well defined idea.

Interesting question! One thing that is known is that for example for games programming for Ai versus humans, hexagonal is preferred, because the Ai performs in a more interesting way. Does that mean it performs more efficiently? Well, it's also more susceptible to noise, or human bluffing of it...

@GuidoJorg I believe that should be an answer.

Clearly it raises complexity, but, from some conversations with some skilled Go-variant designers, hexagons bring their own set of design problems.  (imo, this makes them especially worth pursuing.)

Due to the hexagonal arrangement, each cell in a hexagonal automaton has six neighboring cells, which can provide more complex interactions. This increased connectivity and symmetry can lead to more diverse and intricate patterns. Hexagonal cellular automata are also known to be more efficient in simulating certain natural phenomena, such as fluid flow and crystal growth.


text:
Can DeepDream produce a "dream" from 3 images?
By default using the DeepDream technique, you can create a dreamlike image out of two different images.
Is it possible to easily enhance this technique to generate one image out of three?

Absolutely. The convolutional neural network behind DeepDream can be trained using how many input/sample images you wish. More patterns in the training set determine a more pareidolic output.

@Lovecraft please put this as an answer.


text:
What would the Valkyrie AI robot do on Mars?
I was reading that the Valkyrie robot was originally designed to 'carry out search and rescue missions'.
However, there were some talks to send it to Mars to assist astronauts.
What kind of specific trainings or tasks are planned for 'him' to be able to carry on its own?
Refs:

NASA-JSC-Robotics at GitHub
github.io page
gitlab page

hopefully, it isn't an outdated question now. But, there are three main high-level things that an autonomous agent can do on Mars alongside humans.

Path planning and obstacle avoidance - Since any territorial patterns on Mars don't really exist in any human head but on hard drives, an autonomous agent can effectively map out the surroundings and help in navigating optimally avoiding any unnecessary complications like surface irregularities.

Physical Strength - While you would be almost 3 times stronger on Mars (seemingly, due to the difference in surface gravity), you will still be limited by the things you can do. An autonomous agent can work tirelessly assembling machines (that you brought from Earth), grabbing stuff for you when you are assembling machines, etc.

Optimizing work - An autonomous agent, given that it is equipped with the right set of tools, can analyze the environment and pre-plan the decisions for you. This essentially means, chunking time for achieving optimal performance for the mission, e.g. exploration, experimentation, rest, etc.

Above all, you can leave the autonomous agent back on Mars so that they can work while you plan a trip to Earth.


text:
How should I deal with variable-length inputs for neural networks?
I am a very beginner in the field of AI. I am basically a Pharma Professional without much coding experience. I use GUI-based tools for the neural network.
I am trying to develop an ANN that receives as input a protein sequence and produces as output a drug molecule. Drug molecules can be represented as fixed-length binary (0-1). This length is 881 bits.
However, I do not know how to transform protein sequences of variable length into a fixed-length binary representation.
So, how should I deal with variable-length inputs for a neural network? What is the best way?

Hi and welcome to AI! Maybe you should explain more in detail how the protein sequences look like, and maybe you should provide examples of variable-length protein sequences.

The most common way people deal with inputs of varying length is padding.
You first define the desired sequence length, i.e. the input length you want your model to have. Then any sequences with a shorter length than this are padded either with zeros or with special characters so that they reach the desired length. If an input is larger than your desired length, usually you'd split it into multiple inputs.


text:
Do all filters of the same convolutional layer need to have the same dimensions and stride?
In Convolutional Neural Networks, do all filters of the same convolutional layer need to have the same dimensions and stride?
If they don't, then it would seem the channel produced by each filter would have different sizes. Or is there some way to get around that?

You could pick the stride and dimensions accordingly so even though they are different, the output of each filter has the same dimensions, although as to the benefits of this I'm not sure.

It seems that a similar question has been raised here: https://stackoverflow.com/questions/57438922/different-size-filters-in-the-same-layer-with-tensorflow-2-0
Like answered in the link above, you could combine severall Conv2D ops with different kernel sizes on the same input. You would have to adapt each output with padding, or cropping, so that you could concatenate all of them.
Hope this helps!


text:
Can multiple reinforcement algorithms be applied to the same system?
Can a system, for instance, a robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous task completion.

Hi. Maybe you're looking for multi-agent systems or hierarchical reinforcement learning. Have a look at these topics.


text:
When training a CNN, what are the hyperparameters to tune first?
I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read that doing a grid search for hyperparameters is not the best way to go about training and that random search is better in this case. Is random search really that good?

Learning rate schedule (decrease rate, cyclic etc) 2.Momentum if used(including 0) 3. augmentation (random transformations values) if used

@mirror2image momentum is fixed to 0.9 (it appears that it's common practice to fix it to this value if you want to focus on other hyperparameters). Augmentations are done, but I'm mostly talking about other hyperparameters : regularizarion, weight decay... AND MOST IMPORTANTLY: from where to start

Weight decay is useless in most cases. There are a lot of different regularizations, use common sense for them. Fixed momentum is bad practice. Always good idea to check momentum 0, 0.6, 0.9 Start with different learning rate schedules

@mirror2image so you say: I start with trying out different lr scheduling policies, and during each policy, I do a grid search of the learning rate,  momentum and the parameters of regulatization ?

First - schedule and learning rate with grid search. After that momentum and regularization separately - it's highly likely they are  independent.

This playlist by Andrew Ng helped me understand the different nuances of hyperparameter tuning. I'm sure it'll help you as well. https://www.youtube.com/watch?v=1waHlpKiNyY&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&ab_channel=DeepLearningAI

It depends a lot on what type of architecture you are using. However, most of the standard architectures are quite stable and there is no need for much hypreparameter tuning.

Choose whether you want to use a standard CNN architecture or a more advanced one like a ResNet.
Size? How many conv layers will you be using.
Number of filters in each of the layers. This is where you might need a bit of tuning.
You can try tuning the other parameters that are not specific to the problem, e.g. learning rate, regularization, dropout. If you already have a lot of feel for what works good and what not, then grid search according to your intuition would probably work pretty well. No need for random search in this case.

You can play around with batch Size, epochs, learning rate.
Start with a batch size of 8 and gradually increase two folds => 8,16,32
Start with a higher learning rate to find the direction towards global minima, then reduce learning rate to avoid bouncing around and reach local minima
Start with a low number of epochs to get quick feedback on the rest of hyperparameters. Once you have the optimal direction for hyperparameters move to higher number of epochs

Firstly when you say an object detection CNN, there are a huge number of model architectures available. Considering that you have narrowed down on your model architecture a CNN will have a few common layers like the ones below with hyperparameters you can tweak:

Convolution Layer:- number of kernels, kernel size, stride length, padding
MaxPooling Layer:- kernel size, stride length, padding
Dense Layer:- size
Dropout:- Percentage to keep/drop

Thanks for your answer! Actually I do not intend to change the architecture, so the kernel size, the stride... are not concerned. The main goal of my question is HOW TO GO ABOUT THE TRAINING. What to strat from?


text:
Is running more epochs really a direct cause of overfitting?
I've seen some comments in online articles/tutorials or Stack Overflow questions which suggest that increasing the number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between the number of epochs and overfitting. So I'm looking for an answer which explains if I'm right or wrong (or whatever's in between).
Here's my reasoning though. To overfit, you need to have enough free parameters (I think this is called "capacity" in neural networks) in your model to generate a function that can replicate the sample data points. If you don't have enough free parameters, you'll never overfit. You might just underfit.
So really, if you don't have too many free parameters, you could run infinite epochs and never overfit. If you have too many free parameters, then yes, the more epochs you have the more likely it is that you get to a place where you're overfitting. But that's just because running more epochs revealed the root cause: too many free parameters. The real loss function doesn't care about how many epochs you run. It existed the moment you defined your model structure before you ever even tried to do gradient descent on it.
In fact, I'd venture as far as to say: assuming you have the computational resources and time, you should always aim to run as many epochs as possible because that will tell you whether your model is prone to overfitting. Your best model will be the one that provides great training and validation accuracy, no matter how many epochs you run it for.
EDIT
While reading more into this, I realise I forgot to take into account that you can arbitrarily vary the sample size as well. Given a fixed model, a smaller sample size is more prone to overfitting. And then that kind of makes me doubt my intuition above. Still happy to get an answer though!

@DuttaA haha well thanks. I'm about to post an edit which makes me feel not so definitely correct. Check it out in a minute

I think it also depends on how you train the network. If you have some kind of regulariser (i.e. you add noise to the loss) while training the net, you may not overfit.

Early stopping of training (before loss converge) can sometimes act as stochastic regularizer and  decrease overfitting. That method  is highly unreliable though and could cause "hyperparameter overfitting". That is relationships between number of epochs and overfitting.

If my answer was helpful, please check it as right answer.

@ddaedalus sorry but I don't think your answer addresses the question. I think it explains what overfitting is and why it's a problem (as you aptly titled it). My question takes that knowledge as a prior and tries to challenge what I believe to be a common misconception.

The Problem of Overfitting
In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general, a good way to avoid overfitting, except for fine-tuning, regularization, dropout, etc, is to understand what you have from the learning curve. In most cases, overfitting happens, after some epochs have passed, and, as a result, the training error still decreases, whereas the validation error fluctuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.
Methods: early-stopping, checkpoint.
useful link: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/


text:
Are most things generally discovered because they work empirically and later justified mathematically, or vice-versa?
In the original GloVe paper, the authors discuss group theory when coming up with the equation (4). Is it possible that the authors came up with this model, figured out it was good, and then later found out various group theory justifications that justified it? Or was it discovered sequentially as it is described in the paper?
More generally: In AI research, are most things discovered because they work empirically and later justified mathematically, or is it the other way around?

This question is not only related to AI but research in general (even though you provide the example of GloVe).

There are definitely approaches that are theory driven (like SVMs), and others where the theory comes after the practice (like a lot of deep neural networks). I think it would be difficult to argue strongly that either direction is more common "in general" within AI, or indeed, within any other branch of science.
The approach that is currently in favor will tend to change over time as well. If we develop a new method that we don't understand, it will tend to attract interest from theorists. If they are able to understand it well, that understanding will likely lead to natural next steps that improve the model. As with other areas of science, the empirical cycle is at work.


text:
At which point, does the momentum based GD helps really in this figure?
Classical gradient descent algorithms sometimes overshoot and escape minima as they depend on the gradient only. You can see such a problem during the update from point 6.

In classical GD algorithm, the update equation is
$$\theta_{t+1} = \theta_{t} - \eta \times \triangledown_{\theta} \ell$$
In the momentum based GD algorithm, the update equations are
$$v_0 = 0$$
$$v_{t+1} = \alpha v_t +  \eta \times \triangledown_{\theta} \ell $$
$$\theta = \theta - \eta \times \triangledown_{\theta} \ell$$
I am writing all the equations concisely by removing the obvious variables used such as inputs to loss functions. In the lecture I'm listening to, the narrator says that momentum-based GD helps during the update at point 6 and the update will not lead to point 7 as shown in the figure and goes towards minima.
But for me, it seems that even momentum-based GD will go to point 7 and the update at point 7 will be benefited from the momentum-based GD as it does not lead to point 8 and goes towards minima.
Am I correct? If not, at which point does the momentum-based GD actually help?

Your momentum update equations look wrong. Importantly, the last one which updates parameters is not modified from basic gradient descent. It doesn't use the momentum. Could you check whether this is really the version you want to ask about?

What your professor most probably means is that since momentum adds your previous gradient to the current (as in a moving average). The speed built up till point 7 (moving in to the right) will added to the gradient from 7 to 8 that is going to the left, causing them to cancel each other out (what might cause the update to turn small enough for the loss to converge instead of explode).
So it helps at the point where the gradient suddenly changes direction, as the momentum will cancel it out partially.

Actually it won't go to 7, but will go to somewhere less far away. Because the  previous gradients at 5 and before are less steep, the accumulated gradient with momentum will be smaller and it will not shoot as far. The end effect is less overshoot and so it helps in this case.

Your concern is valid that's why there's another Nesterov accelerated gradient method specifically proposed to further address such edge case where various momentum based methods using past and current gradients may still overshoot at point 6 and even beyond. Nesterov momentum is essentially a form of one-step look ahead momentum which can reduce the amount of oscillation near your steep global/local minima.
However, with the usually very small learning rate the vanilla momentum methods have much better convergence rate near minima in general compared to traditional gradient descent first order methods. And as your source correctly claims in your above sharp minimum rare cases, momentum based optimization methods will obviously dampen the updated amount at each iteration point, thus comparing to traditional gradient descent methods they'll tend to not overshoot at point 6 and beyond.


text:
What are all the possible usages of 'multilayer perceptron'?
The term 'multilayer perceptron' has been used in literature in various ways in the literature.
I am presenting some of them below

As a feed-forward neural network [1].

As a fully connected feed-forward neural network [2].

As a fully connected feed-forward neural network in which each hidden layer has the same number of neurons.

As a fully connected feed-forward neural network in which each hidden layer has the same number of neurons and same activation function.

Afaik, the first definition is generally used, but it seems that there are many alternative definitions.
In this context, I want to know all the possible definitions that are floating in the literature for the word 'multilayer perception'.
I am asking this question because there can be several interpretations if we consider the words of 'multilayer perception' alone as the names suggests the only property required is multiple number of layers.

Words mean whatever the person saying them wants them to mean. Who said the words?

It is a series of weight matrices (possibly with a bias) separated by a non-linear activation function. So one layer can be described as this act_f(Wx+b). Where W is the weight matrix, b is the bias, x the input of the layer and act_f is the activation function. This is function is then repeated (taking in the output of the previous layer as x) possibly with different W, b and act_f's and this is repeated n times where n > 1 (as it's a multilayer preceptron).
This is an perceptron : act_f(Wx+b)
This is an possible example of a multilayer perceptron act_f(W*act_f(Wx+b)+b).
What kind of shape the W has doesn't really matter, you could even have skip connections would still be a mlp.
It's just a word used to describe this type of architecture, just like we use the word cat to describe a cat while there are many different types of cats.

what you're getting at is a major tension in machine learning: there are sometimes different ways of defining the same mathematical concept, and different mathematical definitions for the same term. In this case context is key. In a class on machine learning it will refer to a feed forward network, in some conference communities it will refer to a linear layer. In either case, the math in @hal9000's answer will describe it technically. Though at times they may simplify the notation even further and describe it as a simple weighing e.g.
$$y = \sigma(Wx)$$


text:
Is there any variant of perceptron convergence algorithm that ensures uniqueness?
The perceptron convergence algorithm given below ensures the convergence of weights of the perceptron provided enough data points and iterations.

Although it ensures convergence by finally getting a decision hyperplane that can separate positive samples (P) from negative samples N. It does not ensure the uniqueness of the decision hyperplane.

The solution is not unique, because there are more than one
hyperplanes separating two linearly separable classes.

Are there any variants to this algorithm in the literature that are capable of ensuring uniqueness?

The problem of multiple existing solutions is not uniquely related to the optimization algorithm of choice, it is actually many times related to the problem itself. More precisely, it is a direct consequence of the Rouché–Capelli theorem.
A simple or multilayer perceptron is nothing but a system of linear equations, in which the input set represent the coefficient matrix, the input set plus the targets represent the augmented matrix and the perceptron weights are the free parameters to solve.
As the theorem states, if the 2 matrices have the same rank, there is at least one solution, and if the rank of the 2 matrices is less than the number of unknowns, the solutions will be infinite in number.
Let's consider a very simple function, the XOR function in the image below. We can see that both, the coefficient matrix (input 1 + input 2 column) and the augmented matrix (input1 + input 2 + output) have rank 3, since row 1 and row 3 are not indipendent (i.e. I can rewrite one as a multiple of the other). Moreover, we can see that the number of unknowns is 4, less than the rank. Hence there are infinite solutions, because once I find a solution, for the weights for row 1, I can write infinite sets of weights, multiples of the solution found, that will work as parameters for row 3.
Note that the theorem talks only about coefficients, i.e. known variables, so this result and test holds regardless of the learned weights and therefore regardless of the optimization algorithm of choice.

Like what was mentioned by Edoardo, absent some other constraint, if your data is infinitely separable, then you can find an infinite number of separating hyperplanes. The perceptron algorithm will just find a hyperplane.
However there are constraints you can put on your model such that there only exists a single separating hyperplane.
The most famous is probably the support vector machine. Here, we assume that the "best" hyperplane should also be the one that's the furthest from the closest points in each class (i.e., the maximum-margin hyperplane). There are conditions in which this hyperplane is unique.


text:
How to configure a neural network to selectively change only certain characters in a string?
I'm trying to figure out how to train a neural network to macronize Latin text.
Essentially, in Latin, vowels can either be long or short, and length is indicated with a macronized character: i.e. o is short, ō is long. Most advanced texts don't mark vowel length, which can be an issue because vowel length determines pronunciation and word meaning: liber means book, līber means free, for example.
Is there any way to train a neural network to replace only vowels in a word while contextually evaluating it as part of a larger sentence? I've tried training an RNN many-to-many model to map unmacronized text to macronized text, but it ends up being incredibly inefficient and inaccurate because an unmacronized word can map to any macronized word. Ultimately, the only characters in each word that should ever change are the vowels, and even then only to their macronized counterparts: liber can be liber, līber, libēr, or lībēr only.
There is an abundance of training data in the form of parallel macronized and unmacronized text. This feels like a simple question but I've been unable to find any resources specifically detailing how to implement this. Any pointers would be greatly appreciated.

You could wrap each RNN cell in a custom module that is an identity for consonants (output = input when the input is a consonant) and predicts the macronization of vowels (it outputs the result from the RNN cell when the input is a vowel).
And it should always return the hidden state of the RNN cell so that each cell connects to the next cell for any input.


text:
Does all GAN's in literature need to satisfy the properties of objective function of initial GAN?
Consider the following value function of the initial GAN
$V(D, G) = \mathbb{E}_{x \sim p_{data(x)}} [\log D(x)] + \mathbb{E}_{z \sim p_z(z)} [1- \log D(G(z))]$
The min-max game on the value function: $min_{G} max_{D} V(D, G)$ ensures global optima. And finally $D(x) = 1/2$ for all $x$.
The paper provides the proof for attaining the convergence for $V(D, G)$.
After the paper, several GANs have been proposed and are using different value functions. So, I am wondering whether all the new value functions need to obey the mathematical properties of the initial $V(D, G)$ mentioned above so that the min-max game leads to the convergence point?

What are the properties you are referring to?

The aforementioned value function ensures Global Optimality of $p_g = p_{data}$. Section 4.1 contains the underlying properties I am asking for that ensure convergence.  Many of the papers on GANs introduce new value functions but do not show whether it ensures the global optima. @RaphaelLopezKaufman

$\min \max V(D, G)$ is not the objective function, but the optimization problem. The objective function is $V(D, G)$. Here's an analogy. Let's say you want to find the maximum of $f(x) = x^2$. $y^* = \max_x f(x)$ is your optimization problem and $f$ is the objective function. But it's also possible that some people are referring to the "optimization problem" as an "objective function", but it makes no sense to me.

Your question is also problematic because there could be many papers published and nobody probably knows all these papers. I recommend that you focus your comparison between the original objective and another specific objective (not all other objectives).

@nbro The property I'm asking for is the sole and soul of the GAN. Else, I think we cannot ensure convergence.  So, either all papers need to accept. But, I got doubts because many papers just use new value functions and do not take the task of proving the convergence. So, one counterexample. is enough.

Can you point out a paper where you think the objective function and optimization problem don't ensure that $p_g = p_{data}$? It will be simpler to start from a practical example

@RaphaelLopezKaufman You can take this for example.


text:
How Many Hidden Units in an LSTM?
Is there any rule of thumb for choosing the number of hidden units in an LSTM? Is it similar to hidden neurons in a regular feedforward neural network? I'm getting better results with my LSTM when I have a much bigger amount of hidden units (like 300 Hidden units for a problem with 14 inputs and 5 outputs),  is it normal that hidden units in an LSTM are usually much more than hidden neurons in a feedforward ANN? or am I just greatly overfitting my problem?

When you say "I'm getting better results with my LSTM", you need to be more precise for us to understand whether you're over-fitting or not. Can you provide more info about 1. how you're measuring the performance of your model, 2. your training dataset (i.e. how many training data points do you have?), 3. which task you're trying to solve, 4. which loss you're using, 4. how you're splitting the dataset into training, validation and test datasets, if at all, etc. Having said that, your question is partially a duplicate of https://ai.stackexchange.com/q/3156/2444.

I'm not sure about what you are referring to when you say "number of hidden units", but I will assume that it's the dimension of the hidden vector $h_t \in \mathbb{R}^N$ in this definition of an LSTM.
In general, the larger your model, in your case $N$, the more capacity your model has and, therefore, the more complex a function it can represent.
If by "better result" you mean smaller loss on the training dataset, it's very likely that you will indeed overfit more the more you increase $N$.
However, there are many techniques to increase your model expressiveness without overfitting, such as dropout.


text:
Why is it important/beneficial for an activation function to be zero-meaned?
Conventionally, (although there are plenty of better options) it is being said that as the choice of activation function for hidden layers, tanh should be prefered over sigmoid because it has a zero mean but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would tanh still be the go-to option?

i dont think it's important, sigmoid is just as fine as tanh

The first part of this answer is regarding your concrete question and the second part summarizes things on activation functions in general because I believe that there are more important factors than what exactly the output spaces of the functions are, for the following reasons:
Should activation functions be zero-meaned:
I would say that in general, it is neither beneficial nor important for an activation function to be zero-mean.

but what if the data at hand is 0.5 mean and we are not willing to zero mean the data? Would tanh still be the go-to option?

IMO this intuition is a bit off because the activation functions are not applied to the data space directly, but to the latent spaces that are learnable:
The matrix multiplication of the first layer will be applied prior to the activation function and it will change the topology of the data, hence also attributes like mean and variance. So your (let's say 2-dimensional) input data can have a mean of [-1, -1], but if the matrix of the first layer learns something like a 180° rotation, the mean of the input to the activation function is now [1, 1]. Therefore, I would say that it is not so important for the choice of activation function what the mean of your data is.
However, in certain cases, it may have some useful properties when the latent representations of your data are zero-meaned, but I would expect this to occur in shallow models rather than deep ones and I would say that this is something to find out empirically for the individual problem.
Disclaimer I don't think this question can be answered definitively, however, this is my understanding of the matter, happy to get comments and remarks on this.

Some facts regarding sigmoid $(\sigma)$ and tanh
Properties of the sigmoid $(\sigma)$

(-) The maximum of the derivative of the $\sigma$ is rather small ($\max \sigma'(x) = 0.25$) which can slow down training and prevent deep models from training at all due to the vanishing gradient problem. (s. derivative of sigmoid and derivative of tanh)

(-) The sigmoid saturates towards $-\infty$ and $+\infty$, thus the gradient becomes small when the pre-activation values are very positive or very negative values, this is also a cause for vanishing gradient and prevents the training of deep models.

(+) A very useful property is that $\sigma(x) \in (0, 1)$. This is the reason why it is commonly used as an output activation for binary classification models. For the same reason, it is the function that is used in gates of recurrent models (s. GRU-cell). So the sigmoid in this case can act as a continuous switch to determine the influence that, e.g. newly input features have on the hidden state of the model.

Properties of the tanh

(-) The tanh saturates towards $-\infty$ and $+\infty$ which poses similar drawbacks as for the sigmoid.

(o) The tanh can provide a larger gradient than the sigmoid ($\max \tanh'(x) = 1.0$), although  there are other functions like the ReLU function that will output a large gradient more consistently.

(+) There probably exist some regression problems for which tanh is a good match as an output activation.

(+) The $\tanh(x) \in (-1, 1)$ this is probably one reason why it is commonly used as the activation function inside RNNs (vanilla, GRU, LSTM). This ensures that the hidden state will always be in this range and doesn't explode after several processing steps.

What about the ReLU? The ReLU function is - to my knowledge - the most commonly used activation function for hidden layers (excluding RNNs). It consistently provides a gradient of one which leads to faster learning, especially for deep models. There are problems like dead neurons that occur when individual neurons produce only negative values (pre-activation) in which case the gradient and the output of that neuron will be zero til eternity. However, there are methods to reduce this problem, like controlling the bias, choosing a smaller learning rate, or switching to the Leaky-ReLu that can recover from 'dead' neurons.


text:
Make an NN utilize other NNs as part of its decision process
Suppose I have a NN that learns to predict the time it takes a robot to move between two jobs. That's three inputs (for starters): robot, job A, job B. Not all robots travel at the same speed, and jobs are not symmetrically spaced.
Suppose I have another NN that learns how long a robot takes to do some given job. All robots can do all jobs, but the robots are not equally effective at all jobs.
Now suppose that I want to make some new NN that takes a list of robots and a list of jobs and gives me robot-to-job map and sequence that minimizes the time to accomplish all jobs. Conceptually, how can I make this third NN utilize the knowledge contained in the other two NNs? What do you call this type of architecture? What structural form for this third NN do you recommend?

In tensorflow models can be their own layers as part of a larger architecture.
(If you want to see an example you can check out a notebook I have here and here)
Specifically what you are trying to do is called transfer learning
but in general, a NN model can take any numerical input set and use it to output another numerical output set. The input can even be the outputs of other models, and their hidden layers.
Suppose you have 2 models each with 4 layers, that take 2 datasets X1 & X2 and output 2 other sets:<br

X1 -> M1(L1,L2,L3,L4) -> Y1
X2 -> M2(L1,L2,L3,L4) -> Y2

You can directly concatenate the outputs (Y's) of each model into a new input to a 3rd model:

[Y1, Y2] = X3 -> M3(L1,...) -> Y3

or
you can slice each model and at one of the hidden layers and feed those outputs to M3:

[M1L3, M2L3] = X3 -> M3(L1,...) -> Y3

You can even mix and match in any way you find that works:

[M1L3, M2L2, Y1, Y2] = X3 -> M3(...) -> Y3

If you use a TensorFlow model, 'layers' is a property of the model and you can slice them as needed to build custom layers.
However, if you just want to feed the outputs of each model [Y1,Y2] to M3,
I would probably do this using tensorflow's functional api creating two branches:
Input1 = tf.keras.Layers.Input(...)
Input2 = tf.keras.Layers.Input(...)
x1 = M1(Input1)   #pretrained model 1
x2 = M2(Input2)   #pretrained model 2
Output = M3(x1)
Output = M3(x2)
model = tf.keras.Model(inputs=[Input1,Input2],
                       outputs=[Output]

Hope this helps! :)

How do I tell TensorFlow to not update the borrowed models when training the conglomerate?

to prevent layers from being trained you can explicitly set a 'trainable' parameter for the models .


text:
Is there a state-of-the-art deep learning paper that uses center point regression instead of bounding box regression, for object tracking?
Almost all deep learning based object tracking methods perform bounding box regression. Siamese-based networks which are very popular for object tracking also perform bounding box regression most of the time, although SiamFC type exceptions exist. And some other networks use a detection + tracking mechanism using the center points of the objects, however, what I am looking for is whether any Siamese based networks that predict the center of the object rather than the bounding box exist. I searched the literature but couldn't find any, I would appreciate any guidance.

I know that some bounding box models like yolo has output of 4 channels: hight-center, width-center,hight-size, width. and with they constract the bounding box, so you can modify those models to use only the first 2 channels.


text:
Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?
In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch of unit transitions to train neural networks as it is known that neural networks work well for iid data.
But in games, experience trajectory is important as it contains temporal dynamics. Am I true? If not, all the knowledge required to learn the policy function can be obtained from (out of sequence or randomly sampled) unit transitions alone.
Which one among the both is correct?
Note that unit transition in this question refers to $(s_t, a_t, r_t, s_{t+1})$

This is a confusing question. Please make it clear what your two options are? They are also not currently contradictory - a trajectory can be important and still broken down into parts/reassembled etc. So you need to make it clearer what your concern is about  needing to preserve a trajectory as an irreducable unit. Given that many of the earliest papers on Deep RL solve games using a database of single time step transitions, I believe that you have read them (?) and it is pretty clear that the approach works, then I would need to be be very clear on what your concern is before I could answer

But in games, experience trajectory is important as it contains temporal dynamics. Am I true? I don't know whether it is true or not that the entire trajectory should be taken into account to calculate policy function. I am viewing the first option as a sentence and the second action as a collection of all bigrams. It may be the wrong form of viewing in this case. @NeilSlater

Thanks for trying to clarify it. I still don't understand what your problem is - I just have a vague idea that you think whole trajectories are irreducable for some RL problems, but no clear understanding of why you think that. This is probably something fundamental in RL that you misunderstand, since it is reasonable to claim that all MDPs contain "temporal dynamics", and single-step temporal difference learning can be applied to any strict MDP (plus is often superior approach in terms of sample efficiency to monte carlo methods).

@NeilSlater, okay, I understood that policy can be learned from either total trajectory or unit transitions. I think some entities like expected returns are calculated from trajectory alone and since I'm habituated with the definitions that are generally done using expected returns, I'm thinking that trajectory is mandatory. And transition function of environment can be obtained from the collection of unit transitions also...


text:
Why can't I train like a dataset of samples instead of maintaining replay buffer?
On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode.
But, it seems there is no issue if we allow the completion of an episode and then treat the buffer as a dataset to update the neural networks if I don't have any memory constraints. Isn't it?
Even if the episode is too long, I think we can update neural networks once an episode terminates if I have enough memory. I don't think memory constraint is a reason for updating neural networks during episodes, as we can see large datasets in deep learning competitions and research. One possible reason I am thinking is that an episode may not have a goal state and can continue indefinitely.

In general MDPs do not have goal states. Although using the agent's
actions to achieve certain desirable end states, such as winning a
game or completing a puzzle, is a very common design, there is no
requirement for this. The more general requirement is to maximise some
aggregate of the reward at each time step - usually either a
discounted sum of rewards or the mean reward.

If 'having no goal state(s)' is a reason for using buffer, why do the authors claim it as an algorithm instead of a procedure? Am I missing something?


text:
What should be taken as random variables in the distributions of datasets?
Consider the following two paragraphs taken from the paper titles Generative Adversarial Nets by Ian J. Goodfellow et.al
#1: Abstract

We propose a new framework for estimating generative models via an
adversarial process, in which we simultaneously train two models: a
generative model G that captures the data distribution, and a
discriminative model D that estimates the probability that a sample
came from the training data rather than G. The training procedure for
G is to maximize the probability of D making a mistake. This framework
corresponds to a minimax two-player game. In the space of arbitrary
functions G and D, a unique solution exists, with G recovering the
training data distribution and D equal to 1.......

#2: Excerpt from Introduction

The promise of deep learning is to discover rich, hierarchical models
that represent probability distributions over the kinds of data
encountered in artificial intelligence applications, such as natural
images, audio waveforms containing speech, and symbols in natural
language corpora.

here we saw the word distribution thrice. It is very common to encounter the phrase data distribution in machine learning papers. For the types of data we use in artificial intelligence, there can be infinite random samples. We collect some instances and form a dataset, based on which we try to get the data distribution.
Even most of the literature uses the phrases data distribution or probability distribution. I personally never came across papers that explicitly talk about the random vector for the probability distribution under consideration. What can be the reason for it? Why won't research papers give the list of random variables for the distribution they are discussing? Is it immaterial or is it obvious from the context?

Note: For example, if our discussion is about a 'human face images' dataset and if we use the phrase 'data distribution' then should I assume random variables to be 'pixels' of the image or high-level features like 'eyes, ears, etc.,' or some other? Or is it immaterial? If immaterial, then what should I need to perceive about probability distribution? Should I imagine it as an arbitrary random vector?

The main reason of using the term data distribution over the random variable is to note for the intrinsic relationship the different data samples have with each other.

It is a mathematical way of saying this data is almost random
(consider the average pixel on a human face dataset) but it has some
underlaying relationship (the concept of human faces).

This is also helpful to describe that different images can be similar looking but represent different underlaying concepts. For example: consider 2 dataset: human faces, $x_{human}$, and animal faces, $x_{animal}$. Although most of the images might look similar at a pixel level: similar colors, similar textures, similar backgrounds... The underlaying concept depicted in them differs: human faces vs animal faces.
So, how do you represent that mathematically?: with two overlapping distributions of different shape.

This basically mean: "even though the images might seem random, might overlap in some domains (color, textures, histogram...) there is an intrinsic difference between the two" and this difference (the shape of the distribution) is the target intrinsic concept that the neural network will have to learn.
In the specific case of GANs the left distribution would be the "Generated Faces" and this distribution will try to fit the right distribution that would be "Real Faces".

So, can I safely infer that there is no need to imagine any random vectors?

For the data distribution you are right. There is no need. Another thing that may be confusing is that GANs model the data distribution starting from a random vector (random noise vector) but that is just a trick because in the end that random vector can be seen as the Z vector of an auto-encoder (the vector that tightly packs all the info)


text:
How to manually adjust output from model?
I wonder if it is possible to add manual inference to the output of a model?
For example, I have a model called 'net', and the output value of 'net' is a vector called v = [v1, ... vn]. v is a binary vector. For some reason, I need to manually adjust this output, which means I need to manually flip some 0s to 1s and vice versa.
My question is, is it possible to do so. The reason I have this question are

I am new to torch
If I manually change the variable.data, even it is doable, I did not do any due adjustments on gradient.

Please enlighten me under the context of PyTorch

To answer your question, yes, the computation graph in PyTorch is guaranteed as long as the operation in there is used by the torch package.
# Codesnip example for your case
import torch

# with input x
net = model()
output = net(x) # a binary vector
index = [0, 6, 8] # inverse the value at position (0, 6, 8)
output[index] = 1 - out[index]

Further, I don't know why you need to switch it, but switching, clipping, or forcing the output to be some values will harm the gradient, it makes your function non-differential so be careful when you use it.
Lastly, I think this question is more suitable for StackOverflow since you ask for the code editing and the implementation of your idea.

Yes, what you said is what I am currently trying to deal with. The gradient issue. After I hand-craft the output value from the model, my goal is to guide the gradient so it can converge to my modified output. Do you have any idea on this?

I think with the target vector, the cross entropy loss will help achieve the goal without hand-crafting

It doable of course! Let say you have a Net class with __init__ and __call__, you can do something like this:
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.architect = nn.Sequential(...)

    def __call__(self, x):
        out = self.architect(x)
        # Do anythin you want with the prediction in here
        return out

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
How to distinguish AI modeling from implementation?
Quote from this Eric's meta post about modelling and implementation:

They are not exactly the same, although strongly related. This was a very difficult lesson to learn among mathematicians and early programmers, notably in the 70s (mathematical proofs can demand a lot of non-trivial programming work to make them "computable", as in runnable on a computer).

If they're not the same, what is the difference?
How we can say when we're talking about AI implementation, and when about modelling? It's suggested above it's not an easy task. So where we can draw the line when we talk about it?
I'm asking in general, not specifically for this site, that's why I haven't posted this question in meta

One good way of differentiating modelling and implementation is to consider that models occupy a much higher level of abstraction.
To continue with the mathematical example: even though experimental mathematics might be dependent on computation, the program can be considered as one possible realization of the necessary conditions of a more abstract existence proof.
Over the last 25 years, software engineering methodologies have become quite good at separating models and implementations, e.g. by using interfaces/typeclasses/abstract base classes to define constraints on behavior that is concretely realized by the implementation of derived classes.
AI has always been a battle between the 'neats and the scruffies'. Neats tend to prefer working 'top down' from clean abstractions, 'scruffies' like to work 'bottom up', and 'bang the bits' of the implementation together, to see what happens.
Of course, in practice, interplay between both styles is necessary, but AI as a science progresses when we abstract mechanisms away from specific implementations into their most general (and hence re-useable) form.

In AI (but in general too, I believe), a simplification is that modeling is more akin to Mathematics (and related hard sciences involved, like Physics and... Computer Science), and implementation to Software Engineering.
Let's take a concrete example, really outside of AI: Find the minimum value of a given polynomial, if it exists.
The Mathematician will derivate the polynomial, find the zeros, and checkout convexity to find a minimum (if there is any zero). This procedure is very standard---some will say straightforward. It relies on a body of knowledge and an abstraction level that is appropriate for manual proof.
The Software Engineer approach is actually way longer to explain, and I am going to skip it. The point is that the body of knowledge is related but different: We have to find now a step-by-step procedure for the computer to achieve the result. The Mathematician one could be implemented directly in MathLab, almost verbatim, but we assume MathLab. And to build MathLab, we are back to the problem of making a procedure the computer can execute. We could for example base a procedure on Euler's method to find roots (a "simple" approach that closes on roots step after step), etc.
Simple mathematical operations can be quite complex to implement on a computer. Perhaps the most famous is random-number generation. Mathematically, the concept is pure and clear. Generating an actual random-number is more elusive than it looks, to the point it calls for new models and new implementations...
A concrete example from history: Neural networks. In the 80s and 90s, NNs were weighted graphs that could be executed on computers using graph libraries or similar foundation libs. Choosing the weights was challenging. One day the back-propagation learning model was introduced to automated the choice of weights. The model relied on a procedure dedicated to NNs, using a terminology like partial derivates, gradient descent, chain rules, etc. And later then, clever engineers created libraries to automate the back-propagation procedure. The libraries can be somewhat far from the original model, as engineers learn how to make it computable, even faster (i.e. optimization, approximations/truncations).


text:
Do you need single or multiple networks to detect multiple faces?
Given pictures with multiple features such as faces, can a single AI algorithm detect all of them, or for better reliability is it preferred to use separate instances?
In other words, I'm talking about an attempt of finding all possible human faces in the same picture by a single neural network.

Yes, finding several faces (unknown number of them) on one image by one AI network.

@bpachev Yes, I've added appropriate tag.

AFAIK, normally detection algorithms work in a sub-window of the image and not the whole of it. For example, for a specific size and orientation you slide a sub-window on the image and extract sub-images. Then you apply your algorithm on every sub-image for detection and report the size-and-orientations with positive results.
You can have a single neural network for face detection in this case or you might want to have different detectors for different orientation or any other feature, that is your decision.
There is also the technique of Combining Classifiers by which you can improve the decision of single classifiers by combining them.
Ensemble Learning is another way in which your classifiers are not trained independently but rather together. In fact, the well-known object detector of Viola and Jones uses such a technique.


text:
How should I handle invalid actions (when using REINFORCE)?
I want to create an AI which can play five-in-a-row/Gomoku. I want to use reinforcement learning for this.
I use the policy gradient method, namely REINFORCE, with baseline. For the value and policy function approximation, I use a neural network. It has convolutional and fully connected layers. All of the layers, except for the output, are shared. The policy's output layer has $8 \times 8=64$ (the size of the board) output unit and softmax on them. So it is stochastic.
But what if the network produces a very high probability for an invalid action? An invalid action is when the agent wants to check a square that has one X or O in it. I think it can be stuck in that game state.
How should I handle this situation?
My guess is to use the actor-critic method. For an invalid action, we should give a negative reward and pass the turn to the opponent.

Just ignore the invalid moves.
For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you will always retain an agent which occasionally makes illegal moves.
To me, it makes more sense to just set the probabilities of all illegal moves to zero and renormalise the output vector before you choose your move.

Do you think we need to take the invalid move with negative rewards for sample to train?

Maybe if there is consistently a lot of probability mass on illegal moves. Otherwise probably not.

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and the features $\phi$ is a function of the current state $s$ and an action from the set of actions $A$.
$$
\pi(\theta, a) = \frac{e^{\theta \phi(s, a)}}{\sum_{b \in A} e^{\theta \phi(s, b)}}
$$
To eliminate illegal moves, one would limit the set of actions to only those that were legal, hence $Legal(A)$.
$$
\pi(\theta, a) = \frac{e^{\theta \phi(s, a)}}{\sum_{b \in Legal(A)} e^{\theta \phi(s, b)}}, \, a \in Legal(A)
$$
In pseudocode the formula may look like this:
action_probs = Agent.getActionProbs(state)
legal_actions = filterLegalActions(state, action_probs)
best_legal_action = softmax(legal_actions)

Whether using linear or non-linear function approximation (your neural network), the idea is to only use the legal moves when computing your softmax. This method means that only valid moves will be given by the agent, which is good if you wanted to change your game later on, and that the difference in value between the limited choice in actions will be easier to discriminate by the agent. It will also be faster as the number of possible actions decreases.

The maths and pseudocode do not match here. Softmax over the legal move probabilities will adjust the relative probabilities. E.g. (0.3, 0.4, 0.2, 0.1) filtered with first and third item removed would be (0.0, 0.8, 0.0, 0.2) with your formula, but would be (0.0, 0.57, 0.0, 0.42) using the pseudocode. The pseudocode needs to take the logits, prior  to action probability calculations.

How does one compute the gradient of the filtered version of Softmax? Seems like this would be necessary for backpropagation to work successfuly, yes?

@brianberns Did you manage to find an answer? It seems like that would be the case to me but somehow in my toy example I'm only getting the right answer when using the log probabilities of the unfilitered softmax...

@tryingtolearn You could try to ask that as a separate question on our site (if it has not been asked before: I don't know).

I faced a similar issue recently with Minesweeper.
The way I solved it was by ignoring the illegal/invalid moves entirely.

Use the Q-network to predict the Q-values for all of your actions (valid and invalid)
Pre-process the Q-values by setting all of the invalid moves to a Q-value of zero/negative number (depends on your scenario)
Use a policy of your choice to select an action from the refined
Q-values (i.e. greedy or Boltzmann)
Execute the selected action and resume your DQN logic

Hope this helps.

Comments are not for extended discussion; this conversation has been moved to chat.

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative reward. Absurd? Sure!
But in fact your invalid moves are just a relic of the representation (which itself is straightforward and fine). The best treatment of them is to exclude them completely from any computation.
This gets more apparent in chess:

In a positional representation, you might consider the move a1-a8, which only belongs in the game if there's a Rook or a Queen at a1 (and some other conditions hold).
In a different representation, you might consider the move Qb2. Again, this may or may not belong to the game. When the current player has no Queen, then it surely does not.

As the invalid moves are related to the representation rather than to the game, they should not be considered at all.

Great point.  In [M] games, which are played on Sudoku, the constraints make many positions (coordinates+value) illegal after the first placement.  There is no value in considering these illegal positions from the standpoint of placement, but, an important strategic layer is recognizing which placements minimize value of remaining, unplayed positions. (i.e. if I place an 8 here, it blocks my opponent from placing an 8 in that row, column or region.  Essentially, "how many strategic positions does this placement remove from the gameboard?")

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the mechanism to handle invalid actions. However, those main references are still only pre-prints in the arxiv (not published and presumably not peer-reviewed yet).
On a way to handle that situation, other answers have given good practical methods to ignore the invalid actions. I just want to add one more trick to do that. You can pre-compute a binary vector as the mask for the actions, and add the log of the mask before the softmax operation. The log of 0 is -inf and exp(-inf) is 0 in Pytorch (I don't know if the same applies in Tensorflow).

$P(a_i| A'_t, X_t) = \text{softmax}(u_i+\log{m^t_i}), \forall i \in
 \{1,\dots,|A|\}$ and $m^t_i \in \{0,1\}$

with $P(a_i| A'_t, X_t)$ is the probability to take action $a_i$ given  the action history $A'_t$ up to the $t$-th step and the current environment's features $X_t$, $u$ is the output of the last layer of the model, $A$ is the action space $m^t_i$ is the feasibility mask of action $a_i$ for the current step.
Therefore,
the probability of the invalid action is 0 after the softmax operation.   That way, you can treat the mask as the part of the state as the input to your model. This is actually more handy for algorithm which employs experience memory because the mask then can be saved in the experience memory too.


text:
What is an objective function?
Local search algorithms are useful for solving pure optimization problems, in which the aim is to find the best state according to an objective function.
My question is what is the objective function?

The "objective function" is the function that you want to minimise or maximise in your problem.
The expression "objective function" is used in several different contexts (e.g. machine learning or linear programming), but it always refers to the function to be maximised or minimised in the specific (optimisation) problem. Hence, this expression is used in the context of mathematical optimisation.
For example, in machine learning, you define a model, $\mathcal{M}$. To train $\mathcal{M}$, you usually define a loss function $\mathcal{L}$ (e.g., a mean squared error), which you want to minimise. $\mathcal{L}$ is the "objective function" of your problem (which in this case is to be minimised).
In the context of search algorithms, the objective function could represent e.g. the cost of the solution. For example, in the case of the travelling salesman problem (TSP), you define a function, call it $C$, which represents the "cost" of the tour or Hamiltonian cycle, that is, a function which sums up the weights of all edges in the tour. In this case, the "objective" of your problem is to minimise this function $C$, because, essentially, you want to find an inexpensive tour, which is associated with either a local (or global) minimum of $C$. This function $C$ is the "objective function".
It should now be easy to memorise the expression "objective function", as it contains the term "objective", and the "objective" (or goal) in your (optimisation) problem is to minimise (or maximise) the corresponding function.


text:
Why does nobody use decision trees for visual question answering?
I'm starting a project that will involve computer vision, visual question answering, and explainability. I am currently choosing what type of algorithm to use for my classifier - a neural network or a decision tree.
It would seem to me that, because I want my system to include explainability, a decision tree would be the best choice. Decision trees are interpretable, whereas neural nets are like a black box.
The other differences I'm aware of are: decision trees are faster, neural networks are more accurate, and neural networks are better at modelling nonlinearity.
In all of the research I've done on computer vision and visual question answering, everyone uses neural networks, and no one seems to be using decision trees. Why? Is it for accuracy? I think a decision tree would be better because it is fast and interpretable, but if no one's using them for visual question answering, they must have a disadvantage that I haven't noticed.

For vision tasks, neural network models almost always include a number of layers that pool and convolute. The convolutions, in particular, are very useful - they can make the model generalize better to inputs and maintain performance when inputs have undergone certain linear transformations (e.g. some scaling or a translation along the x-axis). These properties, along with the robust frameworks that exist for developing and deploying neural nets, and the fact that they have been shown to widely produce very good results, are some of the reasons they’re used.
In terms of being a black box, while this is true for a lot of applications it’s actually less true for image-based tasks. The layers of a well-designed and trained convolutional neural network model can actually be visualized and made quite interpretable; from these visualizations, it’s often clear how the representation roughly works. In contrast, I’d argue that while a decision tree is theoretically easier to interpret for some tasks (say medical decision making), this is less the case for vision tasks because we don’t interpret images one pixel at a time or using some readily available feature of the image, such as the width, height, or color frequencies. People are almost always interested in the higher-level representation within an image (say, a cat, a leaf, or a face), and that sort of feature extraction is exactly what CNNs are good at. Decision trees, in contrast, tend to have trouble capturing these higher-level representations.
Distill.pub has a nice explanation of feature visualization that may be of interest.

Thanks Greenstick! I read about CNNs a while ago but had completely forgotten - now that you've explained I can't believe I didn't think of it right away haha. So if I understand correctly, a CNN would be more suitable than a decision tree for computer vision, because CNNs can represent features the same way regardless of location within the image, whereas decision trees can only represent a 'group of pixels' rather than the higher-level idea of features because the lack the convolution and pooling layers.

Yes, more or less, and it’s not just translation invariance but also size (scale) and rotation. Decision trees can represent more than just a group of pixels; I would be remiss to not say that feature engineering is often 90% of the work. A decision tree model can benefit hugely from that but interpretability will take a dip, depending on how you engineer your features. Generally, CNNs don’t require the feature engineering in a meaningful way because they were developed specifically for vision tasks. You can think of them as an algorithm specialized more for vision problems.


text:
Is my understanding of how the convolution with stride 2 works in this example correct?
I'm currently reading this explanation of convolutional neural networks and there's a part around strides that I don't quite understand. I'm just starting with this, so I apologize if this is a really basic question. But I'm trying to develop an understanding and some of these images have thrown me off.
Specifically, in this image

The stride has been increased to 2 and it's using a 3x3 filter (represented by the red, green and blue outline squares in the first picture)
Why is the blue square below the red one and not shifted to the side of the green one ending at the edge of the 7x7 volume? Should it not move left to right then down 2 squares when it reaches the next line?
I'm not sure if the author is just trying to show the stride moving down as it goes, but I think my confusion stems from the fact that the 1 stride image example is only moving in the horizontal direction (as seen below).

Is there something fundamental I haven't grasped here?

Yes, you're right, after the green one, it should also move two steps (because stride = 2) to the right once more. Note that in the $3 \times 3$ output volume picture, there's also still a white cell in the top right corner. That cell would get filled with whatever colour you choose to move to the right after the green one.
The blue one would then follow after what I described above, as the fourth square. I guess the author simply didn't feel like drawing a third square to the right of the green one, because the red + green squares already illustrate how the pattern works. The blue one was probably additionally drawn to illustrate that stride works the same way vertically as it does horizontally, e.g. blue also moves down two rows (because stride = 2).
See the bottom of the picture in the second answer here.