text:
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
Knowing that, my main question concerns Q-learning and off-policy methods:
For me, there is something weird in updating a Q value based on the max Q for a state and a reward value that was not from the action taken? How does this make learning better and makes you learn the optimal policy?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?

The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not affect whether the state has the Markov property. All that matters is that knowing the current state is enough to be determine the next state and reward in principle. You do not need to explicitly know the state transition model, or the reward function.
An example of a non-Markov state for CartPole would be if one of the features was missing - e.g. the current position of the cart, or the angular velocity of the pole. It is still possible to have agents attempt to solve CartPole with such missing data, but the state would no longer be Markov, and it would be a harder challenge.

For me, there is something weird in updating a Q value based on the max Q for a state and a reward value that was not from the action taken? How does this make learning better and makes you learn the optimal policy?

To recap, you are referring to this update equation, or perhaps some variation of it:
$$Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha(R_{t+1} + \gamma\text{max}_{a'}Q(S_{t+1},a') - Q(S_t, A_t))$$
Another correction here, the reward value is related to the action taken and the experience that happened. This is important - the values of $R_{t+1}$ and $S_{t+1}$ are from the experience that was observed after taking action $A_t$, they do not come from anywhere else. In fact this is the only data from experience that is inserted into the update, everything else is based on action value estimates in a somewhat self-referential way (a.k.a. "bootstrapping").
How does this converge to an optimal policy? It follows a process known as generalised policy iteration (GPI), which works as follows:

The target policy is to always take the best action according to your current estimates

In off policy methods only (i.e. does not apply to all GPI), there is a conversion needed from experience in a behaviour policy to value estimates for the target policy. In single-step Q learning, this is where the $\text{max}_{a'}Q(S_{t+1},a')$ comes from. It is calculating the future value from following the target policy*. No interim values are needed, because you are estimating a new value for $Q(S_t,A_t)$, there are no other time steps where you need to care about difference between behaviour and target policy (there is just "now" and "all the future, which is bootstrapped")

Once you update the action value estimates in Q table, that automatically updates the target policy because it always takes the best estimated action.

Each update improves your value estimates, due to injecting some real experienced data in the form of $R_{t+1}$ and $S_{t+1}$.

Each time the estimates improve, the target policy can also improve and become closer to the optimal policy.

When the target policy changes, this makes the estimates less accurate, because they were made for the previous policy. However, the next time the same states and actions are visited they will be updated based on the new improved policy.

With a stochastic environment, it is possible for estimates to get worse due good or bad luck for the agent when it tries different actions. However, over time, the law of large numbers will win, and the value estimates will converge to close to their true values for the current policy, which will make policy improvement inevitable if it is possible (caveat, it is only close to inevitable for the tabular Q-learning method which can be proven to converge eventually on an optimal policy).
There is proof that always taking the maximising action is a strict improvement to a policy when estimates are accurate. It is called the Policy Improvement Theorem.

* To be clear, there is nothing particularly clever about taking the max here. It is literally what the target policy has been chosen to do, so the term $\text{max}_{a'}Q(S_{t+1},a')$ is just the same as $V_{\pi}(S_{t+1})$ - the value of being in state $S_{t+1}$ for the current target policy $\pi$.

Comments are not for extended discussion; this conversation has been moved to chat.

@nbro Now this discussion is gone (404 error), but there was some interesting points in there, is there anyway to retrieve it ?

@JeanMi I have undeleted the room with the comments. Now, you should be able to access it.


text:
What are the differences between an agent and a model?
In the context of Artificial Intelligence, sometimes people use the word "agent" and sometimes use the word "model" to refer to the output of the whole "AI-process". For examples: "RL agents" and "deep learning models".
Are the two words interchangeable? If not, in what case should I use "agents" instead of "models" and vice versa?

Agent
The other answer defines an agent as a policy (as it's defined in reinforcement learning). However, although this definition is fine for most current purposes, given that currently agents are mainly used to solve video games, in the real world, an intelligent agent will also need to have a body, which Russell and Norvig call an architecture (section 2.4 of the 3rd edition of Artificial Intelligence: A Modern Approach, page 46), which should not be confused with an architecture of a model or neural network, but it's the computing device that contains the physical sensors and actuators for the agent to sense and act on the environment, respectively. So, to be more general, the agent is defined as follows

agent = body + policy (brain)

where the policy is what Russell and Norvig call the agent program, which is an implementation of the agent function.
Alternatively, it can be defined as follows

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

This is just another definition given by Russell and Norvig, which I also report in this answer, where I describe different types of agents. Note that these definitions are equivalent. However, in the first one, we just emphasize that we need some means to "think" (brain) and some means to "behave" (body).
These definitions are quite general, so I think people should use them, although, as I said above, sometimes people refer to an agent as just the policy.
Model
In this answer, I describe what a model is or what I like to think a model is, and how it is different from a function.
In AI, a model can refer to different but somehow related concepts.

For example, in reinforcement learning, a model typically refers to $p(s', r \mid s, a)$, i.e. the joint probability distribution over the next state $s'$ and reward $r$, given the current state $s$ and action $a$ taken in $s$.

In deep learning, a model typically refers to a neural network, which can be used to compute (or model) different functions. For example, a neural network can be used to compute/represent/model a policy, so, in this case, there would be no actual difference between a model and an agent (if defined as a policy, without a body). However, conceptually, at a higher-level, these would still be different (in the same way that biological neural networks are different from the brain).

More generally, in machine learning, a model typically refers to a system that can be changed to compute some function. Examples of models are decision trees, neural networks, linear regression models, etc. So, as I also state in the other answer, I like to think of a model as a set of functions, so, in this sense, a model would be a hypothesis class in computational learning theory. This definition is roughly consistent with $p(s', r \mid s, a)$, which can also be thought of as a (possibly infinite) set of functions, but note that a probability distribution is not exactly a set of functions.

In the context of knowledge bases, a model is an assignment to the variables, which represents a "possible world". See section 7.3, page 240, of the cited book.

There are possible other uses of the word model (both in the context of AI, e.g. in the context of planning, there's often the idea of a conceptual model, which is similar to an MDP in RL, and in other areas), but the definitions given above should be more or less widely applicable in their contexts.
What is the difference between an agent and a model?
Given that there are different possible definitions of a model depending on the context, it's not easy to briefly state what the difference between the two is.
So, here's the difference in the context of RL (and you can now find out the differences in other contexts by using the different definitions): an agent can have a model of the world, which allows it to predict e.g. the reward it will receive given its current state and some action that it decides to take. The model can allow the agent to plan. In this same context, a model could also refer to the specific system (e.g. a neural network) used to compute/represent the policy of the agent, but note that people usually refer to $p(s', r \mid s, a)$ when they use the word model in RL. See this post for more details.

Worth mentioning model-free vs model-based in context of RL? There is a question about that on the site that you could link - https://ai.stackexchange.com/questions/4456/whats-the-difference-between-model-free-and-model-based-reinforcement-learning . Answers to that question would tend to contradict your last sentence (the term "model" is avoided for neural networks that output policy or value etsimates in RL context, although it is reasonable in general because "model" can be any system that makes predictions for, or simulates some other system)

@NeilSlater I will link to that post. I say "a model could also refer to...", but you're right that people could still get confused.

In game AI context:

An Agent is a player that plays the game. Basically, its a function that gets the current state of the game and returns the next action.
A Model is a representation of the game.

For example, I have made a Gin-Rummy game + AI-agents.  One aspect in the model was the representation of the deck as a $4*13$ matrix, where each entry in the matrix is the card status (location, whether the card has been seen by opponent).
One can model the same game in different ways, most times there is a tradeoff between representability and simplicity.

Note that in game AI contexts, people also often refer to a trained neural network (or other kind of trained machine learning thing) as the "model", and the full thing that makes decisions (often a combination of a tree search algorithm + such a neural network) as the "agent"

Agents act
The key property of agents is that they act. Quoting one possible definition (Russel&Norvig, AI: A modern approach), "An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators." The environment can be the physical world around us, or some computer system, or a simulation, however, the key part of an agent that distinguishes it from non-agents is that it acts to affect that environment - as opposed to merely observing it and/or making some calculations.
Within the field of artificial intelligence there are many contexts where various models are used to decide upon a course of action. For example, reinforcement learning, or world models that are used in planning systems. In this case the whole system would be an agent, but the model would be just a part of that system.
However, that does not apply for all contexts - there are systems that are used for making decisions not related to any action (e.g. an OCR system recognizing particular letters), and there are systems that make a decision about some action, but are not intended to actually act on that decision (e.g. they show the proposed decision to a human which might act on that information). Such systems may contain models but in this case also the whole system should not be called an agent, because the actual act of acting is out of scope for that system.

Andrew Ng differentiates agents from non-agents as follows:

Example:

Typical agentic design patterns:

And here are a bit more details on each of these agentic design patterns:

Source: https://youtu.be/KrRD7r7y7NY

Rather than providing screenshots, it would be more useful to provide a summary, in words, of their definition of agent and model and how they are different (and still leave the link to the original source). Not just because this shows a bit of laziness but also because it's difficult to read some text in the screenshots. I have not fully read this but this seems to be more specific to LLM. The concept of agent has been around for a lot longer and it's not LLM that will redefine the concept just because a famous guy decided to redefine it. Also, the question is not about agents vs non-agents.

@nbro I thought AI people knew about OCR. My answer explains the current typical use of the term in the context of LLMs, which are rather popular these days

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How would I compute the optimal state-action value for a certain state and action?
I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:

Learning rate = 0.1
Discount factor = 0.95
Exploration rate = 0.1
Default reward = 0
The final reward (for reaching the trophy) = 1

After 500 episodes I got the following results:

How would I compute the optimal state-action value, for example, for state 2, where the agent is standing, and action south?
My intuition was to use the following update rule of the $q$ function:
$$Q[s, a] = Q[s, a] + \alpha (r + \gamma  \max_{a'}Q[s', a'] — Q[s, a])$$
But I am not sure of it. The math doesn't add up for me (when using the update rule).
I am also wondering either I should use the backup diagram to find the optimal state-action q value by propagating the reward (gained from reaching the trophy) to the state in question.
For reference, this is where I learned about the backup diagram.

by using the update rule in the question then you are implicitly using the backup diagram. the way to calculate the optimal q-values is using the update rule in the question.

So basically if I want to compute the optimal state 2 action South (where the agent is standing) considering the old q value =0.9 (q value of the red arrow where the agent is standing) and the max  q(s',a') is the q value of the red arrow in the state beneath it ?

the value of taking south from the agents current location is equal to the immediate reward it receives + the (discounted) q-value for the state it transitions into and action it takes under the current policy. as you're interested in the optimal policy then you want the action to be the one that maximises the q-value so yes it would be $r + \gamma Q(s', 'south')$ where $s'$ is the state directly below the agents current state.

remember though that if you would like to calculate the values 'from scratch' then using the Q-learning update rule is how you would go about it. The way I described is just the definition of a q-value.

Yeah I realized that there is a difference between the update rule and the definition of Q value. Thanks again!

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate
$$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$
where $s'$ is the state we transitioned into (note that this only holds when obtaining the optimal Q-value, if we were using a stochastic policy then we would have to introduce expectations).
Now, this assumes that we have been given/obtained the optimal Q-values. To obtain them, we have to use the update rule (or any other learning algorithm) that you mentioned in your question.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the purpose of storing the action $a$ within an experience tuple?
From what I understand, experience replay works by storing tuples of $(s, a, r, s')$  to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for storing the action $a$.
As I recall, the reward $r$ and the next state $s'$ are both used to calculate the target values. We can then compare these target values to the output we get when we do a forward-pass using state $s$ It seems to me that the stored action $a$ is not required for this process to work; or am I missing something? Why would we store the action $a$ if it isn't used in the training process itself?
Please, forgive me if this question has been answered before. I looked, but was unable to find anything other than generic explanations as to what experience replay is and why we do it.

We need to store the action $a$ as it tells us the action that we took in the state that we are backing up.
Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and next state $s'$. The goal of RL, and in particular DQN (I mention DQN as it is the first algorithm that comes to mind when I think of a replay buffer but it is of course not the only algorithm to make use of one), is that we are trying to learn optimal state-action value functions $Q(s, a)$. We thus want our value function to be able to predict $y = r + \gamma \max_{a'}Q(s', a')$, i.e. given $s$ and $a$ we want to be able to predict $y$. As you can see, we need to know which action we took in state $s$ so that we can train our value function to approximate $y$, and the value function clearly depends on $a$, hence we need to also store $a$ in our experience tuple.

Perhaps I now understand where my mistake was. Do I understand correctly, that for calculating the loss and performing gradient descent, we do not actually compare all of the Q-values, but only the one that corresponds to $Q(s,a)$ to its target value? Until now, I was under the impression we would do something like mean-squared-error, where we compare all the Q-values to their "labels" or rather the targets we calculated using $s'$ and $r$.

You're correct that we only update the Q-value for one action. I posted an answered here that seems like it might explain this in more detail.

I see. I think that answers my question, then :-) Thank you very much

The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to prevent forgetting of experiences that might be useful later on.
To meet these goals, the replay buffer should store tuples required in the learning step.
Most works that use experience replay, including those mentioned before, learn (to approximate) the Q function, i.e. $Q(s,a)$. Clearly, this function depends on the sampled action $a$.
If $a$ is not used at all in the training process then it would not need to be stored in the replay buffer. In such a scenario, however, the problems that motivate the replay buffer may not be present in the first place.


text:
How are certain machine learning models able to produce variable-length outputs given variable-length inputs?
Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable length. How is this possible?

In short, repetition with feedback.
You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work around this when desired input and output are more variable. The most common approaches are:

Padding: Give the ML model capacity to cope with the largest expected dimensions then pad inputs and filter outputs as necessary to match logical requirements. For example, this might be used for an image classifier where the input image varies in size and shape.

Recurrent models: Add an internal state to the ML model and use it to pass data along with each input, in order to work with sequences of identical, related inputs or outputs. This is a preferred architecture for natural language processing (NLP) tasks, where LSTMs, GRUs, and transformer networks are common choices.

A recurrent model relies on the fact that each input and output is the same kind of thing, at a different point in the sequence. The internal state of the model is used to combine information between points in the sequence so that for example a word in position three in the input has an impact on the choice of the word at position seven of the output.
Generative recurrent models often use their own output (or a sample based on the probabilities expressed in the output) as the next step's input.
It is well worth reading this blog for an introduction and some examples: The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy

This answer could probably do with an update to explain how transformers do this. I am not 100% sure, but it probably involves manipulating the input based on previous output, similar to, but not the same as RNNs.


text:
How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?
I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the form of a table containing features such as gender, age, etc.). So far, my researches have led me nowhere. What can I do in order to achieve this?

What you want to do is called multi-task learning. Here's what you do:

Create a second Input.
Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
Put a dense layer of 2 after this.
Put softmax with units = number of classes.
Add CE loss calculated with this one and the ground labels to the loss of U-Net.

This is in regards to TensorFlow. The same can be done in PyTorch easily.


text:
Why the number of training points to densely cover the space grows exponentially with the dimension?
In this lecture (minute 42), the professor says that the number of training examples we need to densely cover the space of training vectors grows exponentially with the dimension of the space. So we need $4^2=16$ training data points if we're working on $2D$ space. I'd like to ask why this is true and how is it proved/achieved? The professor was talking before about K-Nearest Neighbors and he was using $L^{1}$ and $L^{2}$ metrics. I don't think these metrics induce a topology that makes a discrete set of points dense in the ambient space.

$N^d=e^{d\ln(N)}$ is exponential in the dimension $d$. Fill a unit cube in dimension $d$ with small cubes of side length/diameter $1/N$.

First, let's try to build some intuition for what we mean when we say that we want to "densely cover" a $d$-dimensional space $\mathbb{R}^d$ of real numbers. For simplicity, let's assume that all values in all dimensions are restricted to lie in $[0, 1]$. Even with just a single dimension $d=1$, there are actually already infinitely many different possible values even in such a restricted $[0, 1]$ range.
But generally we don't actually care about literally covering every single possible value. Generally, we expect that points in this $d$-dimensional space that are "close" to each other also "behave" similarly, that there's some level of "continuity". Hence, to get "sufficient" or "good" or "dense" coverage of the space, you can somewhat informally assume that every data point you have occupies some space around it. This is the intuition behind Lutz Lehmann's comment under your question: you can think of every point as being a $d$-dimensional cube occupying some volume of your $d$-dimensional space.
Now, if you have a $d$-dimensional space of size $[0, 1]$ along every dimension, and you have little cubes that occupy a part of that space (for instance, cubes of size $0.1$ in every dimension), you will indeed find that the number of cubes you need to fill up your space scales exponentially in $d$. The basic idea is: if some number of cubes $K$ is sufficient to fill up the $d$-dimensional space, and if you then increase the dimensionality to $d+1$, you'll need $dK$ cubes to fill the new space. When you add a new dimension, the complete previous space becomes essentially just one "slice" of the new space.
For dimensions $d = 1, 2, 3$, this is fairly easy to visualise. If you have $d=1$, your space is really just a line, or a line segment if you constrain the values to lie in $[0, 1]$. If you have a $[0, 1]$ line segment, and you have little cubes of length $0.1$, you'll need just ten of them to fill up the line.
Now imagine that you add the second dimension. Suddenly your line becomes an entire plane, or a $10\times10$ square grid. The $10$ cubes are now only sufficient to fill up a single row, and you'll have to repeat this $10$ times over to fill up the entire $2$D space; you need $10^2 = 100$ cubes.
Now imagine that you add the third dimension. What used to be a plane gets "pulled out" into an entire three-dimensional cube -- a large cube, which will require many little cubes to fill! The plane that we had previously is again just a flat slice in this larger $3$D space, and the entire strategy for filling up a plane will have to be repeated $10$ times over to fill up $10$ such slices of the $3$D space; this now requires $10^3 = 1000$ cubes.
Past $3$ dimensions, the story continues in exactly the same way, but is a bit harder for us humans to visualise.

Thank you for the answer. It's much more detailed. So when we say "densely cover the space", it doesn't mean that our points are dense in this space but that they have some space around them and with which we can cover the whole space.

@Daviiid Well, points are still points. Mathematically, they don't occupy any space around them. But "densely covered" isn't a very precise definition, it's a bit informal. The basic intuition is that it should "feel like" you "sufficiently" cover most areas of the space, that you don't have any huge subsections of the space that are completely empty without any points in them at all.


text:
Is my approach to building an RNN to predict the probability that the word is in English appropriate?
Goal
To build an RNN which would receive a word as an input, and output the probability that the word is in English (or at least would be English sounding).
Example
input:  hello
output: 100%

input:  nmnmn
output: 0%

Approach
Here is my approach.
RNN
I have built an RNN with the following specifications: (the subscript $i$ means a specific time step)
The vectors (neurons):
$$
x_i \in \mathbb{R}^n \\
s_i \in \mathbb{R}^m \\
h_i \in \mathbb{R}^m \\
b_i \in \mathbb{R}^n \\
y_i \in \mathbb{R}^n \\
$$
The matrices (weights):
$$
U \in \mathbb{R}^{m \times n} \\
W \in \mathbb{R}^{m \times m} \\
V \in \mathbb{R}^{n \times m} \\
$$
This is how each time step is being fed forward:
$$
y_i = softmax(b_i) \\
b_i = V h_i \\
h_i = f(s_i) \\
s_i = U x_i + W h_{i-1} \\
$$
Note that the $ + W h_{i-1}$ will not be used on the first layer.
Losses
Then, for the loss of each layer, I used cross entropy ($t_i$ is the target, or expected output at time $i$):
$$
L_i = -\sum_{j=1}^{n} t_{i,j} \ln(y_{i,j})
$$
Then, the total loss of the network:
$$
L = \sum L_i
$$
RNN diagram
Here is a picture of the network that I drew:

Data pre-processing
Here is how data is fed into the network:
Each word is split into characters, and every character is split into a one-hot vector. Two special tokens START and END are being appended to the word from the beginning and the end. Then the input at each time step will be every sequential character without END, and the output at each time step will be the following character to the input.
Example
Here is an example:

Start with a word: "cat"
Split it into characters and append the special tags: START  c  a  t  END
Transform into one-hot vectors: $v_1, v_2, v_3, v_4, v_5$
Then the input is $v_1, v_2, v_3, v_4$ and the output $v_2, v_3, v_4, v_5$

Dataset
For the dataset, I used a list of English words.
Since I am working with English characters, the size of the input and output is $n=26+2=28$ (the $+2$ is for the extra START and END tags).
Hyper-parameters
Here are some more specifications:

Hidden size: $m=100$
Learning rate:  $0.001$
Number of training cycles:  $15000$ (each cycle is a loss calculation and backpropagation of a random word)
Activation function: $f(x) = \tanh(x)$

Problem/question
However, when I run my model, I get that the probability of some word being valid is about 0.9 regardless of the input.
For the probability of a word begin valid, I used the value at the last layer of the RNN at the position of END tag after feeding forward the word.
I wrote a gradient checking algorithm and the gradients seem to check up.
Is there conceptually something wrong with my neural network?
I played a bit with $m$, the learning rate, and the number of cycles, but nothing really improved the performance.

Does the training data include non-words too, or only words?

Only words. I saw a similar RNN online that used a database of sentences to generate new ones, and they didn't use "fake sentences", so I thought using only real words in mine would be enough.

while using a neural network for this type of problem is not the ideal use-case, it is a good exercise.
In terms of conceptual issues, the most concerning that I see is the loss: $\sum_{i=1}^N L_i$.
First issue, is that it validates loss at each time step equivalently. This is probably not ideal because in the example (cat), we dont expect it to know its English from just c or ca, but at cat. The quickest fix and probably the best, is to just use $L = L_N$. Though an argument would be that the model should become more and more aware as it gets more letters, and this loss function doesnt achieve that, so another solution would be to add another fixed parameter that you can play with: $L = \sum_i r^{-i}L_i$ where $0 \lt r \lt 1$. Note that $r^{-i}$ can be replaced with any function that increases in size.
Another issue with the loss is it doesnt normalize, meaning on average, larger words will hold more weight to the model than smaller ones, while this may be intended it should be noted and considered (also note if you do end up going with $L=L_N$ this will no longer be a concern.
Hope this helps

Thank you for your answer. I just tried implementing your modification and the performance of the network seemed to improve a bit. Still not perfect, but modifying the loss function was definitely  a step in the right direction. Btw, splitting the word into characters, and feeding them one by one to the network was not meant for testing if each substring is a valid word, but if a substring is a likely sequence of characters, for example: is it probable for the letter "t" to come after the sequence "ca".

are labels done that way? ex: flys -> 0011

Almost, they are encoded using one-hot vectors. ex: flys -> [onehot(START), onehot('f'), onehot('l'), onehot('y'), onehot('s'), onehot(END)]

i was asking about the labels

I don't think I understand. There are no labels. There is just a database of English words.

I think the problem is that you're only training the network on words. Every example in your training data has a desired label of "is a word," and so your network could achieve the lowest possible loss by simply giving a probability of 100% to "is a word" all of the time.
The most straightforward way to fix this would be to also include non-words in your training data. Of course, the words should have a target label of "is a word" and the non-words should have a target label of "is not a word."


text:
How to train a sequence labeling model with annotations from three annotators?
I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.
sentence = ['I', 'like', 'action', 'movies','!']
annotator_1 = ['O','O', 'B_A', 'I_A', 'O']
annotator_2 = ['O','O', 'B_A', 'I_A', 'O']
annotator_3 = ['O','O', 'B_A', 'O', 'O']

The labels follow the BIO format. That is, B_A means the beginning of aspect-term (action) and I_A indicates inside of aspect-term (movie).
Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).
I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.
Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?
Another question.
I merged the annotations in one final list of labels as follows:
final_annotation = ['O','O', 'B_A', 'I_A', 'O']

In this case, the final label is chosen based on the majority of labels among three annotators.
Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?

Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.
Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))

In the first assumption, sentences are fed into the classification model 3 times with different annotations. Could that lead to overfitting?

giving same input with different outputs does not cause overfitting, it creates an internal weighting of how much to trust each value. Depending on data this can also confuse the model.


text:
What is the intuition behind variational inference for Bayesian neural networks?
I'm trying to understand the concept of Variational Inference for BNNs. My source is this work. The aim is to minimize the divergence between the approx. distribution and the true posterior
$$\text{KL}(q_{\theta}(w)||p(w|D) = \int q_{\theta}(w) \ log \frac{q_{\theta}(w)}{p(w \mid D)} \ dw$$
This can be expanded out as $$- F[q_{\theta}] + \log \ p(D)$$ where $$F[q_{\theta}] = -\text{KL}(q_{\theta}(w) || p(w)) + E[\log p(D \mid w)]$$
Because $log \ p(D)$ does not contain any variational parameters, the derivative will be zero. I really would like to summarize the concept of VI in words.
How can one explain the last formula in words intuitive and with it on the fact that one approximates a function without really knowing it / able to compute it?
My attempt would be: Minimizing the KL between the approximate distribution and the true posterior boils down in minimizing the KL between the approximate distribution and the prior (?) and maximizing the log-likelihood that the parameters of the approximate distribution resulted in the data. Is this somehow correct?

Your description of what is going on is more or less correct, although I am not completely sure that you have really understood it, given your last question.
So, let me enumerate the steps.

The computation of the posterior is often intractable (given that the evidence, i.e. the denominator of the right-hand side of the Bayes' rule, might be numerically expensive to approximate/compute or there's no closed-form solution)

To address this intractability, you cast the Bayesian inference problem (i.e. the application of Bayes' rule) as an optimization problem

You assume that you can approximate the posterior with another simpler distribution (e.g. a Gaussian), known as the variational distribution

You formulate this optimization problem as the minimization of some notion of distance (e.g. the KL divergence) between the posterior and the VD

However, the KL divergence between the posterior and the VD turns out to be intractable too, given that, if you expand it, you will find out that there's still an evidence term

Therefore, you use a tractable surrogate (i.e. equivalent, up to some constant) objective function, which is known as the evidence lower bound (ELBO) (which is sometimes known as the variational free energy), which is the sum of 2 terms

KL divergence between the VD and the prior
the likelihood of the parameters given the data

To address your last doubt/question, the ELBO does not contain the posterior (i.e. what you really want to find), but only the variational distribution (you choose this!), the prior (which you also define/choose), and the likelihood (which, in practice, corresponds to the typical usage of the cross-entropy; so the only thing that you need more, with respect to the traditional neural networks, is the computation of the KL divergence): in other words, you originally formulate the problem as the minimization of the KL divergence between the posterior and the VD, but this is just a formulation.

thanks! So in fact instead of using a point estimator or the fully bayesian approach, you still want to have a distribution (and not point estimates) but instead of applying the true bayesian inference, you formulate the task as an optimization. The intuitive explanation of the free energy term might then be s.th. like: we of course want to increase the likelihood of the parameters given the data but at the same time want to stay close to the priors? Optimizing this term gives us a good approx. of the true posterior

@manu Yes, exactly, I would say that's the typical intuition interpretation of what is going on when you optimize the ELBO.

Already, thanks so much! Just one more question - you said that the ELBO is equivalent to the KL up to some constant. What exactly is the constant? or does the marginal likelihood that becomes by calculating derivative refers to this constant?

The constant is the (log of the) evidence, as you had said in the original version of the post. The ELBO is equal to (the negative of the) "KL divergence between the posterior and the VD" + "the evidence": that's why maximizing the ELBO is equivalent to the original optimization problem of minimizing the "KL divergence between the posterior and the VD". As you also had said, the evidence is a constant with respect to the variational parameters, i.e. the mean and variance of the VD, which is what you want to find.


text:
Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?
I have two questions on the Dueling DQN paper. First, I have an issue on understanding the identifiability that Dueling DQN paper mentions:

Here is my question: If we have given Q-values $Q(s, a; \theta)$ for all actions, I assume we can get value for state $s$ by:
$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$
and the advantage by:
$$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$
in which $\mathcal{A}(s)$ is the action space for state $s$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?
and then obtain Q-value using
$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$
or
$$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$
Am I missing something?
My second question is why Dueling DQN does not use the target network as it is used in the DQN paper?

Please, next time, ask only one question per post.

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.
However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because
$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$
by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.
Regarding your second question, I believe the author's appllied the Duelling architecture on Double Deep Q Networks, which is an improvement over the single DQN used by Minh et al in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.

Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN.

I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar.

@Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal.


text:
How would the performance of federated learning compare to the performance of centralized machine learning when the data is i.i.d.?
How would the performance of federated learning (FL) compare to the performance of centralized machine learning (ML), when the data is independent and identically distributed (i.i.d.)?
Moreover, what is the difference in the performance of FL when the data is i.i.d. as compared to non-i.i.d?

There are some works that do this comparison. Briefly, it's been observed that the performance of models trained via FL drops as data distributions between participating agents differ. When data is IID-like though, performance is comparable to centralized training. Some works that I'm aware of are as follows:

Overcoming Forgetting in Federated Learning on Non-IID Data
Improving Accuracy of Federated Learning in Non-IID Settings
Federated Learning with Non-IID Data

There are probably many more around. It's an active area of research.

Thanks for your information! one more question, do you have specific resources for "performance is comparable to centralized training."?

that’s what the results indicate. look at the experiments of paper 2 and 3.


text:
How is the AI in 3d games implemented?
A few days ago, I started looking a bit more into AI and learning about the way it works, and it is very interesting, but I can't find a clear answer on how the artificial intelligence is implemented in 3d shooter games, like COD or practically any 3d game.
I just don't understand how they teach the enemies such different things based on the game to fit its narratives. For example, is the enemy "AI" in 3d games just a bunch of if-else statements, or do they actually teach the enemies to think strategically? In big AAA games, you can clearly see that enemies hide from you in shootings and peek to shoot not just rush and get killed.
So, how is the AI in 3d games implemented? How do they code it? You don't need to explain in detail, but just give me the idea. Do they use algorithms?

Overlap between AI and "Game AI"
Nowadays, if you search for AI online, you will find a lot of material about machine learning, natural language processing, intelligent agents and neural networks. These are not the whole of AI by any means, expecially in a historical context, but they have recently been very successful, there is lots of published material about them.
Games, especially action games, tend not to use these new popular technologies because they have other priorities. However, under the broadest definitions, there is good overlap between AI in general and game AI, or more specifically enemy AI within a game:

Computer-controlled opponents within a game are effectively autonomous agents operating within an environment.

The game's purpose is to entertain the player, and this usually translates into requiring challenging and believable behaviour for actions taken by opponents (for some value of "believable").

A 3D map with a physics engine, other moving agents and possible hazards is a complex environment that needs some effort from a developer in order to define working behaviours.

Many games need to solve one or two classic AI problems such as pathfinding.

All the above means that the term "Game AI" is a good choice for describing how enemy agents are designed and implemented. However, there are large differences between goals of AI applied today in a 3D shooter and the goals of AI used in industry to control decision making.

Action games have a low CPU budget available per enemy for decision-making. Control systems in industry can have dedicated machinery, sometimes multiple machines, just to run the AI.

Action game priority is the game experience of the player, and this translates into strict targeting of a definition of "believable" for enemy behaviour that will entertain. In contrast, AI used in industry has accuracy as a high priority.

Typical game AI in 3D shooters
Developing enemy AI in games is a specialist skill, and only partly related to the purpose of this site. For detailed discussion you may want to look into Game Development Stack Exchange which has many questions and answers on topics like Enemy AI
Very roughly, the key traits of enemy AI in a game like Call of Duty could look like this (I have no inside knowledge of how COD does this, so may be inaccurate in places):

Enemies will have one or more modes of behaviour defined. This might be as simple as "Idle" and "Attack Player", or there could be many. For COD I would suspect there are many, and some may be in a hierarchy - e.g. there may be several sub-types of "Attack" behaviour depending on the enemy design.

Within a mode of behaviour, there may be a few different components defined, only some of which use AI routines. For instance, there will be specific animations related to walking or standing which are not AI, but there will also be some pathfinding AI if the agent is attempting to move around.

Decisions to switch between different modes of behaviour are often scripted with simple triggers, such as detecting player visibility using ray-casting between the location of player and enemy. When game AI fails to produce realistic results, it is often these high level triggers being brittle and not covering edge cases that causes it. Depending on complexity of the game and number of behaviours, there may be an algorithm managing the transitions between them, such as a finite state engine or behaviour trees.

Enemies will be presented with highly simplified observations of the game world from their perspective, in order to speed AI decisions.

AI systems will have CPU budget restricted. A 3D game spends significant resources rendering scenes, and will often try to render quickly, e.g. 100 times per second. There are multiple ways that AI budget can be allocated, but it is relatively common for AI calculations to be spread over muliple frames, and for search structures for tasks like path-finding to persist over time.

There will be a middle ground between scripted behaviour and AI-driven behaviour where analysis is done as part of game design. For instance, pathfinding routes may be pre-calculated to some degree. A system of way points is one example of this - it might be set by the game designer, it may be calculated by an AI component of the game asset-building pipeline, or it may be dynamically calculated and cached during a game session so that multiple enemy units can share it.

When complex AI is not needed to achieve a goal, when a simple caclulation or "puppet-like" behaviour would do just fine, then this could be chosen instead. For instance, an enemy "aiming" at a player can be a simple vector calculation, perhaps with a fudge factor of a miss chance depending on the range to make the enemy seem fallible and not as much like a machine.

I do not know how the behaviour of enemies using cover is implemented in COD. A simple variant would be to have each enemy hard-coded to use a specific piece of cover that usually works well against the player due to map design. However, it is definitley possible to have a search algorithm running (perhaps over multiple frames) that assesses nearby locations that the enemy could reach against the player's current position, and then pick those as variables to plug into the "take cover and fire on player" scripted behaviour. That assessment would use the same kind of visibilty detection between enemy and player that is used to trigger changes between "Idle" and "Attack" behaviours for the enemy.

As an aside, one related thing I find interesting is in using modern AI techniques to blend animations and make interactions between actors and the environment look more realistic. Although it is a lower-level feature than the question about enemy behaviour you are asking, it is an interesting cross-over between robotics and game playing that we will likely see in next-generation games, and probably applied first to the more detailed player model animations. Here is another example applied to a humanoid agent switching smoothly between different tasks, which any game player will recognise as something which game engines cannot do well at the moment - there are usually many jarring transitions caused by events in the game.

This answer seems reasonable to me, but given that some people, like me, are not really familiar with the details of the implementation of these "AI systems" in games, it may be a good idea to provide a reference to games that use the approaches that you mention. For example, when you say "Decisions to switch between different modes of behaviour are often scripted with simple triggers", this seems reasonable for anyone (including me) that played any 3d shooter/game, but maybe you could provide a reference that shows this is true.

@nbro: I think the best reference in general will be the Game Dev Stack Exchange link. I will try to find answers in there that back up my assertions, but I have to re-state that I have no evidence about the specific outlined approach being accurate for COD. There is definitely more than one way to structure enemy behaviour logic in a game.

@nbro: I added links to some ways of managing enemy behaviour transitions in games, which at least demonstrates that these are real things discussed by professional game developers and not something I just made up to answer the question

I have also added an aside to a related topic where modern AI and game development are overlapping and we should expect to see some results from it in near future with smoother movement and behaviour transitions in animations. So there is room for this kind of AI in games soon, although we're a long way off from each enemy in a 3D shooter having a decision-making neural network as standard.

Thanks! Now, I think your answer can be more useful for people like me that don't have a good knowledge of the topic. A few weeks ago, I also read a paper (I think Game AI Revisited by Yannakakis), which, if I remember correctly, is consistent with what you just said, so that's why I also upvoted your answer. You may be interested in that paper, which also talks about procedural content generation and stuff like that (if I remember correctly).

Most of the so called 'AI' enemies in games nowadays are based on human logics and not machine learning, although they're called AIs.
Mentioning about games, the first thing for ML is reinforcement learning. Let the bots play round to get rewards (positive) or punishment (negative rewards). These bots may finally know how to hide or move out a little bit to shoot.
I can guess of some possible actions for bots in shooter games:
Move U/D/L/R, jump, duck, find target (coordinates already in memory, valid only in view angle), shoot.
Training bot in 3D environments is much harder than in 2D games, since every looking angle of the bot is another environment state, 360 degrees around and 360 degrees up/down. And enemies can be at any location, in or off the frame.

OP is not asking how to implement ML for games, but how existing game AI works.

yeah, i said it's all human logics for present games, no ML


text:
Why do we sample vectors from a standard normal distribution for the generator?
I am new to GANs. I noticed that everybody generates a random vector (usually 100 dimensional) from a standard normal distribution $N(0, 1)$. My question is: why? Why don't they sample these vectors from a uniform distribution $U(0, 1)$? Does the standard normal distribution has some properties that other probability distributions don't have?

Seems to me that the normal distribution in high dimensions is causing problems with heavily under-sampled regions, so I'm wondering the same thing.

It has become our human bias that data will arrive from a normal distribution. It is also the most prevalent distribution in nature occurring in many places. Hence, we sample from a normal distribution. Also, central limit theorem works around means lying around normal distribution.
It is not taboo to use others if they are helpful to your network. But, a data from one distribution can transformed into other and if that is something that is required, the network will to do it since it can approximate anything (although a weak assumption, but, hey, its working right?).

Sometimes the uniform distribution can be more appropriate. I am not familiar with the details of the GAN, but there's probably a reason for sampling from the Gaussian or maybe this is an assumption or maybe it makes more sense, in this case, to put more density or specific values. Maybe you should explain what that vector that the OP is referring to is used for, cuz, if it is similar to the latent vector in VAEs, then the explanation of sampling from a Gaussian is not just "because we like Gaussians" or "because it's a habit", which is somehow what your answer seems to partially suggest.

It would be great if you could write the answer then. And yes uniform distributions are used. Did I say anywhere that others can't be used or it is taboo? And when I said it occurs in most places, it also occurs in images which GANs are prevalently used for. In a photograph, the subject is the important data, the rest of it is not important. That's a Gaussian for you.

You wrote "It has become our human bias that data will arrive from a normal distribution. ... Hence, we sample from a normal distribution". No, it's not a human bias (at least not of all humans). The fact that certain empirical distributions, such as the height of humans, seem to follow a normal distribution does not imply a bias or that we will select a normal distribution because we are biased towards it.

Moreover, you say "Also, central limit theorem works around means lying around normal distribution.", but the CTL is about possibly non-Gaussian distributed r.v.s that collectively follow a Gaussian: how does this relate to the GAN case?

I am giving him the social proof for normal distribution. Even non-Gaussian inadvertently follow the normal distribution somehow. I told you about the distribution of information in images. That is normal. That is enough.


text:
Is it appropriate to use a softmax activation with a categorical crossentropy loss?
I have a binary classification problem where I have 2 classes. A sample is either class 1 or class 2 - For simplicity, lets say they are exclusive from one another so it is definitely one or the other.
For this reason, in my neural network, I have specified a softmax activation in the last layer with 2 outputs and a categorical crossentropy for the loss. Using tensorflow:
model=tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=64, input_shape=(100,), activation='relu'))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Dense(units=32, activation='relu'))
model.add(tf.keras.layers.Dropout(0.4))
model.add(tf.keras.layers.Dense(units=2, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

Here are my questions.

If the sigmoid is equivalent to the softmax, firstly is it valid to specify 2 units with a softmax and categorical_crossentropy?

Is it the same as using binary_crossentropy (in this particular use case) with 2 classes and a sigmoid activation, and if so why?

I know that for non-exclusive multi-label problems with more than 2 classes, a binary_crossentropy with a sigmoid activation is used, why is the non-exclusivity about the multi-label case uniquely different from a binary classification with 2 classes only, with 1 (class 0 or class 1) output and a sigmoid with binary_crossentropy loss.

Let's first recap the definition of the binary cross-entropy (BCE) and the categorical cross-entropy (CCE).
Here's the BCE (equation 4.90 from this book)
$$-\sum_{n=1}^{N}\left( t_{n} \ln y_{n}+\left(1-t_{n}\right) \ln \left(1-y_{n}\right)\right) \label{1}\tag{1},$$
where

$t_{n} \in\{0,1\}$ is the target
$y_n \in [0, 1]$ is the prediction (as produced by the sigmoid), so $1 - y_n$ is the probability that $n$ belongs to the other class

Here's the CCE (equation 4.108)
$$
-\sum_{n=1}^{N} \sum_{k=1}^{K} t_{n k} \ln y_{n k}\label{2}\tag{2},
$$
where

$t_{n k} = \{0, 1\}$ is the target of input $n$ for class $k$, i.e. it's $1$ when $n$ is labelled as $k$ and $0$ otherwise (so it's $0$ for all $K$ except for one of them)
$y_{n k}$ is the probability that $n$ belongs to the class $k$, as produced by the softmax function

Let $K=2$. Then equation \ref{2} becomes
$$
-\sum_{n=1}^{N} \sum_{k=1}^{2} t_{n k} \ln y_{n k} =
-\sum_{n=1}^{N} \left( t_{n 1} \ln y_{n 1} + t_{n 2} \ln y_{n 2} \right)
\label{3}\tag{3}
$$
So, if $[y_{n 1},  y_{n 2}]$ is a probability vector (which is the case if you use the softmax as the activation function of the last layer), then, in theory, the BCE and CCE are equivalent in the case of binary classification. In practice, if you are using TensorFlow, to choose the most suitable loss function for your problem, you could take a look at this answer.


text:
What are the popular approaches to estimating the Q-function?
I need the q-value for my RL training, there are some approaches:

Brute-force the action sequence (this won't work for long sequence)
Use a classic algorithm to optimise and estimate (this ain't much AI)
Create Monte Carlo samples and train an approximator network for calculating q-value

I find the Monte Carlo method above rather widely applicable to different problems, and the more computing power, the more precise it is. Any other methods for calculating q-value?

How much RL do you know? When you say classic algorithms do you mean Dynamic Programming algorithms?

anything, not just dynamic programming. i can use some kind of greedy and random optimisation for the action sequence after the action in q(s,a); that's what i meant about 'classic algo'

Do you know any existing RL methods, such as Q-learning, SARSA? If I knew your existing knowledge of RL it would be easier to answer the question.

@datdinhquoc To me, this question is not fully clear (e.g. I don't understand what you mean by "Brute-force the action sequence"), so I agree with David that you should describe what you know about RL, and why you're trying to estimate a Q-table without apparently wanting to use RL algorithms, such as Q-learning. When we use "approximation" in RL we typically mean function approximation, but it doesn't seem that you meant that. You just want to "estimate" the state-action value function, apparently. So, what is your question? Are you asking which algorithms are there to estimate Q(s, a)?

'brute-force the action sequence' in future steps to get the best action at present state

yeah, approaches to estimate q(s,a), to get the best (or at least good) action at current state

@datdinhquoc Why don't you simply use Q-learning?

There's are some solutions to calculating q-values; find the exact values:

Brute-force the action sequence to find max (not pratical)
Do recursion on Bellman equation to get max (the same like action sequence brute-force, not pratical)

Estimate the q-values:

Based on different problems to solve, apply some classic algorithms or human logics to estimate; during estimation, some heuristic tactics may be used
Do a lot of randomisation to find max, including Monte Carlo tree search

Gradually optimise the q-values:
(1) Do optimisation based on Bellman equation (Q-learning):
$$ q_t(s_t,a_t) = q_t(s_t,a_t) + \alpha(r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1})) - q_t(s_t,a_t))$$
Bellman equation is true when the temporal difference (the part multiplied with $\alpha$) reaches zero, which means the max of time t+1 reaches exact value.
(2) Do optimisation based on Bellman equation (Q-network), fit the neural network to expected value:
$$r + \gamma\times\max(q_{t+1}(s_{t+1},a_{t+1}))$$


text:
What is the difference between feature extraction with or without data augmentation?
Here's an extract from Chollet's book "Deep Learning with Python" about using pre-trained CNN to predict class from a photo set (p. 146):

At this point, there are two ways you could proceed:

Running the convolutional base over your dataset, recording its output to a Numpy array on disk, and then using this data as input to
a standalone, densely connected classifier similar to those you saw in
part 1 of this book. This solution is fast and cheap to run, because
it only requires running the convolutional base once for every input
image, and the convolutional base is by far the most expensive part of
the pipeline. But for the same reason, this technique won’t allow you
to use data augmentation.

Extending the model you have (conv_base) by adding Dense layers on top, and running the whole thing end to end on the input data. This
will allow you to use data augmentation, because every input image
goes through the convolutional base every time it’s seen by the model.
But for the same reason, this technique is far more expensive than the
first.

The first method is called (1) and the second is (2).
If I use data augmentation to expand my data set, then could (1) be as good ad (2)? If no, why?

I am not sure how Chollet's first technique is supposed to work (it's not clear to me from this description and maybe it would be clear if I read more about it in the book), but he's saying "this technique won’t allow you to use data augmentation", but you're saying "If I use data augmentation to expand my dataset, then could (1) be as good as (2)?". I'm not sure why Chollet is saying you cannot use data augmentation in (1), but it seems that you didn't get that part. Can you clarify this?

There are two ways that you could perform data augmentation:

Up front, by expanding the input dataset into a larger one, performing a range of changes to each input then storing the result. This appears to be what you are suggesting.

Just in time, by sampling from possible augmentations on each epoch, or even per sample when building a mini-batch. This appears to be what Chollet is suggesting.

Chollet's approach allows for augmentation to include finer degrees of augmentation that are different each time an input is considered, e.g. rotations of any angle, selecting a slightly different area from a larger image each time. For your approach you could consider the same set of augmentations but they would have to be "frozen in" at the time of building a dataset, and you would not be able to consider all possible variations for each image because it would make the dataset too large.
Both approaches are valid, and both would be called data augmentation. Chollet's approach obtains better re-use of image samples in the long term, and may result in better generalisation in the final trained network. Your approach may allow for more efficient use of CPU time to obtain a result that passes a threshold in accuracy. In some cases the difference between approaches may be minor compared to that caused by other changes in hyperparameters.


text:
Which algorithm can be used for extracting text patterns in tabular data?
I am working with tabular data that is similar to the below:

Name
Phone Number
ISO3 Country
Amount
Email
...
...
Outcome
Possible Reason

Leona Sunfurry
(555)-555-5555
United States
58.96
leo_sun@gmail.com
...
...
0
Not ISO3 country

Diana Moonglory
(333)-555-5555
USA
8.32
di.moon@gmail.com
...
...
1

Fiora Quik
(111)-555-5555
FRA
0.35
null
...
...
1

Darius Guy
12345678901234
CAN
555.01
null
...
...
0
Too many digits in phone

LULU
(333)-555-5555
CAN
0.00
null
...
...
0
Odd name format

Eve K.
(111)-555-5555
FRA
69.25
e.k@gmail.com
...
...
1

Lucian Light
(999)-555-5555
ENG
65.00
null
...
...
1

Lux D.
(333)-555-5555
USA
11.64
test@test.com
...
...
1

Jarvin Crown
(333)-555-5555
USA
1357.13
j4@gmail.com
...
...
0
Unknown reason

The table contains information about users. Some of the fields are user-generated while others are generated by the program (like device location, amount, etc.). When this data is collected, it is sent to third parties (we will say a bank). Sometimes the bank rejects the data and it is not good for our users. The rejection could have happened because the user did not input the data correctly or the banks did not like how a field is formatted despite the data being correct and acceptable to other banks.
So we want to find the fields that are causing the most errors and how to fix the issue.
Does it make sense to do pattern recognition on the values to find the reason why the row was rejected? It would need to be an alpha-numeric type of algorithm, it seems.
We know the outcomes from the bank which is labeled as Outcome.  Although we have labeled data, it still feels like we need an unsupervised learning algorithm because we do not have labels on why the rows of data were rejected.
Does anyone know what type of algorithm would be best? Any feedback would be appreciated!

You should first segregate the rejected samples. You can use then use string matching or something more complex (like creating embeddings and then, taking L2 distance between them) between the different field names you have and the comment for rejection. Whichever field gets the highest score, you increase the rejection count for that field. In the end, you will have a tally of who is your biggest enemy.
You can create some rules which prevent injection of wrong data (like your password should be 7 characters long or something along these lines) or post-process your entries to match a uniform format.

Okay, I believe I understand. Are you familiar with an L2 library or functions in Python by any chance?
Also, I tried using correlation coefficients to find which fields correlated most highly with the outcome. Do you have an opinion on that option, albeit less sophisticated than an actual algorithm?

L2 distance you can easily find using numpy's linalg.norm or in scipy.spatial.distance.euclidean. But, you need to convert textual data to numbers. Fields which have negative correlation, if the outcome goes, they down, then they have an antagonistic relationship. Just use string matching and find the bad variables that is good starting approach. Afterwards, you need to devise some rules or methods things, that will take more time.


text:
Should one use an "other" category in image classification?
In image classification, there are sometimes images that do not fit in any category.
For example, if I build a CNN in Keras to classify Dogs and Cats, does it help (in terms of training time and performance) to create an "other" (or "unclassified") category in which images of houses, people, birds, etc., are classified? Is there any research paper that discusses this?
A similar question was asked before here, but, unfortunately, it has no answer.

Related question and my answer: https://ai.stackexchange.com/questions/17721/why-do-cnns-sometimes-make-highly-confident-mistakes-and-how-can-one-combat-th/17722#17722 basically you need to use a Bayesian neural network which produces good epistemic uncertainty in order to do out of distribution detection

Other similar questions have been asked here in the past: this, this or this.

Does this answer your question? In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?

It is not advisable because if you use an "other" class, you are just increasing problems for your network. Since "other" means not dog and not cat, then, what common feature does it have? Most of the time the "other" images won't have many features in common. If they do, then go ahead and make an "other" class.
There is a better way: if the probabilities for both cat and dog are less than a threshold (you need to decide that, take, 0.5), then, you can say it is an "other" object.

Thank you. If I know that some image categories (for ex. horses) will appear often but they are not of practical interest (my goal is to only tag Cats and Dogs), does it in that case make sense for model performance to make a third category?

It only makes sense if the third class has visual features similar to the classes of interest. Since, it will be a source of confusion in decision, it makes sense to make the network to learn to separate them out. On the other hand, if confusion is happening then you can increase the threshold for classification.


text:
What is the difference between a language model and a word embedding?
I am self-studying applications of deep learning on the NLP and machine translation.
I am confused about the concepts of "Language Model", "Word Embedding", "BLEU Score".
It appears to me that a language model is a way to predict the next word given its previous word. Word2vec is the similarity between two tokens. BLEU score is a way to measure the effectiveness of the language model.
Is my understanding correct? If not, can someone please point me to the right articles, paper, or any other online resources?

Simplified: Word Embeddings does not consider context, Language Models does.
For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word.
Think of the following two sentences:

The fish ate the cat.

and

The cat ate the fish.

If you averaged their word embeddings, they would have the same vector, but, in reality, their meaning (semantic) is very different.
Then the concept of contextualized word embeddings arose with language models that do consider the context, and give different embeddings depending on the context.
Both word embeddings (e.g Word2Vec) and language models (e.g BERT) are ways of representing text, where language models capture more information and are considered state-of-the-art for representing natural language in a vectorized format.
BLEU score is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Which is not directly related to the difference between traditional word embeddings and contextualized word embeddings (aka language models).

A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-th missing word using a language model. In this way, you can estimate which is the most probable word using for example the conditional probability $P(w_i=w|w_1,…,w_n)$.
An example of a simple language model is an n-gram where instead of conditioning on all previous words, you look only to the previous n words.
Word embeddings are a distributed representation of a word. Instead of using an index or a one-hot encoding to represent a word, a dense vector is used.  If two words have similar embeddings then these words share some properties. These properties are driven by the way embeddings are constructed, for example in word2vec two words with similar embeddings are two words that often appear in the same context, which is not to say they have the same meaning. Sometimes words with opposite meanings can have similar embeddings just because they are placed within the same sentences/contexts.
The BLEU score is a way to quantify the translation quality of an automatic translation. The score aims to look at how different model translation is to human translation.


text:
Why is no activation function needed for the output layer of a neural network for regression?
I'm a bit confused about the activation function in the output layer of a neural network trained for regression. In most tutorials, the output layer uses "sigmoid" to bring the results back to a nice number between 0 and 1.
But in this beginner example on the TensorFlow website, the output layer has no activation function at all? Is this allowed? Wouldn't the result be a crazy number that's all over the place? Or maybe TensorFlow has a hidden default activation?
This code is from the example where you predict miles per gallon based on horsepower of a car.
// input layer
model.add(tf.layers.dense({inputShape: [1], units: 1}));

// hidden layer
model.add(tf.layers.dense({units: 50, activation: 'sigmoid'}));

// output layer - no activation needed ???
model.add(tf.layers.dense({units: 1}));

In regression, the goal is to approximate a function $f: \mathcal{I} \rightarrow \mathbb{R}$, so $f(x) \in \mathbb{R}$. In other words, in regression, you want to learn a function whose outputs can be any number, so not necessarily just a number in the range $[0, 1]$.
You use the sigmoid as the activation function of the output layer of a neural network, for example, when you want to interpret it as a probability. This is typically done when you are using the binary cross-entropy loss function, i.e. you are solving a binary classification problem (i.e. the output can either be one of two classes/labels).
By default, tf.keras.layers.Dense does not use any activation function, which means that the output of your neural network is indeed just a linear combination of the inputs from the previous layer. This should be fine for a regression problem.

Thanks for the explanation. But if the weights in a neural network can be any number, won't the output also vary wildly? Miles per gallon should be a value of around 20 to 100 for example, but if the random weight is 2.000.000, won't you get a crazy output?

@Kokodoko Yes, this can happen, but the weights should converge to some reasonable values, once you optimize the objective function. Moreover, you can also limit the weights  and activations e.g. by using specific activation functions that squash the inputs to the neurons to certain ranges, which is your case (in that example, they are using the sigmoid in the hidden layer, which squashes the inputs to the range [0, 1]). You can also have regularization of the weights, to make them small.


text:
Is feature engineer an important step for a deep learning approach?
I'd like to ask you if feature engineering is an important step for a deep learning approach.
By feature engineering I mean some advanced preprocessing steps, such as looking at histogram distributions and try to make it look like a normal distribution or, in the case of time series, make it stationary first (not filling missing values or normalizing the data).
I feel like with enough regularization, the deep learning models don't need feature engineering compared to some machine learning models (SVMs, random forests, etc.), but I'm not sure.

Here is an older related (if not duplicate) question.

No, feature engineering is not an important step for deep learning (EDIT: compared to other techniques) provided that you have enough data. If your dataset is big enough (which varies from task to task), you can perform what is called an end-to-end learning.
To further clarify, according to this article, deep neural nets trained with backpropagation algorithm are basically doing an automated feature engineering.

I feel like with enough regularization, the deep learning models don't need feature engineering compared to some machine learning models (SVMs, random forests, etc.)

That is basically correct. Beware, you need a large dataset. When a large dataset is not available, you will do some manual work (feature engineering).
Nevertheless, it is always a good idea to look at your data first!
EDIT
I would also like to quote Rich Sutton here:

We want AI agents that can discover like we can, not which contain
what we have discovered. Building in our discoveries only makes it
harder to see how the discovering process can be done.

Perhaps this statement is more true with Deep Learning than with previous techniques, but we are not quite there yet. And as user nbro rightfully pointed out in the comments below, you may still need to normalise your data, pre-process it, remove outliers, etc. Thus in practice, you may still need to transform your data to a certain degree, depending on many factors.

Thank you for the two articles and for your answer.

End-to-end learning is somehow orthogonal to what the OP means by "feature engineering". If e.g. your neural network trained "end-to-end" (which simply means that you have only one net, so you don't have intermediate steps, or training procedures, etc.) is not learning anything, you may need to analyse your data, look at the histograms, find anomalies, no matter how much data you have. So, I think that this answer is misleading, and saying that "feature engineering" is not important is actually wrong. The more interesting question would be: when is feature engineering required?

@nbro I think we don't have a common language here. "Feature engineering is the process of using domain knowledge to extract features from raw data" (Wikipedia). End-to-end in contrast will not require using domain knowledge. For example, in computer vision people would traditionally do Gabor filters (domain knowledge). Today, people tend to avoid that as the filters are learned automatically by a convnet. Moreover, the filters learned automatically tend to be more efficient compared to handcrafted ones. Please also check this out http://incompleteideas.net/IncIdeas/BitterLesson.html

@penkovsky Even if you use a ConvNet, you may still need to normalise your data, pre-process it, remove outliers, etc. So, yeah, ConvNets learn features, but that doesn't mean that you cannot have a feature engineering step where you actually improve your (raw) data so that learning can happen faster.

"> you may need to analyse your data, look at the histograms, find anomalies, no matter how much data you have. <"
I also specifically wrote "Nevertheless, it is always a good idea to look at your data first!" However, I don't really see any reason to elaborate on that point since data analysis and visualisation have been discussed exhaustively.

From what I believe, feature engineering is important, it's a part of the job of ML network designer.
Network designing involves

Feature engineering: What should be in the input to the network, as processed from similar or totally different data
Deciding network shape, layer shapes, types of neurons in layers, etc.
Feature engineering again (but labels), in the output, what should the output be, either regression values or classes

And possibly also tasks rather simple as mentioned in the question: filling missing values, normalising data, create pre-feeding normalisation steps in code, etc.

Thank your for your answer and sorry for my late reply. I see. I think I didn't provide enough details in my question, but for example with images, I think it's better to feed them directly to the neural network, provided they work well with this type of data like CNNs, than to feed some other features like HOG descriptors.


text:
Why did the developement of neural networks stop between 50s and 80s?
In a video lecture on the development of neural networks and the history of deep learning (you can start from minute 13), the lecturer (Yann LeCunn) said that the development of neural networks stopped until the 80s because people were using the wrong neurons (which were binary so discontinuous) and that is due to the slowness of multiplying floating point numbers which made the use of backpropagation really difficult.
He said, I quote, "If you have continuous neurons, you need to multiply the activation of a neuron by a weight to get a contribution to the weighted sum."
But the statement stays true even with binary (or any discontinuous activation function) neurons. Am I wrong? (at least, as long as you're in the hidden layer, the output of your neuron will be multiplied by a weight I guess). The same professor said that the perceptron, ADALINE relied on weighted sums so they were computing multiplications anyways.
I don't know what I miss here and I hope someone will enlighten me.

Not answering your question directly, but I think a large contribution to the halted development was the lack of computing power. We simply couldn't easily test proposed methods because computers were too slow at the time

@Recessive thank you for the answer. I'll take it generally that there was a lack of computer power and not exactly the lack of computing power for multiplying floating numbers.

@DuttaA I agree, but throughout the video it's not said that the weights are 0 or 1. Sure the activation function does outputs zero or one, but the weights can be any real number. I mean the professor said that the perceptron was equipped with potentiometers to vary the weights of the connexions so the weights were real numbers.

@Daviiid Not necessarily, the GPU was only widely available in early 2000, which is what is primarily used now for fast floating point multiplication, so in the 50's-80's there was an even greater disadvantage to this than other problems.

@DuttaA oh yeah I didn't think about it, thank you !

I will first address your main question "Why did the development of neural networks stop between 50s and 80s?"
In 40-50s there was a lot of progress (McCulloch and Pitts); the perceptron was invented (Rosenblatt). That gave rise to an AI hype giving many promises (exactly like today)!
However, Minsky and Papert have proved in 1969 that a single-layer architecture is not enough to build a universal approximating machine (see e.g. Minsky, M. & Papert, S. Perceptrons: An Introduction to Computational Geometry, vol. 165 (1969)). That led to the first disappointment in "AI". Which lasted until several major breakthroughs in 1980s: the proof of universal approximating capabilities of multi-layer perceptron by Cybenko, the popularisation of backpropagation algorithm (Hinton and colleagues), etc.
I agree with LeCun that using continuous activation functions have enabled the backpropagation algorithm at the time. It is only recently that we have learned to backpropagate in networks with binary activation functions (2016!).

Thank you for your answer! LeCunn also says in the same lecture that we can approximate any function with an SVM, is it related to the universal approximating machine you mentioned ? And do you perhaps know in which article it was proved that we can approximate any function with an SVM please ? And that's crazy how it's until recently researchers found a method to train networks with binary activation functions. Like it was theoretically known that backpropagation works with discontinuous functions but we weren't able to do so in practice.

He probably refers to the Universal approximation theorem https://en.wikipedia.org/wiki/Universal_approximation_theorem. You may want to read e.g. this article http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.7873&rep=rep1&type=pdf

Thank you for your clarification and for the article. (I tried to cite you penkovsky with the @ but it's not working I'm sorry)

Happy to help. You may want to accept the answer if you find it satisfying.

Yes that's right, I always forget to do it. Thank you !


text:
Compute state space from variables in Q-learning (RL)
I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state.
Let's say, in my problem, there are the following variables, which I'm using to compute state:
x in range 0-3
y in range 0-3
d in range 0-3
g in range 0-1
a in range 0-1
s in range 0-4
br in range 0-4
bu in range 0-4
gl in range 0-1
So, the state space is equal to $64000$ ($4 * 4 * 4 * 2 * 2 * 5 * 5 * 5 * 2$). I'd like to create a number, from the above variables, which is contained in the range $[0, 63999]$.
My previous idea was to create a binary number from the binary representation of state variables (just write them next to each other and convert into an int). It seems to fail if a variable is not a power of two (bonus question: why doesn't it work?).

It seems to fail if a variable is not a power of two (bonus question: why doesn't it work?).

This does not work because you are wasting some space, some values are not used in your representation.
For example with your 0-4 variables, you need 3 bits, but you only use 000, 001, 010, 011 and 100 values. The values 101, 110 and 111 are still part of the representation you are using. It doesn't matter that you don't need to use them, you have created a representation where they exist. Every time you encode 5 values as 3 bits, you are being 62.5% efficient, and the efficiencies multiply to get the overall efficiency of your representation (proportion of states you need to represent compared to the size of the representation).
Sometimes this is acceptable. Your example state representation using bitwise coding per variable would still fit easily into a 32 bit integer for storage, and you also have the convenience of easily extracting the individual values quickly using bit masks. It's a bit less convenient if you were hoping to build a Q table as an array with the state as a simple offset.
In your example case it is not super-wasteful, your array will be around 4 times larger than optimal due to wastage in 3 separate places. For convenience of coding simplicity you might accept this 8MB space per action over 2MB space per action (assuming 32 bit floating point values for action values). 6MB of dead space is not much to worry about - the chances are that the programming language you have loaded to run it wastes far more space on features that you are not using.
If you have a few more variables and a few more wasted bit patterns, then the waste could be more noticeable and important. That might also be true if you are using a space-efficient compiled language on a system where memory resources are low.

Let's say in my problem there are following variables which i'm using to compute state . . . So the state space is equal to 64000 (4 * 4 * 4 * 2 * 2 * 5 * 5 * 5 * 2)

You are very close to a working answer for the most efficient representation here. You can use products of each variable's size to separate terms, to create multiplication factors for each variable. Imagine building up a cuboid out of the first three terms, and needing to address each cubic building block sequentially with an index position $i$. You would do this:
$$i = x + 4y + (4 \times 4)d = x + 4y + 16d$$
If you had only thie first 5 variables you would do this:
$$i = x + 4y + (4 \times 4)d + (4 \times 4 \times 4)g + (4 \times 4 \times 4 \times 2)a = x + 4y + 16d + 64g + 128a$$
To cover your whole set of variables, keep extending the same pattern. Each variable added to the end of the list is multiplied by the product of the space required by all previous variables. The last variable in your example, gl, would be multiplied by $32000$.
Reversing this encoding is more involved, especially if you only want to access a single variable. You need to start with the value of $i$ then repeatedly use an integer divmod operation to find the next unknown variable at the end of the list - using the same multiplier for that variable as when constructing the code - and a remainder for calculating the next variable along.
Luckily in reinforcement learning you probably don't need to do any reversing of state id to individual variables - you can maintain a structure of all current variables for the state when running the environment, and only need this compressed version to create an offset into the Q table for lookups for the policy or updates to stored values.


text:
How can I compress the states of a reinforcement learning agent?
I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like convolutional neural networks will not work here.
Is there some general solutions to reduce/compress the size of the states for reinforcement learning algorithms?

Compression will be lossy, some detailed features in the state will be off calculation.
A common technique might be using max-pool function or layer (before feeding to policy network if RL here is deep RL).
Max-pooling is very lossy, use some other classic compression algos such as Zip, Rar but using these classic no-loss compression is weird and extremely slow in the model pipeline.
Possible solution if allowing lossy data, commonly: Use max-pool (giving out high contrast data), average-pool (giving out blurred data).
To keep data intact, TensorFlow can compress tensors: "only sacrificing a tiny fraction of model performance. It can compress any floating point tensor to a much smaller sequence of bits."
See: https://github.com/tensorflow/compression

I think the OP is looking for lossy compression that works with state representations in use as input features, in which case zip etc will not work. However, it is not 100% clear from the question. In addition, pooling algorithms only work with dimensional patterns of identical signals (images, audio, financial series etc) - you should make that clear and it looks like the OP wants to exclude those solutions.

Exactly, the format of input data is in the form of tabular data, so I'm not sure max pooling or other signal processing solutions will work here.

this may be relevant then: https://github.com/tensorflow/compression


text:
Why do feedforward neural networks require the inputs to be of a fixed size, while RNNs can process variable-size inputs?
Why does a vanilla feedforward neural network only accept a fixed input size, while RNNs are capable of taking a series of inputs with no predetermined limit on the size? Can anyone elaborate on this with an example?

You are talking about two different types of 'size'. The size of the input for a FFNN and a RNN must always remain fixed for the same network architecture, i.e. they take in a vector $x \in \mathbb{R}^d$ and could not take as input for instance a vector $y \in \mathbb{R}^b$ where $b \neq d$. The size you refer to in the context of the RNN is the length of the input sequence.
What you are getting confused with is that RNN's can make predictions relating to sequences, that is imagine now rather than one $x$ we have a sequence of related, i.e. not i.i.d. data such as time series data, data $\{x_i\}_{i=1}^n$. Assuming we are given some initial $h_0$ (a hidden state) then an RNN will take as input $x_1$ and $h_0$ and output a prediction $y_1$ and a new hidden state $h_1$. In general an RNN will take as input $x_n$ and $h_{n-1}$ and output $y_n$ and $h_n$ where the hidden state is passed as input to the RNN at the next time step. However, the dimensionality of all the $x$'s and $h$'s will all be the same, i.e. $x_i \in \mathbb{R}^d$ and $h_i \in \mathbb{R}^c$ for all $i$, where $d$ does not necessarily have to be equal to $c$ (and in my experience rarely is).
Note that RNN's can also perform sequence to sequence prediction (such as language translation) where the predicted sequence can be a different length to the input sequence, as is the case when doing translation (the input sentence is not necessarily the same length in the translated language). They do this by having an encoder and a decoder which are two separate RNNs. The encoder is fed the input sequence and we maintain all the hidden states outputted by the encoder $\{e_i\}_{i=1}^n$. The decoder is then given a token that represents the start of a sequence and the last hidden state of the encoder $e_n$ as input to which it will make a prediction on what the word should be (I believe but am not 100% certain that the decoder outputs a probability distribution over the dictionary of words it can predict from) and then the word chosen is passed as input to the decoder at the next step, along with the hidden state from the decoder at the previous step and another word is predicted. This continues until the system predicts an End of Sequence token (or until it is forced to by some time limit). More details can be found in this paper but I believe in NLP it is much more common to use attention models now rather than the methods introduced in this paper.
Now, with this data $\{x_i\}_{i=1}^n$ you could in theory pass each one individually to a FFNN but a) it would not capture the sequential nature of the data as the assumption is that each data point is independent of the others and you can see this from the architecture of a FFNN -- they are Directed Acyclic Graphs, the Acyclic-ness is what causes the issue as there is no recursion which stops any sequential information from being passed from one time step to another and b) training using SGD would likely cause issues as we have violated the i.i.d. assumption needed for SGD to converge to local optima. More info on the i.i.d. requirement can be found here.

So, how can RNNs transform variable-length sentences (e.g. in English) to other variable-length sentences (e.g. in French), i.e. for tasks like machine translation (where sentences can have a different number of words or words could have a different number of letters)? :D So, you may want to give the example of machine translation to be more concrete and how could the variable-length word/sentence solved in this task.

NLP is not my cup of tea but I can give a brief explanation of eg seq2seq and point them to a relevant paper!


text:
What should the input and output of the Q-network be in the case of an ordinal action space?
I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible action (e.g. if you have three possible actions you will have three output units in your network). This makes a lot of sense from a computational standpoint and seems to work fine if you are dealing with categorical action spaces (e.g "left" or "right").
However, I am currently working with an action space that I discretized and the actions have an ordinal meaning (e.g. you can drive left or right in 5-degree increments). I assume that the action-value function has some monotonicity in the action component (think driving 45 degrees to the left instead of 40 will have a similar value).
Am I losing information on the similarity of actions, if I use a network that has an output unit for each possible action?
Are there implementations of the DQN available in which actions are used as network inputs?

I tried to summarise what your main problem is by putting a question that attempts to include all your other questions in the title. Please, make sure that your main question is the one in the title. If it's not, please, feel free to edit your post again to put in the title your main question.

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor:

If all actions are in the output, your neural network function is $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|} = [\hat{q}(s,a_1), \hat{q}(s,a_2), \hat{q}(s,a_3) ...]$, and you take the maximum value from the output vector as the greedy action.

If the action is provided as an input argument, your neural network function is $f(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \hat{q}(s,a)$, and to find the maximum value you construct and run a mini-batch over all possible values of $a$ for the given state.

Also see the answer to this question: Why does Deep Q Network outputs multiple Q values?
In your case, you would like to take advantage of similar values of $a$ because you expect that to work well with the approximation. As you correctly suggested, this will only work with the second approach using action as an input. So, use the steering angle, normalised into a suitable range for input to a neural network, as an input. Every time that you need to find $\text{max}_a Q(s,a)$ for the Q-learning algorithm, you must construct a mini-batch of the current state concatenated with each of the discrete steering angles that you want to consider as actions in the DQN, and run the current (or target) neural network forward.
If you want to go further and use a continuous action space, you will need to change which reinforcement learning method you are using. The various policy gradient and actor-critic approaches, such as REINFORCE, A3C, DDPG etc can cope with continuous actions, because they drop the need to find $\text{max}_a Q(s,a)$, which becomes impractical for very large action spaces.

Thank you Neil! I would like to upvote your answer but unfortunately miss the reputation. Are you by any chance aware of any library that provides an implementation of the second approach? I assume if I try to implement it, it will probably lead to some inefficiencies.

@PontrYagin No sorry I do not know of any library for this. So far I have implemented all my DQNs directly in Keras or PyTorch.