text:
What exactly is a hidden state in an LSTM and RNN?
I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.
Does this make sense?
I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &= [a, b, c, d, \dots,y , z]\\
Y &= [b, c, d, e, \dots,z , a]
\end{align}
We will first try to train a multi-layer perceptron (MLP) with one input and one output from $X$ and $Y$. Here, the details of the hidden layers don't matter.
We can write this relationship in maths as
$$f(x)\rightarrow y$$
where $x$ is an element of $X$ and $y$ is an element of $Y$ and $f(\cdot)$ is our MLP.
After training, if given the input $a = x$, our neural network will give an output $b = y$ because $f(\cdot)$ learned the mapping between the sequence $X$ and $Y$.
Now, instead of the above sequences, try to teach the following sequences to the same MLP.
\begin{align}
X &= [a,a,b,b,c,c,\cdots, y,z,z]\\
Y &= [a,b,c,\cdots, z,a,b,c, \cdots, y,z]
\end{align}
More than likely, this MLP will not be able to learn the relationship between $X$ and $Y$. This is because a simple MLP can't learn and understand the relationship between the previous and current characters.
Now, we use the same sequences to train an RNN. In an RNN, we take two inputs, one for our input and the previous hidden values, and two outputs, one for the output and the next hidden values.
$$f(x, h_t)\rightarrow (y, h_{t+1})$$
Important: here $h_{t+1}$ represents the next hidden value.
We will execute some sequences of this RNN model. We initialize the hidden value to zero.
x = a and h = 0
(a,next_hidden) <- f(x,h)
prev_hidden = next_hidden

x = a and h = prev_hidden
(b,next_hidden) <- f(x,h)
prev_hidden = next_hidden

x = b and h = prev_hidden
(c,next_hidden) <- f(x,h)
prev_hidden = next_hidden

and so on

If we look at the above process we can see that we are taking the previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.

I have a question regarding the output and hidden state here.
here i already know what i output will look like at each time step then then should the hidden state be unknown?
a normal neural net is farely simple i give it some inputs and corresponding outputs and it calculates the weigths and biases in order to predict a data set that you give it.. can i be using lstm to do the same thing?

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.
A simple RNN then might have an input $x_t$, a hidden layer $h_t$, and an output $y_t$ at each time step $t$. The values of the hidden layer $h_t$ are often computed as:
$h_t = f(W_{xh}x_t + W_{hh}h_{t-1})$
Where $f$ is some non-linear function, $W_{xh}$ is a weight matrix of size $h\times x$, and $W_{hh}$ is a weight matrix of size $h\times h$. I've left out the bias terms for simplicity.
Thus, the values of the hidden layer $h_t$ depend on the input $x_t$ as well as on the previous hidden state $h_{t-1}$ (literally, the previous values of the hidden layer).

Clearest explanation, bravo

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?
The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format.
I'd recommend you read this blog on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can produce some target output. Each layer within a neural network can only really "see" an input according to the specifics of its nodes, so each layer produces unique "snapshots" of whatever it is processing. Hidden states are sort of intermediate snapshots of the original input data, transformed in whatever way the given layer's nodes and neural weighting require.
The snapshots are just vectors so they can theoretically be processed by any other layer - by either an encoding layer or a decoding layer in your example.


text:
Where does entropy enter in Soft Actor-Critic?
I am currently trying to understand SAC (Soft Actor-Critic), and I am thinking of it as a basic actor-critic with the entropy included. However, I expected the entropy to appear in the Q-function. From SpinningUp-SAC, it looks like the entropy is entering through the value-function, so I'm thinking it enters by the $\log \pi_{\phi}(a_t \mid s_t)$ in the value function?
I'm a little stuck on understanding SAC, can anyone confirm/explain this to me?
Also, side-note question: is being a soft agent equivalent to including entropy in one of the object functions?

In the answer I'll be using notation similar to the one from the SAC paper.
If we look at the standard objective function for policy gradient methods we have
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) +  \mathbb E_{a_{t+1} \sim \pi(a|s_{t+1})}[ \mathbb E_{s_{t+2} \sim p(s|s_{t+1}, a_{t+1})} [r(s_{t+1}, a_{t+1}) + V(s_{t+2})]]]]\\
&\cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r( s_t, a_t)]
\end{align}
If you keep unwinding this $V(s_{t+i})$ you will get expected sum of rewards.
We can define soft state value as
\begin{align}
V(s_t) &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathcal H(\cdot|s_t)]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) + \mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]
\end{align}
third equality comes from the fact that $\mathbb E_{a \sim \pi(a|s_t)}[-\log(\pi(a|s_t))]$ is nonrandom so its the same thing as if we are sampling over $\pi$ only once.
In maximum entropy framework objective function would then be
\begin{align}
J_\pi &= V_\pi(s_t)\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t) - \log(\pi(a_t|s_t))]\\
&= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) - \log(\pi(a_t|s_t)) + V(s_{t+1})]]\\
& \cdots\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) -\log(\pi(a_t|s_t))]\\
&= \sum_t \mathbb E_{(a_t, s_t) \sim \rho_\pi} [r(s_t, a_t) + \mathcal H(\cdot|s_t)]
\end{align}

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is a probability distribution in machine learning?
If we were learning or working in the machine learning field, then we frequently come across the term "probability distribution". I know what probability, conditional probability, and probability distribution/density in math mean, but what is its meaning in machine learning?
Take this example where $x$ is an element of $D$, which is a dataset,
$$x \in D$$
Let's say our dataset ($D$) is MNIST with about 70,000 images, so then $x$ becomes any image of those 70,000 images.
In many papers and web articles, these terms are often denoted as probability distributions
$$p(x)$$
or
$$p\left(z \mid x \right)$$

What does $p(\cdot)$ even mean, and what kind of output does $p(\cdot)$ give?
Is the output of $p(\cdot)$ a scalar, vector, or matrix?
If the output is vector or matrix, then will the sum of all elements of this vector/matrix always be $1$?

This is my understanding,
$p(\cdot)$ is a function which maps the real distribution of the whole dataset $D$. Then  $p(x)$ gives a scalar probability value given $x$, which is calculated from real distribution $p(\cdot)$. Similar to $p(H)=0.5$ in a coin toss experiment $D={\{H,T}\}$.
$p\left(z \mid x \right)$ is another function that maps the real distribution of the whole dataset to a vector $z$ given an input $x$ and the $z$ vector is a probability distribution that sums to $1$.
Are my assumptions correct?
An example would be a VAE's data generation process, which is represented in this equation
$$p_\theta(\mathbf{x}^{(i)}) = \int p_\theta(\mathbf{x}^{(i)}\vert\mathbf{z}) p_\theta(\mathbf{z}) d\mathbf{z}$$

In many papers and web articles we notice these terms as probabiltiy distributions Please link the paper or web page so that it makes easier to get hold of the question and write an answer.

Random variables
You do not necessarily need to understand the concept of a random variable (r.v.) to understand the concept of a probability distribution, but the concept of a random variable is strictly connected to the concept of a probability distribution (given that each random variable has an associated probability distribution), so, before proceeding, you should get familiar with the concept of an r.v., which is a (measurable) function from the sample space (the set of possible outcomes of an experiment) to a measurable space (you can ignore the definition of a measurable space and assume that the codomain of the random variable is a finite set of numbers).
Probability measure, cdf, pdf and pmf
The expression "probability distribution" can be ambiguous because it can be used to refer to different (even though related) mathematical concepts, such as probability measure, cumulative distribution function (c.d.f.), probability density function (p.d.f.), probability mass function (p.m.f.). If a person uses the expression "probability distribution", he (or she) intentionally (or not) refers to one or more of these mathematical concepts, depending on the context. However, a probability distribution is almost always a synonym for probability measure or c.d.f..
For example, if I say "Consider the Gaussian probability distribution", in that case, I could be referring to either the c.d.f. or the p.d.f. (or both) of the Gaussian distribution. Why couldn't I be referring to the p.m.f. of the Gaussian distribution? Because the Gaussian distribution is a continuous distribution, so it is a distribution associated with a continuous random variable, that is, a random variable that can take on continuous values (e.g. real numbers), so a Gaussian distribution does not have an associated p.m.f. or, in other words, no p.m.f. is defined for the Gaussian distribution. Why don't I simply say "Consider the p.d.f. of the Gaussian distribution." or "Consider a Gaussian p.d.f."? Because it is unnecessarily restrictive, given that, if I say "Consider the Gaussian distribution" I am implicitly also considering a p.d.f. and c.d.f. of the Gaussian distribution.
Similarly, in the case of a discrete distribution, such as the Bernoulli distribution, only the c.d.f. and p.m.f. are defined, so the Bernoulli distribution does not have an associated p.d.f.
However, it is important to recall that both continuous and discrete distributions have an associated c.d.f., so the expression "probability distribution" almost always (implicitly) refers to a c.d.f., which is defined based on a probability measure (as stated above).
Notation
In the same vein, the notation $p(x)$ can be as ambiguous as the expression "probability distribution", given that it can refer to different (but again related) concepts. However, $p(x)$ usually refers to a probability measure (so it refers to a probability distribution, given that a probability distribution is almost always a synonym for probability measure). In this case, assuming for simplicity that the r.v. is discrete, $p(x)$ is a shorthand for $p(X=x)$, which is also written as $\mathbb{P}(X=x)$ or $\operatorname{Pr}(X=x)$, where $X$ is a r.v., $x$ a realization of $X$ (that is, a value that the r.v. $X$ can take) and $X=x$ represents an event. Given that an r.v. is a function, the notation $X=x$ may look a bit weird.
In the case of a discrete r.v., $p(x)$ can also refer to a p.m.f. and it can be defined as $p_X(x) = \mathbb{P}(X=x)$ (I added the subscript $X$ to $p$ to emphasize that this is the p.m.f. of the discrete r.v. $X$). In the case of a continuous r.v., the p.d.f. is often denoted as $f$. In the case of both discrete and continuous r.v.s, the c.d.f is usually denoted with $F$ and it is defined as $F_X(x) = \mathbb{P}(X \leq x)$, where $\mathbb{P}$ is again a probability measure (or probability distribution). The p.d.f. of a continuous r.v. is then defined as the derivative of $F$. At this point, it should be clear why a probability distribution can refer to different but related concepts, but, in any case, it always refers to a probability measure.
Empirical distributions
There are also empirical distributions, which are distributions of the data that you have collected. For example, if you toss a coin 10 times, you will collect the results ("heads" or "tails"). You can count the number of times the coin landed on heads and tails, then you plot these numbers as a histogram, which essentially represents your empirical distribution, where the adjective "empirical" usually refers to the fact that there is an experiment involved.
Multivariate r.v.s and distributions
To complicate things even more, there are also multivariate random variables and probability distributions. However, all the concepts above more or less are also applicable in this case.
Parametrized distributions
A parametrized probability distribution, often denoted by $p_{\theta}$, is
a family of probability distributions (defined by the parameters $\theta$), rather than a single probability distribution. For example, $\mathcal{N}(0, 1)$ refers to a single Gaussian distribution with zero mean and unit variance. However, $\mathcal{N}(\mu, \sigma)$, where $\theta=(\mu,  \sigma)$ is a variable, is a family (or collection) of distributions.
Conclusion
To conclude, it is completely understandable that you are confused, given that the terminology and notation are used inconsistently, and there are several involved concepts, which I have not extensively covered in this answer (for example, I have not mentioned the concept of a probability space). If you get familiar with the concepts of probability measures, random variables, p.m.f., p.d.f., c.d.f., etc., and how they are related, then you will start to get a better feeling of the whole picture.

A probability distribution in ML is the same as a probability distribution elsewhere.
A probability distribution (or probability function, or probability mass function, or probability density function) is any function that accepts as input elements of some specific set $x \in X$, and produces as output, real-valued numbers between 0 and 1 (inclusive), such that $\int_{x \in X} p(x) = 1$ or for discrete sets, $\sum_{x \in X} p(x) = 1$.
These distributions can also be more complex. For example, a conditional probability distribution $P(Y|X)$ or a joint probability distribution $P(X,Y)$ accept more than one input, but again, are constrained to producing an output in the range 0 to 1, and to ensuring that the summation of the output over all possible inputs is exactly 1.
When these conditions are met, the functions output can be interpreted as a belief about the percentage of times the input event will occur, out of all events, or as a degree of believe in the input event having occurred vs. other events (i.e. it can be interpreted as a probability).

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How can I formulate the k-knights problem as a constraint satisfaction problem?
There are three things in every constraint satisfaction problem (CSP):

Variables
Domain
Constraints

In the given scenario, I know how to identify the constraints, but I don't know how to identify the variables and the domain.
The given scenario is:

You are given a $n \times n$ board, where $n \geq 3$. On this board, you have to put $k$ knights where $k < n^2$, such that no knight is attacking the other knight. The knights are expected to be placed on different squares on the board. A knight can move two squares vertically and one square horizontally or two squares horizontally and one square vertically. The knights attack each other if one of them can reach the other in a single move. For example, on a $3 \times 3$ board, we can place $k=5$ knights.

So, the input is $m = 3, n = 3, k = 5$. There are two solutions.
Solution 1
K A K
A K A
K A K

Solution 2
A K A
K K K
A K A

There are many possible ways to encode this problem, and some will be more advantageous than others
An encoding that seems like a reasonable starting point to me is:

Variables: Let $S$ be a set of $k$ variables representing the coordinates of knights on the chess board.
Domain: The domain of each variable is initially all vectors in $[n]^2$. Denote the components of these vectors with $[r, c]$.
Constraints:

Let A be the set of 8 attacking moves: $A=\{[2,1],[2,-1],[-2,1],[-2,-1],[1,2],[1,-2],[-1,2],[-1,-2]\}$
$\forall_{i < j} S_i \neq S_j$ (no knights on the same squares)
$\forall_{i < j} [r_i, c_i] \notin \{[r_j + x, c_j + y] | x,y \in A\}$ (no knight attacks another knight)

That should be it. Note the use of $\forall_{i<j}$ in the constraints is to reduce the total number of constraints by half, exploiting the symmetry that knight $i$ can attack knight $j$ iff knight $j$ can attack knight $i$. You could also use $\forall_{i\neq j}$, but it would increase your constraint count to no gain.


text:
Can CNNs be applied to non-image data, given that the convolution and pooling operations are mainly applied to imagery?
When using CNNs for non-image (times series) data prediction, what are some constraints or things to look out for as compared to image data?
To be more precise, I notice there are different types of layers in a CNN model, as described below, which seem to be particularly designed for image data.
A convolutional layer that extracts features from a source image. Convolution helps with blurring, sharpening, edge detection, noise reduction, or other operations that can help the machine to learn specific characteristics of an image.
A pooling layer that reduces the image dimensionality without losing important features or patterns.
A fully connected layer also known as the dense layer, in which the results of the convolutional layers are fed through one or more neural layers to generate a prediction.
Are these operations also applicable to non-image data (for example, times series)?

Usually, you need to ensure that your convolutions are causal, meaning that there is no information leakage from the future into the past. You could start by looking at this paper, which compares Temporal Convolutional Networks (TCN) with vanilla RNNs models.

You can use CNN for time-series data. The Convolutional Recurrent Neural Network (RCNN) is one of the examples.
Convolutional layers basically extract features from images. It is not related to time-series data processing.
Some CNNs (such as in ResNet, Highway Networks, and DenseNet) use some recurrent concepts to improve their prediction, but they all are within single datapoint reasoning. You can go through these concepts to improve your intuitions.

Maybe you should provide a link to a paper that describes the RCNN that you were thinking of.


text:
Eligibility vector for softmax policy with policy gradients
There is this nice result for policy gradients that the gradient of some performance measure, $\nabla v_{\pi_{\theta}}(s_0)$ (here, in the episodic case for the starting state $s_0$ and policy $\pi$, parametrised by some weights $\theta$) is equal to the expectation gradient of the logarithm of the policy, i.e.
$$\nabla v_{\pi_{\theta}}(s_0)=\mathbb{E}\Big{[}\sum_{t=0}^{T-1}\nabla_\theta\log(\pi_{\theta}(a_t|s_t)]\cdot G_t\Big{]},$$
where $G_t$ is the discounted future reward from state $s_t$ onward and $s_T$ the final state of some trajectory $(s_0, a_0, s_1, a_1, ..., s_{T-1}, a_{T-1}, s_T)$.
Now, when using a softmax policy, $\nabla_\theta\log(\pi_{\theta}(a_t|s_t)$ can be written as
$$\nabla_\theta\log(\pi_{\theta}(a_t|s_t))=\phi(s_t,a_t)-\mathbb{E}[\phi(s_t,\cdot)],$$
where $\phi(s,a)$ is some input vector of a state-action tuple.
However: what exactly is this vector? A typical input with policy gradients (for example in a neural network) is a feature vector for the state and the output a vector with dimensions equal to the number of actions, e.g. $(14, 15, 11, 17)^T$ for four possible actions. The softmax-function now scales these outputs, which results in the logits $(.042, .114, .002, .842)^T$ in this example.
What I would usually do in neural networks is take some input vector, for example something that describes if there are borders in a grid world, e.g. $\phi(s)=(1, 0, 0, 1)^T$, and multiply that with my weight matrix $\theta$ (and add biases b), i.e. $\theta\phi(s)+b$. So, continuing above example, $1\cdot \theta_{1,1} + 0\cdot \theta_{1,2} + 0\cdot \theta_{1,3} + 1\cdot \theta_{1,4} = 14$ and $1\cdot \theta_{2,1} + 0\cdot \theta_{2,2} + 0\cdot \theta_{2,3} + 1\cdot \theta_{2,4} = 15$.
But what is $\phi(s,a)$ here? And how would I compute $\nabla_\theta\log(\pi_{\theta}(a|s))=\phi(s,a)-\mathbb{E}[\phi(s,\cdot)]$?

Calculation of gradient
\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
&= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}
is only valid for linear function approximation with action preferences of form
\begin{equation}
h(s, a, \theta) = \theta^T \phi(s,a)
\end{equation}
and softmax policy
\begin{equation}
\pi(a|s) = \frac{e^{h(s,a,\theta)}}{\sum_{a'} e^{h(s,a',\theta)}}
\end{equation}
The gradient would be calculated as it is written. For example, if your current state is $s = (1, 1)$ and in that state you have actions $a_0 = 0$ and $a_1 = 1$ and probabilities for those actions are $\pi(a_0|s) = 0.7$, $\pi(a_1|s) = 0.3$ then gradient for action $a_0$ would be
\begin{equation}
\nabla_{\theta} \log(\pi_{\theta}(a_0|s)) = (1, 1, 0)^T - (0.7 \cdot (1, 1, 0)^T + 0.3\cdot (1,1,1)^T) = (0, 0, -0.3)^T
\end{equation}
Feature vector $\phi$ can be basically anything you want. For example you could stack state feature and action (like I did in small example), you could use polynomials, radial basis functions, tile coding, etc.
If you're using multilayered neural network you would have to propagate gradients through all layers, usually done with backpropagation algorithm. Easiest way is to use automatic differentiation software (e.g. Tensorflow) which can do that for you so you don't have to write your implementation. All you have to do is define your objective function that you want to optimize
\begin{equation}
J_\theta = \sum_t \log(\pi(a_t|s_t, \theta)) G_t
\end{equation}
and software will automatically calculate gradient $\nabla J_{\theta}$ and update weights.

Thanks a lot! Just for the sake of completion I wanted to add the "likelihood ratios trick", i.e. $\nabla_\theta \log{\pi_\theta (a|s)}=\frac{\nabla_\theta \pi_\theta (a|s)}{\pi_\theta (a|s)}$, that could be used to derive the gradient of the log by hand.


text:
Why is the stationary distribution independent of the initial state in the proof of the policy gradient theorem?
I was going through the proof of the policy gradient theorem here: https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#svpg
In the section "Proof of Policy Gradient Theorem" in the block of equations just under the sentence "The nice rewriting above allows us to exclude the derivative of Q-value function..." they set
$$
\eta (s) = \sum^\infty_{k=0} \rho^\pi(s_0 \rightarrow s, k)
$$
and
$$
\sum_s \eta (s) = const
$$
Thus, they basically assume, that the stationary distribution is not dependent on the initial state. But how can we justify this? If the MDP is described by a block diagonal transition matrix, in my mind this should not hold.

I think your doubt is completely reasonable. Probably there is an additional assumption that they (both Lilian Weng and Rich Sutton (pag.269)) do not make explicit in the proof and that is that your MDP is not only stationary, but also ergodic. A particular property of those systems is that the probability of eventually reaching a state $s$ from a starting point $s_0$ is 1. In such a case it is clear that $\eta(s)$ exists and is independent of any $s_0$ chosen.
Clearly, an MDP with block-diagonal transition matrix does not satisfy such an assumption since the starting point completely restricts those states you can reach in an infinite time.
What I do not understand is why Rich Sutton does mention ergodicity as a necessary condition in the case of a "continuing task", as opposed to "episodic tasks" (pag.275). For me, their proof requires this condition in both cases.
As an additional note, I also think that Lilian Weng does not really explain why we should buy that from the initial reasonable definition $J(\theta)=\sum_\mathcal S d^{\pi_\theta}(s)V^{\pi_\theta}(s)$ we should accept the much simpler one $J(\theta)=V^{\pi_\theta}(s_0)$. I guess the only reason is that the gradient of the initial expression does require to know the gradient of $d^{\pi_\theta}(s)$ and so you would be accepting the approximation:
$$\nabla_\theta J(\theta)=\nabla_\theta\left(\sum_\mathcal S d^{\pi_\theta}(s)V^{\pi_\theta}(s)\right)\approx\sum_\mathcal S d^{\pi_\theta}(s)\nabla_\theta V^{\pi_\theta}(s),$$
where the last term is just $\nabla_\theta V^{\pi_\theta}(s_0)$ under the ergodicity assumption.

Yes, the second part you mentioned left me also a bit confused. I don't quite see why $\sum d^\pi (s) V^\pi (s) = V^\pi (s_0) $ should be true, even if the system is ergodic. Generally, $V^\pi(s_0)$ makes more sense to me since in practice we always start from $s_0$ and don't sample $s_0$ from $d^\pi (s)$.

I'm not really saying that the expected value of $V^{\pi}(s)$ is $V^\pi(s_0)$. What is true (if the system is ergodic) is that the expected value of $\nabla V^{\pi}(s)$ is $ \nabla V^\pi(s_0)$, given that the gradient is the same for all states since $\eta(s)$ does not depend on $s_0$.

About what performance metric is better, I consider that usually you want your policy to be robust, so you not only want to consider one single $s_0$. What happens then if by some perturbance your agent is forced to start in a different state? At least you should consider an initial distribution $\rho(s_0)$. This is done in the TRPO paper, for example.


text:
What is the relationship between the training accuracy and validation accuracy?
During model training, I noticed various behaviour in between training and validation accuracy. I understand that 'The training set is used to train the model, while the validation set is only used to evaluate the model's performance...', but I'd like to know if there is any relationship between training and validation accuracy and, if yes,

what exactly is happening when training and validation accuracy change during training and;

what do different behaviours imply

For instance, some believe there is overfitting problem if training > validation accuracy. What happens if one is greater than the other alternately, which is the case below?
Here is the code
inputs_1 = keras.Input(shape=(10081,1))

layer1 = Conv1D(64,14)(inputs_1)
layer2 = layers.MaxPool1D(5)(layer1)
layer3 = Conv1D(64, 14)(layer2)
layer4 = layers.GlobalMaxPooling1D()(layer3)

inputs_2 = keras.Input(shape=(104,))
layer5 = layers.concatenate([layer4, inputs_2])
layer6 = Dense(128, activation='relu')(layer5)
layer7 = Dense(2, activation='softmax')(layer6)

model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])
model_2.summary()

X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10185], df[['Result_cat','Result_cat1']].values, test_size=0.2)
X_train = X_train.to_numpy()
X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1])
X_train_1 = X_train[:,0:10081,:]
X_train_2 = X_train[:,10081:10185,:].reshape(736,104)

X_test = X_test.to_numpy()
X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1])
X_test_1 = X_test[:,0:10081,:]
X_test_2 = X_test[:,10081:10185,:].reshape(185,104)

adam = keras.optimizers.Adam(lr = 0.0005)
model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])

history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])

model summary
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:15: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[<tf.Tenso..., outputs=[<tf.Tenso...)`
  from ipykernel import kernelapp as app
Model: "model_3"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_5 (InputLayer)            (None, 10081, 1)     0
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 10068, 64)    960         input_5[0][0]
__________________________________________________________________________________________________
max_pooling1d_3 (MaxPooling1D)  (None, 2013, 64)     0           conv1d_5[0][0]
__________________________________________________________________________________________________
conv1d_6 (Conv1D)               (None, 2000, 64)     57408       max_pooling1d_3[0][0]
__________________________________________________________________________________________________
global_max_pooling1d_3 (GlobalM (None, 64)           0           conv1d_6[0][0]
__________________________________________________________________________________________________
input_6 (InputLayer)            (None, 104)          0
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 168)          0           global_max_pooling1d_3[0][0]
                                                                 input_6[0][0]
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 128)          21632       concatenate_3[0][0]
__________________________________________________________________________________________________
dense_6 (Dense)                 (None, 2)            258         dense_5[0][0]
==================================================================================================
Total params: 80,258
Trainable params: 80,258
Non-trainable params: 0

and the training process
__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041

Notice how at first acc is lower than val_acc and later is greater than val_acc. Can someone please shed some light what could be happening here? Thank you

very interesting questions:
1. what exactly is happening when training and validation accuracy change during training

The accuracy change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of the current batch averaged with the accuracy of all the previous batches so far. See keras.utils.generic_utils.Progbar.
The val_acc is computed only at the end of one epoch and it is computed with all your validation dataset at once (considering it as a single batch, so if you have 100 images for validation it will compute accuracy as a single batch of 100 images)

2. what do different behaviours imply

The acc and val_acc normally differ from each other due to different split sizes.

Try same experiment with validation_split=0.01 and validation_split=0.4 and you will see how both accuracy and val_acc will change.
Normally the greater the validation split, the more similar both metrics will be since the validation split will be big enough to be representative (let's say it has cats and dogs, not only cats), taking into account that you need enough data to train correctly. This explains why in some cases the val_acc is higher than accuracy and vice versa.

Overfitting only occurs when the graph fashion or tendency changes and val_acc starts to drop and accuracy keeping increasing. This means that your model can not do any better with the validation dataset (non previously seen images).

I work with loss and val_loss which are highly correlated with accuracy. Normally the loss is the inverse, so interpret the comments above in the inverse sense (sorry about the confusion but I'm taking this example from my current experiments) I hope it helps:

There are 2 experiments, orange and grey.

In both experiments, val_loss is always slightly higher than loss (because of my current validation split which it happens to be also 0.2, but normally is 0.01 and val_loss is even higher).

On both experiments the loss trend is linearly decreasing, this is because gradient descent works and the loss functions is well defined and it converges.

Orange experiment is overfitting from epoch 20 onwards because, the val_loss won't drop any more and, on the contrary, it start increasing.

Grey experiment is just right, both loss and val_loss are still decreasing, and although the val_loss might be greater than loss it is not overfitting because it is still decreasing. So that is why it is still training :)

Complex concepts here, I hope I was able to explain myself clearly! Cheers

Thank you for the detailed explanation! Just a few questions: 1) If loss is computed after each batch and val_acc is computed only at the end of one epoch, does val_acc/accuracy reflect the accuracy of the entire model or only the batch? 2) Acc vs loss: when one is increasing, does the other necessarily have to decrease proportionally? Thank you

In Keras after computing the accuracy for the batch, it is averaged with the accuracy from all the previous batches, so yes, it takes them all into account and hence it is representative for the accuracy of your model. Same with val_acc (but with only one batch)
In 99% of cases yes, but it depends on how you define loss and accuracy (in standard keras metrics they are inversely proportional)

I edited the response to add this

Also note that using mode.evaluate to the training data may give you a different loss than what is printed during model.fit, since fit calculates the loss after each epoch separately and the model's parameters are being updated in-between.


text:
Does the neural network calculate different relations between inputs automatically?
Suppose you want to predict the price of some stock. Let's say you use the following features.
OpenPrice
HighPrice
LowPrice
ClosePrice

Is it useful to create new features like the following ones?
BodySize = ClosePrice - OpenPrice

or the size of the tail
TailUp = HighPrice - Max(OpenPrice, ClosePrice)

Or we don't need to do that because we are adding noise and the neural network is going to calculate those values inside?
The case of the body size maybe is a bit different from the tail, because for the tail we need to use a non-linear function (the max operation). So maybe is it important to add the input when it is not a linear relationship between the other inputs not if it's linear?
Another example. Consider a box, with height $X$, width $Y$ and length $Z$.
And suppose the real important input is the volume, will the neural network discover that the correlation is $X * Y * Z$? Or we need to put the volume as input too?
Sorry if it's a dumb question but I'm trying to understand what is doing internally the neural network with the inputs, if it's finding (somehow) all the mathematically possible relations between all the inputs or we need to specify the relations between the inputs that we consider important (heuristically) for the problem to solve?

On paper, one expects a complex enough network to determine any complicated function of a limited number of inputs, given a large enough dataset. But in practice, there is no limit to the possible difficulty of the function to be learnt, and the datasets can be relatively small on occasion. In such cases - or arguably in general - it is definitely a good idea to define some combination of the inputs depending on some heuristics as you suggested. If you think some combination of inputs is an important variable by itself, you definitely should include it in your inputs.
We can visualize this situation in TensorFlow playground. Consider the circular pattern dataset on top left corner with some noise. You can use the default setting: $x_1$ and $x_2$ as inputs with 2 hidden layers with 4 and 2 neurons respectively. It should learn the pattern in less than 100 epochs. But if you reduce the number of neurons in the second layer to 2, it is not going to get as good as before. So, you are making the model more complicated to get the correct answer.
You can experiment and see that one needs at least one 3 neuron layer to get the correct classification from just $x_1$ and $x_2$. Now, if we examine the dataset, we see the circles so we know that instead of $x_1$ and $x_2$, we can try $x_1^2$ and $x_2^2$. This will learn perfectly without any hidden layers as the function is linear in these parameters. The lesson to be learnt here is that, our prior knowledge of the circle ($x_1^2 + x_2^2 = r^2$)  and familiarity with the data helped us in getting a good result with a simpler model (smaller number of neurons), by using derived inputs.
Take the spiral data at the lower right corner for a more challenging problem. For this one, if you do not use any derived features, it is not likely to give you the correct result, even with several hidden layers. Keep in mind that every extra neuron is a potential source of overfitting, on top of being a computational burden.
Of course the problem here is overly simplified but I expect the situation to be more or less the same for any complicated problem. In practice, we do not have infinite datasets or infinite compute times and the model complexity is always a restriction, so if you have any reason to think some relation between your inputs is relevant for your final result, you definitely should include it by hand at the beginning.

The question is related to "feature extraction". Firstly, to tackle a regression problem like both the problems stated by you, you need to provide the neural network with the most relevant inputs that have a effect on the output. Eg. If you want your network to add x and y, you need to provide it training examples like  input(x=1, y=3)  and output (sum=4). This will make your network do exactly what you want.
But suppose you do not know whether what inputs should you train your network on, neural networks can take care of that too. Look at this example:
Look at the first truth table. Notice that the output column is actually the first input column and the other two input columns are just random. Eventually, the network learns this relationship and provides the correct results. What we learnt: if you are unsure about which inputs should you choose for your network, just provide as many as possible, or as many combinations as possible. Neural networks excel in finding relationships in input data.
Next, talking of the volume problem, this is what I have been doing recently. It's actually an example of function approximation. Usually, the problem has multiple inputs and a single output (just like the addition problem), but the inverse is also possible. i.e., input : sum and output: x & y. This comes under one to many function mapping and multivariate regression. So YES, you need to provide the volume as input and x,y and z as outputs while training. The recommended configuration is one neuron in input layer, at least 6 hidden neurons and 3 neurons in output layer For magical results, you can use a deeper neural network rather than the shallow one suggested by me. But remember, neural networks have been proved to be **Universal Approximators*


text:
How can I minimise the false positives?
I have 50,000 samples. Of these 23,000 belong to the desired class $A$. I can sacrifice the number of instances that are classified as belonging to the desired class $A$. It will be enough for me to get 7000 instances in the desired class $A$, provided that most of these instances classified as belonging to $A$ really belong to the desired class $A$. How can I do this?
The following is the confusion matrix in the case the instances are perfectly classified.
[[23000   0]
 [  0 27000]]

But it is unlikely to obtain this confusion matrix, so I'm quite satisfied with the following confusion matrix.
[[7000   16000]
 [  500 26500]]

I am currently using the sklearn library. I mainly use algorithms based on decision trees, as they are quite fast in the calculation.

Hi and welcome to this community! I've reformulated part of your post to make it clearer. Make sure I've not changed the intended meaning.

I have two classes. I'm only interested in the first one.
As a rule the desired class is called positive.
It is important to me that in the positive class got as
little as possible false negative labels that are predicted.
I added two examples with a confusion matrix.

That's what I thought. I've provided an answer below. I may edit it later to provide a complete example.

I think you're looking for the minimization of false positives, that is, the instances that are classified as belonging to the desired class (the positive part of false positives) but that do not actually belong to that class (the false part of false positives). In practice, given your constraints, you may want to maximize the precision, while maintaining a good recall.
In this answer to the question How can the model be tuned to improve precision, when precision is much more important than recall?, the user suggests performing a grid search (using sklearn.grid_search.GridSearchCV(clf, param_grid, scoring="precision")) to find the parameters of the model that maximize the precision. See also the question Classifier with adjustable precision vs recall.

Thanks! Some things already done.
I have an opinion that basically algorithms try to separate both classes to the maximum.
I don't have to do that, it's enough that as few false negatives as possible get into the desired class.


text:
How are the inputs passed to the neural network during training for the XOR classification task?
Let's suppose we have to train a neural network for the XOR classification task.
Are the inputs $(00, 01, 10, 11)$ inserted in a sequential way? For example, we first insert the 00 and change the weights, then the 01 and again slightly change them, etc. Or is there another way it can be implemented?

Hello and welcome to the community. There are multiple ways you can train a neural network: stochastic, mini-batch and batch. What you explained is the stochastic mode, where you input one training example 01 for example, calculate the gradients and update the networks weights before the next training example is fed. You could also select multiple such examples (a mini-batch) and update the weights only after you computed all the outputs (for this particular mini-batch). Finally you can use a batch size which is equal with the total number of examples in your dataset so you will update the weights only after you have all the outputs for all samples. Each of those methods have their own strengths and weaknesses, depending on your dataset you might prefer one over the others.


text:
Concrete example of latent variables and observables plugged into the Bayes' rule
In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule
$$p_{\theta}(z|x)=\frac{p_{\theta}(x|z)p(z)}{p(x)}$$
for a given latent variable and observable?
I understand with VAE's we're essentially getting an approximation to $p_{\theta}(z|x)$ that models the distribution that we think approximates the latent variables, but I need a concrete example to really understand it.

Let's assume the probability distributions are Gaussian (or normal) distributions. In other words, in the Bayes' rule
\begin{align}
p(z|x)=\frac{p(x|z)p(z)}{p(x)}
\tag{1}\label{1}
\end{align}
The posterior $p(z|x)$, the likelihood $p(x|z)$, the prior $p(z)$ and the evidence (or marginal) $p(x)$ are Gaussian distributions. You can assume this because Gaussian distributions are closed under conditioning and marginalization.
For simplicity, let's further assume that they are univariate Gaussian distributions. Given that the Gaussian distribution is a continuous probability distribution, it has an associated probability density function (rather than a probability mass function, which is associated with discrete probability distributions, such as the Bernoulli distribution). The probability density function of the Gaussian distribution is
\begin{align}
f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} } \tag{2}\label{2}
\end{align}
where $\mu$ and $\sigma^2$ are respectively the mean and variance of the Gaussian distribution and $x$ is a variable (similarly to a variable $x$ in any mathematical function $f(x)$). So, given a concrete value for $x$, for example, $x=1$, then $f(x=1, \mu, \sigma^2)$ is a so-called density value (rather than a probability, which a probability mass function returns, given an input). For example, let's assume that the mean $\mu=0$ and the variance $\sigma^2 = 1$, then, for $x=1$, the density will be
$$
f(1 \mid 0, 1) = \frac{1}{\sqrt{2\pi} } e^{ -\frac{1}{2} }
$$
So, to obtain the concrete density value, I've just replaced the concrete values of $x$, $\mu$ and $\sigma^2$ in equation \ref{2}.
To calculate the posterior $p(z|x)$ in equation \ref{1}, you just need to replace the likelihood $p(x|z)$, the prior $p(z)$ and the evidence $p(x)$ with the Gaussian probability density shown in equation \ref{2}, so you will have
\begin{align}
p(z|x)=\frac{f_{X\mid Z}(x \mid \mu_{X\mid Z}, \sigma^2_{X\mid Z}, z) f_{Z}(z \mid \mu_{Z}, \sigma^2_{Z})}{f_{X}(x \mid \mu_{X}, \sigma^2_{X})}
\tag{3}\label{3}
\end{align}
I've explicitly added a subscript to the means and variances of each probability density, given that, for example, the mean of the probability density $f_{X\mid Z}$ might be different than the mean of the probability density $f_{Z}$ or $f_{X}$, etc. So, to get the actual density value (a real number) that represents $p(z|x)$, you just need to replace $f_{X\mid Z}$, $f_{Z}$ and $f_{X}$ with the definition of the Gaussian density function in \ref{2} with their actual mean and variance values. I'll let you do this, given that this is really just a matter of picking up a concrete value for the means and variances and doing some algebra.
If you assume the posterior, the likelihood, the prior or the evidence to have a different distribution, you will do the same thing, but using the probability density or mass function of your chosen distribution.
In the context of the variational auto-encoder, you will be learning the mean and variance of the distribution, so the mean and variance will not be fixed, but they will be the parameters that you want to find. However, this does not change the way you apply the Bayes' rule.


text:
How does maximum approximation of the posterior choose a distribution?
I was learning about the maximum a posteriori probability (MAP) estimation for machine learning and I found a nice short video that essentially explained it as finding a distribution and tweaking the parameters to fit the observed data in a way that makes the observations most likely (makes sense).
However, in mathematical terms, how does it determine which distribution best fits the data?
There are so many distributions out there that it could be any of them and the parameters you could fit them could be infinitely large.

Introduction: MAP finds a point estimate!
As opposed to your apparently current belief, in maximum a posteriori (MAP) estimation, you are looking for a point estimate (a number or vector) rather than a full probability distribution. The MAP estimation can be seen as a Bayesian version of the maximum likelihood estimation (MLE). Therefore, I will first remind you of the objective of MLE.
Maximum likelihood estimation (MLE)
Let $\theta$ be the parameters you want to find. For example, $\theta$ can be the weights of your neural network. In MLE, we want to find a point estimate (rather than a full distribution). The objective in MLE is
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) \tag{1}\label{1}
\end{align}
where $p(X \mid \theta)$ is the likelihood of the data $X$ given the parameters $\theta$. In other words, we want to find the parameters $\theta$ such that $p(X \mid \theta)$ is the highest, where $X$ is your given training data, so $X$ is fixed.
The notation $p(X \mid \theta)$ can be confusing because, in a conditional probability distribution, $p(a\mid b)$, we often assume that $b$ is given and $p(a\mid b)$ is a distribution over $a$. However, in the case of MLE, $\theta$ in $p(X \mid \theta)$ is not fixed, but it is a variable, while $X$ is given and fixed. Hence we call $p(X \mid \theta)$ a likelihood rather than a probability density or mass function. Moreover, we often denote the likelihood as $\mathcal{L}(\theta; X) = p_{\theta}(X)$ (and there are other notations, but this is, in my opinion, the least confusing one), because we want to emphasize that the likelihood is actually a function of the variable $\theta$. However, this is notation can also be confusing because we equate a function of a variable $\theta$ to a probability distribution over $X$. However, you should note that $p_{\theta}(X)$ is parametrized by $\theta$.
Therefore, the MLE estimation \ref{1} can also be written as follows
\begin{align}
\theta^*
&=
\operatorname{argmax}_\theta \mathcal{L}(\theta; X) \\
&=\operatorname{argmax}_\theta p_{\theta}(X)
\tag{2}\label{2}
\end{align}
where $\theta^*$ is the point estimate of the objective function.
This notation emphasizes the fact that we want to find $\theta$, such that the probability of the given data $X$ is maximized.
Maximum a posteriori (MAP)
MAP is similar to MLE, but the objective is slightly different. First of all, we assume that $\theta$ is a random variable, so we have an associated probability distribution $p(\theta)$.
Recall that the Bayes' rule is the following
\begin{align}
p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{p(X)}
\tag{3}\label{3}
\end{align}
The objective function in MAP estimation is
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta \frac{p(X \mid \theta) p(\theta)}{p(X)} \\
&= \operatorname{argmax}_\theta p(\theta \mid X) \tag{4}\label{4}
\end{align}
Given that $p(X)$ does not depend on $\theta$, for the purposes of optimization, we can ignore it, so equation \ref{4} becomes
\begin{align}
\theta^*
&= \operatorname{argmax}_\theta p(X \mid \theta) p(\theta) \\
&= \operatorname{argmax}_\theta p(\theta \mid X)
\tag{5}\label{5}
\end{align}
which is the MAP objective.
What is the relationship between MLE and MAP?

In MAP, the objective is \ref{5}, which includes a prior over $\theta$, while, in MLE, equation \ref{1}, there is no such thing.

Therefore, in MAP, we can assume that the parameters $\theta$ follow a certain distribution, thanks to the usage of $p(\theta)$.

In both MAP and MLE, we want to find a point estimate (which can be a number, if you have just one parameter, or a vector of size $N$, if you have $N$ parameters).
MAP is equivalent to MLE if you use a uniform prior, that is, if $p(\theta)$ is a uniform distribution.

Which distribution fits the data $X$?
In MAP, the human (you, me, etc.) chooses the family of distributions. For example, you can assume that your parameters $\theta$ follow a Gaussian distribution, so $p(\theta)$ will be a Gaussian distribution over the parameters. Why do I say "family"? For example, in the case of a Gaussian distribution, you have two parameters that control the shape of the distribution, namely, the mean and variance. Depending on the concrete values of these two parameters, you will have different Gaussian distributions, so you call all these Gaussian distributions a family.
How do you find $\theta$?
To find $\theta^*$, you can use an optimization method like gradient descent or, in certain cases, you can find a closed-form solution. See also Which distributions have closed-form solutions for maximum likelihood estimation?.
Resources
The following blog post MLE vs MAP: the connection between Maximum Likelihood and Maximum A Posteriori Estimation, by Agustinus Kristiadi (a Ph.D. student in machine learning), might also be useful, so I suggest you read it. It will give you more details that I've left out on purpose to avoid cluttering this answer.


text:
What are the state-of-the-art results on the generalization ability of deep learning methods?
I've read a few classic papers on different architectures of deep CNNs used to solve varied image-related problems. I'm aware there's some paradox in how deep networks generalize well despite seemingly overfitting training data. A lot of people in the data science field that I've interacted with agree that there's no explanation on why deep neural networks work as well as they do.
That's gotten me interested in the theoretical basis for why deep nets work so well. Googling tells me it's kind of an open problem, but I'm not sure of the current state of research in answering this question. Notably, there are these two preprints that seem to tackle this question:

Generalization in Deep Learning (2019)
Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness (2019)

If anyone else is interested in and following this research area, could you please explain the current state of research on this open problem? What are the latest works, preprints or publications that attempt to tackle it?

Introduction
The paper Generalization in Deep Learning provides a good overview (in section 2) of several results regarding the concept of generalisation in deep learning. I will try to describe one of the results (which is based on concepts from computational or statistical learning theory, so you should expect a technical answer), but I will first introduce and describe the general machine learning problem and I will give a definition of the generalisation gap problem. To keep this answer relatively short, I will assume the reader is familiar with certain basic machine learning and mathematical concepts, such as expected risk minimization, but, nonetheless, I will refer the reader to more detailed explanations of the concepts (at least the first time they are mentioned). If you are familiar with the basic concepts of computational learning theory (e.g. hypotheses), you will be advantaged.
Machine Learning Problem
In the following description, unless stated otherwise, I do not make any assumption about the nature of the variables. However, I will occasionally provide examples of concrete values for these variables.
Let $x \in \mathcal{X}$ be an input and let $y \in \mathcal{Y}$ be a target. Let $\mathcal{L}$ be a loss function (e.g. MSE).
Then the expected risk of a function (or hypothesis) $f$ is defined as
\begin{align}
R[f]
&= \mathbb{E}_{x, y \sim \mathbb{P}(X, Y)} \left[ \mathcal{L}(f(x), y) \right] \\
&= \int \mathcal{L}(f(x), y) d\mathbb{P}(X=x, Y=y),
\end{align}
where $\mathbb{P}(X, Y)$ is the true joint probability distribution of the inputs and targets. In other words, each $(x, y)$ is drawn from the joint distribution $\mathbb{P}(X, Y)$, which contains or represents all the information required to understand the relationship between the inputs and the targets.
Let $A$ be a learning algorithm or learner (e.g. gradient descent), which is the algorithm responsible for choosing a hypothesis $f$ (which can e.g. be represented by a neural network with parameters $\theta$). Let
$$S_m = \{(x_i, y_i) \}_{i=1}^m$$
be the training dataset. Let
$$f_{A(S_m)} : \mathcal{X} \rightarrow \mathcal{Y}$$
be the hypothesis (or model) chosen by the learning algorithm $A$ using the training dataset $S_m$.
The empirical risk can then be defined as
$$
R_{S_m}[f] = \frac{1}{m} \sum_{i=1}^m \mathcal{L} (f(x_i), y_i)
$$
where $m$ is the total number of training examples.
Let $F$ be the hypothesis space (for example, the space of all neural networks).
Let
$$
\mathcal{L_F} = \{ g : f \in F , g(x, y) = \mathcal{L}(f(x), y)\}
$$ be a family of loss functions associated with the hypothesis space $F$.
Expected Risk Minimization
In machine learning, the goal can be framed as the minimization of the expected risk
\begin{align}
f^*_{A(S_m)}
&= \operatorname{argmin}_{f_{A(S_m)}} R[f_{A(S_m)}] \\
&= \operatorname{argmin}_{f_{A(S_m)}} \mathbb{E}_{x, y \sim \mathbb{P}(X, Y)} \left[ \mathcal{L}(f_{A(S_m)}(x), y) \right] \tag{1}\label{1}
\end{align}
However, the expected risk $R[f_{A(S_m)}]$ is incomputable, because it is defined as an expectation over $x, y \sim \mathbb{P}(X, Y)$ (which  is defined as an integral), but the true joint probability distribution $\mathbb{P}(X, Y)$ is unknown.
Empirical Risk Minimization
Therefore, we solve the approximate problem, which is called the empirical risk minimization problem
\begin{align}
f^*_{A(S_m)} &= \operatorname{argmin}_{f_{A(S_m)} \in F} R_S[f_{A(S_m)}] \\
&=
\operatorname{argmin}_{f_{A(S_m)} \in F} \frac{1}{m} \sum_{i=1}^m \mathcal{L} (f_{A(S_m)}(x_i), y_i)
\end{align}
Generalization
In order to understand the generalization ability of $f_{A(S_m)}$, the hypothesis chosen by the learner $A$ with training dataset $S_m$, we need to understand when the empirical risk minimization problem is a good proxy for the expected risk minimization problem. In other words, we want to study the following problem
\begin{align}
R[f_{A(S_m)}] -  R_S[f_{A(S_m)}] \tag{2}\label{2}
\end{align}
which can be called the generalization gap problem. So, in generalization theory, one goal is to study the gap between the expected and empirical risks.
Clearly, we would like the expected risk to be equal to the empirical risk $$R_S[f_{A(S_m)}] = R[f_{A(S_m)}]$$ because this would allow us to measure the performance of the hypothesis (or model) $f_{A(S_m)}$ with the empirical risk, which can be computed. So, if $R_S[f_{A(S_m)}] = R[f_{A(S_m)}]$, the generalization ability of $f_{A(S_m)}$ roughly corresponds to $R_S[f_{A(S_m)}]$.
Therefore, in generalization theory, one goal is to provide bounds for the generalisation gap $R[f_{A(S_m)}] -  R_S[f_{A(S_m)}]$.
Dependency on $S$
The hypothesis $f_{A(S_m)}$ is explicitly dependent on the training dataset $S$. How does this dependency affect $f_{A(S_m)}$? Can we avoid this dependency? Several approaches have been proposed to deal with this dependency.
In the following sub-section, I will describe one approach to deal with the generalization gap problem, but you can find a description of the stability, robustness and flat minima approaches in Generalization in Deep Learning.
Hypothesis-space Complexity
In this approach, we try to avoid the dependency of the hypothesis $f_{A(S_m)}$ by considering the worst-case generalization problem in the hypothesis space $F$
$$
R[f_{A(S_m)}] -  R_S[f_{A(S_m)}] \leq \sup_{f \in F} \left( R[f] -  R_S[f] \right)
$$
where $\sup_{f \in F} \left( R[f] -  R_S[f] \right)$ is the supremum of a more general generalization gap problem, which is greater or equal to \ref{2}. In other words, we solve a more general problem to decouple the hypothesis (or model) from the training dataset $S$.
Bound 1
If you assume the loss function $\mathcal{L}$ to take values in the range $[0, 1]$, then, for any $\delta > 0$, with probability $1 - \delta$ (or more), the following bound holds
\begin{align}
\sup_{f \in F} \left( R[f] -  R_S[f] \right) \leq 2 \mathcal{R}_m \left( \mathcal{L}_F \right) + \sqrt{\frac{\log{\frac{1}{\delta}} }{2m}} \tag{3} \label{3}
\end{align}
where $m$ is the size of the training dataset, $\mathcal{R}_m$ is the Rademacher complexity of $\mathcal{L}_F$, which is the family of loss functions for the hypothesis space $F$ (defined above).
This theorem is proved in Foundations of machine learning (2nd edition, 2018) by Mehryar Mohri et al.
There are other bounds to this bound, but I will not list or describe them here. If you want to know more, have a look at the literature.
I will also not attempt to give you an intuitive explanation of this bound (given that I am also not very familiar with the Rademacher complexity). However, we can already understand how a change in $m$ affects the bound. What happens to the bound if $m$ increases (or decreases)?
Conclusion
There are several approaches to find bounds for the generalisation gap problem \ref{2}

Hypothesis-space complexity
Stability
Robustness
Flat minima

In section 2 of the paper Generalization in Deep Learning, bounds for problem \ref{2} are given based on the stability and robustness approaches.
To conclude, the study of the generalization ability of deep learning models is based on computational or statistical learning theory. There are many more results related to this topic. You can find some of them in Generalization in Deep Learning. The studies and results are highly technical, so, if you want to understand something, good knowledge of mathematics, proofs, and computational learning theory is required.

Thanks! While it does make sense, I'll continue looking for a concrete example to understand how $P(X,Y)$ contains or represents all the information required to understand the relationship between the inputs and the targets. Maybe I should also try asking this particular follow-up question on stats SE? Or maybe I'll see if I can find info on that in the book you linked to.

@ShirishKulhari $P(X, Y)$ is a probability distribution of both the random variables $X$ and $Y$. In other words, given a specific realisation of $X$ and $Y$, e.g. $X=x$ and $Y=y$, then $P(X=x, Y=y)$ tells you the probability (or density) that the random variables $X$ and $Y$ take the values $x$ and $y$ at the same time, so it tells you how probable these two realisations are, in this sense $P(X, Y)$ contains the way $X$ and $Y$ vary together (see the concept of independence). So, in a certain way, you can find a function $f(x) = y$ based on the probabilities that $x$ and $y$ appear together.

@ShirishKulhari Regarding your second question, where you say "but I'm not sure how the entire hypothesis space can change with change in". In my answer, I am not saying that the entire hypothesis space, denoted by $F$ (capital $f$), depends on $S$. I am just saying that the single hypothesis $f_{A(S_m)} \in F$, where $F$ is the hypothesis space, explicitly depends on the training dataset $S_m$, by definition. Have a look at how $f_{A(S_m)}$ is defined in my answer. So, there are hypothesis $f \in F$, which do not depend on $S_m$, but maybe depend on another dataset.


text:
A* and uniform-cost search are apparently incomplete
Consider the following diagram of a graph representing a search space.

If we start at $B$ and try to reach goal state $E$, the lowest-cost first search (LCFS) (aka uniform-cost search) algorithm fails to find a solution. This is because, $B$ selects $A$ over $C$ to expand as $f(A)=g(A)=36 < f(C)=g(C)=70$. $f(n)$ is the cost function of node $n$, and $g(n)$ is the cost of reaching node $n$ from the start state. Continuing further, from $A$, LCFS will now select $B$ to expand, which in turn will select $A$ again over $C$. This leads to an infinite loop. This shows LCFS is incomplete (not guaranteed to find a solution, if one exists).
For A*, we define $f(n)=g(n)+h(n)$, where $h(n)$ is the expected cost of reaching goal state from node $n$. If we define Manhattan distance ($L_0$ norm) for $h(\cdot)$, books (such as Artificial Intelligence: A Modern Approach (3rd Ed) by Stuart Russell and Peter Norvig) says A* is bound to find the solution (since it exists). However, I couldn't find how. Using, A*, $B$ will still select $A$ since $f(A)=36+(h(A)=40)=76 < f(C)=70+(h(C)=30+50)=150$. You see, this means, when $A$ expands back $B$, $B$ will again select $A$, and an infinite loop ensues.
What am I missing here?

You forgot to calculate and take into account the costs of the actual paths. You forgot to accumulate the cost of the edges for going forward and backward multiple times!
The evaluation function of uniform-cost search (UCS) is $f(n) = g(n)$, where $g(n)$ represents the cost of the path from the start node to $n$. The evaluation function of A* is $f(n) = g(n) + h(n)$, where $h(n)$ is an admissible heuristic. UCS is a special case of A*, with the admissible heuristic $h(n) = 0, \forall n$. The evaluation function is used to choose the next node to visit from the fringe, which is the set of nodes that can potentially be visited. Whenever we visit a node, we remove it from the fringe. To expand a node $X$ means to add the children of $X$ to the fringe. Whenever you visit a node, you will also expand it.
Let's apply UCS to your specific example. Initially, we check whether $B$ is a goal node or not. It is not, so we expand $B$. We can add $B$ to a list of visited (or expanded) nodes (graph search) or not (tree search). Let's use the tree search, so we will not be keeping track of the expanded nodes, which means that we could expand a node more than once. $B$ has two children, $A$ and $C$, so we add them to the fringe, $\mathbb{F} = \{A, C\}$. Should we now visit $A$ or $C$? We will visit the one with the smallest value of the evaluation function, which is $A$, given that $f(A)=g(A)=36 < f(C)=g(C)=70$, so we remove $A$ from the fringe, which is now $\mathbb{F} = \{ C \}$. Is $A$ a goal node? No, so we expand it, but it only has one child, $B$, so we add $B$ to the fringe, so $\mathbb{F} = \{ C, B \}$. The cost of the path to reach $B$ by going first to $A$ and then to $B$ is $f(B) = 36 + 36 = 72$ (given that you go back and forward on the same edge, so you need to accumulate the cost of these trips!) and $f(C) = 70$, so we visit $C$, so we remove it from the fringe, which is now $\mathbb{F} = \{ B \}$.
You should be able to work out the remaining search (I haven't actually done it). I suggest you watch the video Uniform Cost Search, by John Levine, who shows a concrete example of how UCS and A* work.


text:
Isn't deep fake detection bound to fail?
Deep fakes are a growing concern: the ability to credibly alter a video may have great (negative) impacts on our society. It is so much of a concern, that the biggest tech companies launched a specific challenge: https://deepfakedetectionchallenge.ai/.
However, from what I understand, most deep fake generation techniques rely on the use of adversarial models. One model generates a new image, while another model tries to detect if the image is doctored or not. Both models "learn" from being confronted with the other.
That being said, if a good deep fake detection model emerges (from the previous challenge, or not), wouldn't it be rendered useless almost instantly by learning from it in an adversarial setting?

Not necessarily it depends on the function of the problem space for both the GANs.
A real world example: a batter's reaction time and a pitchers max speed are actual bounded values based on genetics and physics. If the max speed a pitcher can pitch is greater than the max reaction time a human needs to effectively hit against them they will permanently be a better pitcher because the threshold of reaction time.
We don't yet know if a maximum threshold on realistic fake image generation exists or if a threshold on detection exists.
As both reach near perfect accuracy it could be that the amount of nodes needed to detect a nearly perfect generated image from real image is more neurons than atoms in the universe, or conversely the amount of nodes needed to generated a nearly perfect image could reach impossible proportion we won't know until we continue to make better and better networks that close in on the boundary of generation and detection of real vs fake images from a neural network.
Edit:
Let's imagine this problem. One adversary edits an image with a colored line of pixels the goal hide the line by editing the image, the student is responsible with finding the line after the adversary changes the image. The problem can become infinitely difficult change all pixels to be the color of the line. The line is impossible to find the adversary always wins, if it finds this solution meaning it is in its reachable problem space based on its hardware capabilities and its learning model. Which we should assume it would because it's such a simple task.
Deep fake detection is not bound to fail because the limit on the effectiveness of a generative model may have a steeper limit than a discriminator at near optimal performance. I have not seen any paper about this specifically and in fact I believe the discriminator has a more difficult job in most cases I just disagree with deciding at this moment that the detectors are doomed. The combination of creating real images motion and sound perfectly in sink is not a trivial problem, in some scenarios it is basically impossible.

Thanks for your contribution. However your batter / pitcher comparison seems a bit off as those actions require very different qualities (power versus reaction time... as far as I understand that sport). Do you have any source on why this kind of asymetry would arise between adversarial models ? (which I somewhat expect to perform similarly given comparable ressources - architecture, computation power ... etc.)

Good point about the cat-and-mouse game that is anything involving misuse of new technology and measures to counteract.  Hope to see more of you on SE:AI!

@DukeZhou Thanks, is there any topic specifically in SE:AI you'd want me to look out for? Or to look into?


text:
What are causative and exploratory attacks in Adversarial Machine Learning?
I've been researching Adversarial Machine Learning and I know that causative attacks are when an attacker manipulates training data. An exploratory attack is when the attacker wants to find out about the machine learning model. However, there is not a lot of information on how an attacker can manipulate only training data and not the test data set.
I have read about scenarios where the attacker performs an exploratory attack to find out about the ML model and then perform malicious input in order to tamper with the training data so that the model gives the wrong output. However shouldn't such input manipulation affect both the test and training data set? How does such tampering only affect the training data set and not the test data set?

When someone is able to do a causative attack it means there is a mechanism by which they are able to input data into the network. Maybe a website where people can input their images and it outputs a guess on what is in the picture and then you click if it got it right or not. If you continue to input images and lie to it it will obviously get worse and worse if they use the user input to add to the test set. Most people are careful and don't mess around with mixing new data into the testing sample. If they did something like mixed the user input training and test then resampled something like that could occur but most people don't do that. It's bad practice and even worse than leaving your NN open to tampering from malicious user input. Information isn't really added to the knowledge in the model till it is fed into the model and backpropagation occurs.
An exploratory attack is sending tons of inquiries to the model to gain information about the data set they have built into the model even to the point of extracting data about individuals pieces of data that are built into the model. Then, with this information, they could try to reconstruct the data set. They could attempt to trick the network by sending strange generated inputs.
In the paper Adversarial Machine Learning (2011), by Ling Huang et al., in section 2, the authors define these terms, under the category influence.

Influence
Causative - Causative attacks alter the training process through influence over the training data.
Exploratory - Exploratory attacks do not alter the training process but use other techniques, such as probing the detector, to discover information
  about it or its training data.

They also provide other related definitions.

Security violation
Integrity - Integrity attacks result in intrusion points
  being classified as normal (false negatives).
Availability - Availability attacks cause so many classification errors, both false negatives and false positives, that the system becomes effectively unusable.
Privacy - In a privacy violation, the adversary obtains information from the learner, compromising the secrecy or privacy of the system’s users.
Specificity (a continuous spectrum)
Targeted - In a targeted attack, the focus is on a single or small set of target points.
Indiscriminate - An indiscriminate adversary has a more flexible goal that involves a very general class of points, such as “any false negative.”


text:
How to calculate the size of a 3d object from an image?
I am wondering how to calculate the size of a 3d object in an image without knowing the focal length of the camera but the distance from the camera to the object.

There are hundred of papers on this task some older than I am! Normally this is done by trying to form a box shape around the image than estimate the volume. This task is typically done with multiple images so the two can generate a more clear picture of the size of the object than one image alone. An object could be 'infinitely' large but its mass could be behind the surface you can see with the picture. With the height and length dimensions extracted from the image and with the distance from the camera calculating the size of said surface is fairly easy.
 Stack overflow about problem
Frustum PointNets for 3D Object Detection From RGB-D Data (2018) By Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas
From contours to 3D object detection and pose estimation (2017) By Nadia Payet Sinisa Todorovic


text:
What is "conditioning" on a feature?
On page 98 of Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning the author writes;

Redacted phase space: Studying the distribution of inputs and the
  network performance after conditioning on standard physically-inspired
  features can help to visualize what new information the network is
  using from the jet. Training the network on inputs that have been
  conditioned on specific values of known features can also be useful
  for this purpose.

I cannot find other references to conditioning features. What does that mean?

This is conditioning in the sense of conditional probability. The idea is that the authors have some "standard physically-inspired features". They are splitting the data up into bins based on the values of these features, and then training a model for each bin. They are then examining the differences between the models. Usually this is done to learn something about the benefits of using the different features, and about the relationships between features and outputs.

That makes sense, do you happen to have an example of that technique? I guess it could give you good ways to make cuts on the input, but I'm not sure what else it would demonstrate.

@Clumsycat A lot of good examples are illustrated by Simpson's paradox (https://en.wikipedia.org/wiki/Simpson%27s_paradox). More generally, this technique is key in establishing causal linkages.


text:
How can I merge two datasets?
I want to merge 2 data sets in one, but don't know the right approach to do it. The datasets are similar, the last column is the same - will or not them buy a product. In the first dataset, users who only will buy, in second - only who won't buy.
The 1st dataset contains 500 rows and 2nd 10000 rows. What will be the right approach to merge it? How can I normalize them? And to point for an algorithm that the last column is the main sequence on what it should learn?
Example:
id    income date will_buy

23123 200    10.5 Yes

and second dataset:
id    income date will_buy

2323  100    10.5 No

You can use append function:
final = df1.append(df2, ignore_index=True)

To set the last column as labels, you set them as so by:
labels = np.array(final["will_buy"])

So, when calling the fit method on the model you build, you set labels = labels.


text:
If an image contains two distinct objects, should I create a copy of this image with distinct labels for each copy?
Suppose we want to detect whether an object is one of the following classes: $\text{Object}_1, \text{Object}_2, \text{Object}_3$ and $\text{Person}$. Should the annotated images only contain bounding boxes for either a person or an object? In other words, suppose an image has both $\text{Object}_1$ and $\text{Person}$. Should you create a copy of this image where the first version only has a bounding box on the object and the second copy only has a bounding box on the person?

You should use both classes together. Let's say you use the method you proposed. Then they will be contradicting each other as one teaches the network to recognize people, not objects and the other teaches the network to recognizes objects not person. There is no need for seperation of the two classes, unless you are making two seperate classifier. Hope I can help you.

If we already have a very good model for detecting the objects, couldn't adding bounding boxes to the people within the same images potentially lower the performance for the initial object detection model? Maybe that could be a reason to separate the classes?

It should not affect the accuracy for a lot. If performance don't really matter you can use two models so performance won't change for the object detection model. However it will be two times as slow as there is two models to run through. If this is a real time task, you should add another class instead.

Would adding annotations of people from a totally different data set (e.g. a public data set) to the existing data set help in person detection? Or is it ideal to have the people come from the same images in which there are also objects?

If you use two seperate models, it shouldn't matter too much. The model may confuse some object as person though if no other object is in the data. If the original data have an enough sample size maybe you should use that as your task is still to achieve a good testing accuracy on the private dataset. For the same model of course use the original data. Hope I can help you @PrimeNumber

If for multiple images, the orientation of the object changes, but the person remains relatively still, wouldn't this lead to overfitting?