text:
What AI algorithm could I use to trap an agent in a game?
Imagine a game with grid size 10x10, there is a good guy and a bad guy and obstacles in the grid, i.e. essentially a maze. The goal of the bad guy is to find the good guy and trap him by erecting walls around him. The good guy is able to move in the maze using a simple random movement, only in 4 directions - up, down, right or left.
I've used the A* algorithm to find the shortest path to the good guy but am still unsure as to how to go about trapping the good guy in his own space?

The good guy is able to move in the maze using a simple random movement, only in 4 directions - up, down, right or left.

What do you mean by "erecting walls around him"? It seems that you imply that the bad guy has more actions than up, down, right and left. It has also the actions of building walls. But how? You need to explain more in detail how this works.

Agree you need to define how the "bad guy" builds walls (can he build in adjacent cells, or anywhere?), and what it means to trap the "good guy" (seal him in a 1x1 cell, seal him in a 2x1 with the bad guy, cut him off from the bad guy?).


text:
Is a Conv2DTranspose the same as a full convolution?
I am currently creating a GAN model from scratch (following this tutorial: https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-an-mnist-handwritten-digits-from-scratch-in-keras/) but I can't find out how to implement Conv2DTranspose from scratch.  Is a Conv2DTranspose the same as a full convolution?  If not, how would one implement it?

What do you mean by "Is a Conv2DTranspose the same as a full convolution?"? Same in terms of what?

Yes I mean are they referring to the same opperation.

Con2DTranspose is an upsampling method used to increase the size of an image.
When we perform convolution, the size of the image decreases,
but in some scenarios, we want our image size to be the same as the input image size. Hence we use this convolution.
Here you will find Keras implementation on Conv2DTranspose
https://github.com/keras-team/keras/blob/v2.7.0/keras/layers/convolutional.py#L1093-L1394

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why is the Graph Isomorphism Network powerful?
I am reading a paper known as GIN, How powerful are graph neural networks?, Xu et al. 2019
The paper, Lemma 5 and Corollary 6, introduces Graph Isomorphism Network (GIN).
In Lemma 5,

Moreover, any multiset function $g$ can be decomposed as $g (X) = \phi(\sum_{x\in X} f(x))$ for some function $\phi$

Similarly, in Corollary 6,

Moreover, any
function $g$ over such pairs can be decomposed as $g (c, X) = \varphi((1+\epsilon) f(c)+ \sum_{x\in X} f(x))$ for some function $\varphi$.

Finally, it makes MLP by compositing $f^{(k+1)}$ and $\varphi^{(k)}$. i.e

$f^{(k+1)} \varphi^{(k)}$

I know that $h(X)$ or $h(c,X)$ is injective, because they are unique to $c$ and $X$(in Lemma 5 and Corollary 6, respectively).
What I don't understand is: in the statements in Lemma 5 and Cor 6, $\phi$ and $\varphi$, I don't know whether they are injective or not.
My question is then: why is GIN powerful? i.e, Why does GIN preserve injectivity?
This paper explains that injectivity should be preserved to get powerful GNN. How can I answer this question?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Apart from ontologies, which other methods for knowledge representation are there in Artificial Intelligence?
From what I have been reading, I see statements like

Ontology is a common method used for knowledge representation in
artificial intelligence.

But there is never really a discussion around what other options are available. To allow me to research other options I am hoping someone could maybe suggest alternatives?
As an extra question, do you have a preference for a particular system? If so, why?


text:
Why would SARSA diverge (but not Expected SARSA or Q-learning)?
In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (after 100 episodes) for larger values of alpha (alpha > 0.9). The graph is for the cliff walking gridworld example whose description is also given (from the paper by van Seijen et al).

As the image mentions, the image is taken from a paper by van Seijen and others titled "A Theoretical and Empirical Analysis of Expected Sarsa". In the image below from the van Seijen paper from Section VII A Discussion, the authors mention that the reason for the better interim performance of SARSA as compared to its asymptotic performance for larger values of alpha, is the divergence of Q-values. The authors, however, fail to mention the reason for the divergence.

What would be the reason that SARSA diverges but not Expected SARSA or Q-learning?
According to me, SARSA might have a higher variance than Expected SARSA, but it should behave, on average, the same as Expected SARSA.
Additionally, shouldn't Q-learning be at greater risk of diverging Q values since, in its update, we maximise over actions (and I have in fact seen a number of instances where there is a problem of diverging Q values in DQNs)?
The majority of papers I have looked at only talk about the problem from the function approximation perspective.

I think a useful piece of information to answer this question is a representation of the safe and optimal policies that can be learned on the cliff grid world. SARSA learns the safe path while Q-learning (and on the long run also Expected SARSA) learns the optimal path.
The reason lies in how the different algorithms select the next action.

"shouldn't Q-learning be at greater risk of diverging Q values since in it's update, we maximise over actions"
This is actually the reason why Q-learning doesn't suffer of divergence issues in this simple world. Choosing always the action that maximize the reward is what allows Q-learning to learn directly the optimal policy. And you can see it on the graphs, the learning rate alpha doesn't really affect the final performance in the 50k runs case.
The same logic applies to Expected SARSA. Remember that SARSA choose a random action, while Expected SARSA take the reward expected value into account, which makes Expected SARSA closer to Q-learning than SARSA itself.
-SARSA $$Q(S_{t}, A_{t}) + \alpha((R_{t+1}) + \gamma Q(S_{t+1}, A_{t+1}) - Q(S_{t}, A_{t}))
$$
-Q-learning $$Q(s_{t}, a_{t}) + \alpha((r_{t+1}) + \gamma max_{a} Q(s_{t+1}, a) - Q(s_{t}, a_{t}))
$$
-Expected SARSA $$Q(s_{t}, a_{t}) + \alpha((r_{t+1}) + \gamma \sum_{a} \pi(a | s_{t+1}) Q(s_{t+1}, a) - Q(s_{t}, a_{t}))
$$
This still doesn't explain why SARSA fails with high learning rates though. To answer that we just need to combine in a different light what we already said:

SARSA is not designed to depend on high future rewards
SARSA prefers policies that minimize risks

Combine these 2 points with a high learning rate, and it's not hard to imagine an agent struggling to learn that there is a goal cell G after the cliff, cause the high learning rate keeps giving high value to each random move action that keep the agent in the grid. Unfortunately for the agent, moving randomly gives you rewards but it also increase the chances of falling into the cliff. A small learning rate on the other hand push down the value of each action that simply keep you in the grid, and gives more change to SARSA to learn that the action of going down in the cell up to the bottom right corner is actually the real deal.
Hope this gives a bit more clarity, I suggest to maybe try to use some of the umpteenth noteboooks out there to visualize the policies learned by the algorithms, which is much more insightful than simply looking at graphs, even though is understandable that authors focus on the latter.

"and on the long run also Expected SARSA" $\rightarrow$ Expected SARSA is learning the safe path from the graphs (scoring around -20), because there is no decay of $\epsilon$ in the experiment. It won't learn the optimal path in the long run in this experiment. The fixed $\epsilon$ is also evident in Q-learning's results. If $\epsilon$ were to decay, then Expected SARSA and Q-learning would end up with identical performance on the optimal policy. But that's not what the comparison is trying to show.


text:
Is a test accuracy of 0.74 good enough, given a dataset of about 700 samples, and, if not, how can I improve it?
I am new to neural networks. I am trying to solve a binary classification problem. Specifically, I want to determine whether a patient has or not a certain disease based on the dataset.
The dataset has about 700 samples of different patients. I divided the sets into training and test (test size = 0.3).
My model has 1 input layer, 5 hidden layers, and 1 output layer. I used ReLU for the input and hidden layers, and I used the sigmoid for the output layer.
During the compilation of the model, I used stochastic gradient descent (SGD) as the optimizer and the mean squared logarithmic error for the loss.  I used mini-batch gradient descend (batch size = 4) for the training.
I am trying to calculate the accuracy on the test set I created previously.

The model evaluation for train set is about: 0.07 (loss) 0.76 (accuracy).

The model evaluation for test set is about: 0.07 (loss) 0.74 (accuracy).

Firstly, I would like to know if this is a good value for a model. Is the accuracy too small?
Plus, I would like to know if there's a way to improve accuracy based on my model.
I am trying to work on a project, so I was wondering if these values are acceptable.

Hello. Welcome to AI SE. Please, take the time to go through our on-topic page, which describes the type of question that you can ask here, and How do I ask a good question?. Now, you should edit your post in order to describe more in detail the 1. task you're solving (I suppose you're solving a classification problem, but what are you classifying?), 2. how much data you're using for training, testing, etc. [continuing on the next comment]

are you computing the accuracy on a separate dataset (i.e. the test dataset) or training dataset, and 4. describe the architecture of your neural network and other hyper-parameters, e.g. the optimizer, 5. what is your ultimate goal? do you want to deploy your model in production or are you just playing with neural networks?

I edited my post. Thank you for your adviceses.

Thanks! Now your question looks a lot better than before. So, you used 700*0.3 = 210 samples for testing and to compute the accuracy of 0.74 (and the remaining for training)? If that's the case, your dataset is very small. Unless the problem is simple, I don't think this amount of data is enough. So, you could probably get a higher test accuracy if you use more data. 5 hidden layers for your neural network seems also too many for such a small amount of data. Have you looked at the training/test/validation curves in order to determine whether your model has potentially overfitted the data?

You should also make sure that you randomly shuffled the data before splitting it into training and test datasets.

Yes, the dataset is too small but I need it for my project. I used 5 hidden layers because with a few less hidden layers the loss and the accuracy are worse. I tried to create a plot to see graphically what's going on but I didn't see overfitting.

is always hard to give suggestions without having access to data examples. From what you posted the only reasonable feedback someone can give is that the model seems to be learning fine since training and test accuracy are close in values. Might very well be the limit this approach can reach. Of course there are many things you could try. Larger batch and adding dropout (if not already part of your model architecture) are quick examples. But again, for specific suggestions we need as most detailed information as possible.


text:
Is there a tutorial for understanding the proof of convergence for TD learning?
I'm reading the article An Analysis of Temporal-Difference Learning
with Function Approximation (1997), but the mathematics inside seems overly complicated for me. Answers to some similar questions had pointed out that these proof typically involves stochastic approximation.
My question is: are there any good tutorials (textbooks, list of papers, etc.) on stochastic approximation (or similar topics) that prepare you for reading proof like this? It will be best if it is rather self-contained under my mathematical maturity.
I have an undergraduate-level mathematical analysis and probability theory foundation and have touched some measure theory.

This paper didn't even prove the hard results in SA. The book neurodynamic programming proves the basic SA result but, it uses a martingale convergence theorem which requires measure theory based probability. To be honest this is all kind of moot, nobody is going to be proving anything useful with NN based RL approximations.


text:
What is the difference between Probabilistic Graphical models and Graph Neural networks?
While going over PGMs and GNNs, it seems like both leverage the graph data structure. The former has been used to represent causal associations (among other things), while the latter has a varied set of applications. Do these techniques intersect?


text:
Does the converted (now square) distorted image of a face affect the accuracy of the calculation of the similarity in FaceNet?

As far as I know,

FaceNet requires a square image as an input.

MTCNN can detect and crop the original image as a square, but distortion occurs.

Is it okay to feed the converted (now square) distorted image into FaceNet? Does it affect the accuracy of calculation similarity (embedding)?
For similarity (classification of known faces), I am going to put some custom layers upon FaceNet.
(If it's okay, maybe because every other image would be distorted no matter what? So, it would not compare normal image vs distorted image, but distorted image vs distorted image, which would be fair?)
Original issue: https://github.com/timesler/facenet-pytorch/issues/181.

You don't have to crop out the detection box as-is, you could crop out a square centred on the box. Have you tried that for comparison?

@NeilSlater Yeah right, but the problem is, if I do so, unnecessary portions like hair will be included. For example, in general, a human's face is vertically long. If I crop the image in a square without distortion, then it's very likely the hair would be included. But a same person(especially woman) can have long or short hair depending on images. And I wonder if the calculation of embedding with facenet would be affected by such a thing. Actually, facenet expects "only face" in a square format as an input, and I've seen examples of distortion to make face a square. Would it be okay?

I don't know if it is OK or not, it depends on what inputs the model was trained on. You either need advice from someone who has used the specific model that you are using, or you need to try the different approaches that you are concerned about and measure their accuracy. There is no general rule covering all image classifiers.

As Neil Slater said it depends on how the model was trained. Now if you go to the FaceNet implementation in TF in github you can see that in the face alignment they do resize without aspect ratio keeping (old scipy.misc.imresize does not keep aspect ratio) so if you are using this implementation the answer to your question is: it does not affect the accuracy.
But if we want to elaborate the golden rule is: not to change the face aspect ratio because that is causing a distortion of the input data for the embedding computation.
In most face recognition tasks once you detect a bounding box you do a resize with aspect ratio keeping. You can accomplish that with several libraries:

Pillow Thumbnail method
Albumentations LongestMaxSize method

I would recommend to use resize with aspect ratio keeping.


text:
How to determine if Q-learning has converged in practice？
I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes.
At each state, I can choose the action corresponding to the maximum Q value at the state (the greedy action that the agent would take). And all the actions connect some states together. I think that would show me a road from start to goal, which means the result converges.
But some others think that as long as the agent learns how to reach the goal, the result converges. Sometimes the success rate is very high but we cannot get the road from Q table. I don't know which one means the agent is trained totally and what the converged result means.

A typical and practical way to measure the convergence to some solution (so not necessarily the optimal one!) of any numerical iterative algorithm (such as RL algorithms) is to check if the current solution has not changed (much) with respect to the previous one. In your case, the solutions are value functions, so you could check if your algorithm has converged to some value function e.g. as follows
$$
c(q_t, q_{t-1}, \epsilon) =
\begin{cases}
1,  &\text{if } |q_t(s, a) - q_{t-1}(s, a)| < \epsilon, \forall s \in S, a \in A \\
0, & \text{otherwise}
\end{cases},
\tag{1}\label{1}
$$
where

$c$ is the "convergence" function (aka termination condition) that returns $1$ (true) if your RL algorithm has converged to some small enough neighbourhood of value functions (where those value functions are "indistinguishable"), and $0$ otherwise
$q_t$ is the value function at iteration $t$
$\epsilon$ is a threshold (aka precision or tolerance) value, which is a hyper-parameter that you can set depending on your "tolerance" (hence the name); this value is typically something like $10^{-6}$

Of course, this requires that you keep track of two value functions.
You can also define your "convergence" function $c$ in \ref{1} differently. For example, rather than using the absolute value, you could use the relative error, i.e. $\left|\frac{q_t(s, a) - q_{t-1}(s, a)}{q_t(s, a)} \right|$. Moreover, given that RL algorithms are exploratory (i.e. stochastic) algorithms, the value function may not change (much) from one iteration to the other, but, in the next one, it could significantly change because of your exploratory/behavioural actions, so you may also want to take into account more iterations, i.e. after e.g. $N > 1$ iterations, if the value function does not change much, then you could say (maybe probabilistically) that your RL algorithm has converged to some small neighbourhood of value functions in the space of value functions.
Note that these approaches do not guarantee that your RL algorithm has converged to the global optimal value function, but to some locally optimal value function (or, more precisely, small neighborhood of value functions). Q-learning is guaranteed to converge to the optimal value function in the tabular setting (your setting), but this is in the limit; in practice, it is more difficult to know if Q-learning has converged to an optimal or near-optimal value function.
Maybe you can also have a look at episodic returns of the policy derived from your final value function, but without upper and lower bounds on the optimal returns, you don't know much about the global optimality of your policy/value function.
Yes, you can check if the policy makes the agent reach the goal, but many policies could do that job, i.e. that does not say that the policy is the best (or optimal) one, i.e. that's a necessary (provided the goal is reachable and the reward function models your actual goal) but not sufficient condition (for optimality). The optimality here is usually a function of the return (given that is what you are usually trying to optimize).

This answer is based on my general knowledge of numerical algorithms and not much on how people (in the RL community) determine when their Q-learning (or, in general, RL) algorithms converge. It's possible that people may be using other approaches to determine the convergence in practice. Right now, I don't recall serious approaches: typically, they just comment on the collected returns/rewards plots. So, I encourage other people that are familiar with RL research to give other answers, if they know/remember other approaches.


text:
Why do we have two similar action selection strategies for UCB1?
In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit problem (2000/2014), at time step $t$, an action is selected using the following formula
$$
a^*(t) \doteq \arg \max _{i=1 \ldots k}\left(\hat{\mu}_{i}+\sqrt{\frac{2 \ln t}{n_{i}}}\right) \tag{1}\label{1},
$$
where

$\hat{\mu}_{i}$ is an estimate of the expected return for arm $i$
$n_i$ is the number of times the action $i$ is selected
$k$ is the number of arms/actions

On the other hand, Sutton & Barto (2nd edition of the book) provide a slightly different formula (equation 2.10)
$$
a^*(t) \doteq \arg \max _{i=1 \ldots k}\left(\hat{\mu}_{i}+c\sqrt{\frac{\ln t}{n_{i}}}\right) \tag{2}\label{2},
$$
where $c > 0$ is a hyper-parameter that controls the amount of exploration (as explained in the book or here).
Why do we have these two formulas? I suppose that both are "upper confidence bounds" (and, in both cases, they are constants, though one is a hyper-parameter), but why (and when) would we use one over the other? They are not equivalent because $c$ only needs to be greater than $0$, i.e. it can be arbitrarily large (although, in the mentioned book, the authors use $c=2$ in one experiment/figure). If $c = \sqrt{2}$, then they are the same.
The answer to my question can probably be found in the original paper that introduced UCB1 (which actually defines the UCB1 as in \ref{1}),  or in a paper that derives the bound, in the sense that the bound probably depends on some probability of error, but I have not fully read it yet, so, if you know the answer, feel free to derive both bounds and relate the two formulas.

In the PDF of the original paper for UCB1 you linked, in page 242-243 the authors proves why non-optimal machines get played much less (in fact, logarithmically less) than the optimal ones. $c$ decides whether they indeed would, and $c=\sqrt{2}$ is the minimum choice of $c$.
We want to show that the number of runs for non-optimal machines ($n_i$, for non-optimal $i$s, in your notation) is asymptotically logarithmic. In other words, you may run them for a few times and well, it's fine, but not too often. We're devising some indicator value $a_i(t)=\hat \mu_i+(\epsilon...)$ such that the mistaken cases, where values of non-optimal ones surpass values of optimal ones ($a^*(t)<a_i(t)$), are minimized.
Think about the last inequality. We know that $\mu^* > \mu_i$ (again, optimal and non-optimal ones). Therefore, for that inequality to be true, it seems either the left-hand side should be quite small or the right-hand side should be quite large. But wait, $\hat \mu$s are actually some random trials for $\mu$, so we cannot claim directly from $\mu^* > \mu_i$ to $\hat{\mu^*} > \hat{\mu_i}$; it might be that we just need more trials.
The equations (7), (8) and (9) of the paper is the three conditions mentioned in the paragraph above; left-hand side is small, right-hand side is large or the trials are lacking. Well, in fact, as we stated the number of runs ... is asymptotically logarithmic at first, the third case can be eliminated(!), assuming that we've run this machine enough.
For the first and second case, since $\hat \mu_i$ is the average of some random variable in $[0, 1]$, we can use Chernoff–Hoeffding bound (or so called in the paper; stated as Hoeffding's inequality in Wikipedia). Now, a good choice of $(\epsilon ...)$ will guarantee (from Hoeffding's inequality) that the first two cases will occur sufficiently scarcely, or in other words, in the order of $t^{-4}$. To achieve this, we need $c \ge \sqrt 2$.
Now back to the third case, the enough number of runs is actually $l = \left \lceil 2 c^2 \ln n / (\mu^* - \mu_i)^2\right \rceil$. Thus, you may choose larger $c$ but receive longer convergence speed in penalty.
Funnily enough, after all the proofs the authors find $c=1/4$ to converge well and actually perform substantially better(!!) than $c=\sqrt{2}$. It seems they could not prove the bound as we did above.

@user9947 the t^{-4} comes from there being a t^{2} multiplying factor outside. Together they become t^{-2} which converges fast enough to converge to a constant. The sum of t^{-1} does not converge for instance.

Sufficiently scarcely is just scarce enough for logarithmic regret expectation to hold.


text:
How to find a parameter combination for a black box using AI?
I am working on a project where I encountered a component which takes 96 arguments (all integer values) and outputs 12 float values.
I would like to find a useful combination of these 96 values to receive the output that I want while avoiding random guessing, so the desired behavior would be that I provide the outcome and receive the 96 inputs to use them in my component.
Unfortunately I am not that experienced in that field. If I think about how I can implement this, my first thought was a kind of classification task, since I could build a dataset but the problem here is that I need integer values.
A second guess was a regression but would that be possible for y as an output vector?
Are there some other approaches that could fit to my use case?

"would that be possible for y as an output vector?": please, could you clarify this phrase ?

I‘m sorry, I was talking about multi-output regression but in this case I would have 96 output values and I was wondering if this could yield any useful result.


text:
What is the difference between step_model and train_model in the OpenAI implementation of the A2C algorithm?
I'm struggling a little with understanding the OpenAI implementation of A2C in the baselines (version 2.9.0) package. From my understanding, one step_model acts in different parallel environments and gathers experiences (calculates the gradients, I think), and sends them to the train_model that trains with them. After this, the step_model gets updated from the train_model.
What I am unsure about is if both step_model and train_model are actor-critic models or if step_model is actor and train_model is a critic (or vice versa). Does the step_model use the advantage function or is it just the train_model?


text:
How does one stack multiple observations in the input layer of a convolutional neural network?
The paper, Deep Recurrent Q-Learning for Partially Observable MDPs, talks about stacking multiple observations in the input of a convolutional neural network.
How does this exactly work? Do the convolutional filters loop over each observation (image)?
(I know this isn't the right group to request this, but I'll highly appreciate if someone could also suggest a framework that helps with this.)

I didn't read that paper, but it seems like you're talking about creating inputs (which, I suppose, represent states of the environment) to the value network which are composed of $N > 1$ successive frames/images (i.e. a state is a sequence of successive frames). This was also done in the original DQN paper. This is done mainly to provide more context, i.e. to partially solve the "partial observability" problem. As far as I remember from DQN, you just stack the frames, and that's going to be treated as a single state to the value network, though that may not be actually a "state".

Anyway, in that specific paper, they may be doing something different (especially after having looked at figure 2). Maybe later, if, meanwhile, someone doesn't answer this question, I will quickly read it and provide an answer.


text:
Are there any known models/techniques to determine whether a person in a store is a customer or a store representative?
Are there any known models/techniques to determine whether a person in a store is a customer or a store representative?
For example, customer representatives can wear uniforms and then one possible way to identify customer representatives is by the color of their uniform, texture, etc. On the other hand, a customer can also wear the same color clothes as that of a customer representative. Likewise, a customer representative could be wearing "normal clothes." So the main problems that may occur could be:

A customer becomes misclassified as a customer representative.
A customer representative representative becomes misclassified as a customer

So using clothing as the only proxy to classify people as customers or customer representatives seems to be flaky. Any other known ideas?

Access to the back of the store, basically if you can track them entering and leaving an employees only area.
But this requires a really good tracking algorithm and if you lose track you're going to have to fall back on some other properties.


text:
If REINFORCE agent suddenly drops, how do I verify if it's due to catastrophic forgetting?
I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from catastrophic forgetting. OTOH, c51 is not able to learn much and performs like a random policy throughout.
The environment looks like this -

action = [66, 1]
states = [20, 1]
max possible state value = 20
steps per episode = 20
Hidden Layer dimension = (128, 128)
learning rate = 0.001 (constant throughout)
Epsilon (exploration factor) = 0.2 with decay of 0.05 every 4000 episodes
Discount factor = 0.9
relay memory size = 10,000

Every episode runs for 20 steps and the rewards are collected for every step.

actual episode value is the plot is the x-axis value multiplied by 50
What could be the possible reasons for such a performance of c51 and DQN? And based on the state space, are my hyperparameters correct or some of them need more tuning? I will increase the replay memory size but other than that to check for catastrophic forgetting, I am not sure how to diagnose other issues.


text:
Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?
If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here.
If that's the case, then why does a negative reward for every step encourage the agent to quickly reach the goal (which is a specific type of behavior/policy), given that such a reward function has the same optimal policy as the shifted reward function where all rewards are positive (or non-negative)?
More precisely, let $s^*$ be the goal state, then consider the following reward function
$$
r_1(s, a)=
\begin{cases}
-1, & \text{ if }  s \neq s^*\\
0, & \text{ otherwise}
\end{cases}
$$
This reward function $r_1$ is supposed to encourage the agent to reach $s^*$ as quickly as possible, so as to avoid being penalized.
Let us now define a second reward function as follows
\begin{align}
r_2(s, a)
&\triangleq r_1(s, a) + 1\\
&=
\begin{cases}
0, & \text{ if }  s \neq s^*\\
1, & \text{ otherwise}
\end{cases}
\end{align}
This reward function has the same optimal policy as $r_1$, but does not incentivize the agent to reach $s^*$ as quickly as possible, given that the agent does not get penalized for every step. So, in theory, $r_1$ and $r_2$ lead to the same behavior. If that's the case, then why do people say that $r_1$ encourage the agents to reach $s^*$ as quickly as possible? Is there a proof that shows that $r_1$ encourages a different type of behaviour than $r_2$ (and how is that even possible given what I have just said)?

I have just found that a related question had already been asked, but my question is different, in that I am looking for a proof (or, at least, good logical argument) that shows that $r_1$ and $r_2$ lead to different behaviours, although they shouldn't.

The linked proof is not valid for episodic problems. More concretely, it assumes that absorbing states are also modified, but that would break the episodic nature (because reward values other than zero would be processed after the end of an episode)

@NeilSlater Why do you say that it assumes that "absorbing states" are modified? Do you mean that I assumed that the reward for the terminal/absorbing state can also be changed (i.e. be different than zero)? Maybe you should comment under the answer/proof to that linked post, and we can discuss there the limitations of that proof.

I think here "quickly" means that the training itself will be faster. If I imagine a gridworld problem for a moment, both $r_1$ and $r_2$ will result in a policy that gives the shortest path to the goal cell. I would say negative rewards encourage exploration and "quickly" means that we converge to the optimal policy quicker.

@HaiNguyen You may be right (i.e. the final policy may be the same, but the way you get to it may be different), but I would like to see a formal proof or, at least, an step-by-step example that shows that's the case. We probably can come up with such an example in a very simple environment with 2-3 states or in simple grid world.

Your examples are equivalent. But it is possible to find a constant yielding  a different optimal policy.
Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so is by reaching $s^*$.
Consider $r_3$ :
$$
r_3(s, a)=
\begin{cases}
1, & \text{ if }  s \neq s^*\\
2, & \text{ otherwise}
\end{cases}
$$
With a sufficiently large $\gamma$, moving infinitely without reaching $s^*$ is now the optimal solution.
For the generic case
$$
r_4(s, a)=
\begin{cases}
\alpha, & \text{ if }  s \neq s^*\\
\beta, & \text{ otherwise}
\end{cases}
$$
the threshold is found by comparing the results of the series $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t_m}$, where $t_m$ is the maximum episode length, and $\alpha + \alpha^2 + \alpha^3 + ... + \alpha^{t^*}$, where $t^*$ is the length of the episode following the fastest policy.
In the example of $r_3$, it is trivial to find examples where the fastest policy isn't optimal. Imagine a race, the agent starts on the left and gets either $\alpha$ or $\beta$ points, depending on where it is. With $\gamma = 0.9$ and no time-limit (infinite episodes) the optimal policy is  to move randomly, but in the second-to-last house, avoid the goal state. With $\gamma = 0.1$, the optimal policy is to move randomly (not really, probably there would be a slight advantage in moving right), but in the second-to-last house, enter the goal.

This answer is unclear (to me at least). You say "With a sufficiently large $\gamma$". Why can't that also be said for $r_1$ and $r_2$? $\gamma \in [0, 1)$: this is an assumption. I also don't get your point here "it is trivial to find examples where the fastest policy isn't optimal.". Note that we are talking about "optimal policies" associated with theoretically equivalent reward functions (i.e. theoretically, they have the same optimal policy or policies, which follows from the proof).

I think you're assuming that you have some RL algorithm that, in a finite number of steps, does not reach the goal and prefers to wander around, given that seems to be optimal policy. And, as someone had already stated in the comments, in practice, these reward functions may lead to different policies, in a finite number of steps, with some algorithm like Q-learning. However, your answer is not clear enough.

What are these $a$ and $b$ letters in your diagram? It's not even clear what algorithm you're assuming (if any). It's not clear where this conclusion "the optimal policy is to move randomly, but in the second-to-last house, avoid the goal state" comes from too. As far as I am concerned, this answer is not clear. Maybe start from your definition of the optimal Bellman equation and express it in terms of rewards. If that's where you started from, then find the contradiction to (or mistake in) the proof. Or maybe start by stating your assumpions.


text:
How do I derive the gradient of the log-likelihood of an RBM?
In a Restricted Boltzmann Machine (RBM), the likelihood function is:
$$p(\mathbf{v};\mathbf{\theta}) = \frac{1}{Z} \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}$$
Where $E$ is the energy function and $Z$ is the partition function:
$$Z = \sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}$$
The log-likelihood function is therefore:
$$ln(p(\mathbf{v};\mathbf{\theta})) = ln\left(\sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}\right) - ln\left(\sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}\right)$$
Since the log-likelihood function cannot be computed, its gradient is used instead with gradient descent to find the optimal parameters $\mathbf{\theta}$:
$$\frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}} = -\frac{1}{\sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}} \sum_{\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}\right] + \frac{1}{\sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}} \sum_{\mathbf{v},\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}\right]$$
Since:
$$p(\mathbf{h}|\mathbf{v}) = \frac{p(\mathbf{v},\mathbf{h})}{p(\mathbf{v})} = \frac{\frac{1}{Z} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}}{\frac{1}{Z} \sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}} = \frac{e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}}{\sum_{\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}}$$
Then:
$$\frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}} = -\sum_{\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot p(\mathbf{h}|\mathbf{v}) \right] + \frac{1}{\sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}} \sum_{\mathbf{v},\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}\right]$$
Also, since:
$$ \frac{e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}}{Z} = \frac{e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}}{\sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h};\mathbf{\theta})}} = p(\mathbf{v},\mathbf{h})$$
Then:
$$\begin{align} \frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}} &= -\sum_{\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot p(\mathbf{h}|\mathbf{v}) \right] + \sum_{\mathbf{v},\mathbf{h}} \left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \cdot p(\mathbf{v},\mathbf{h})\right] \\ &= -\mathbb{E}_{p(\mathbf{h}|\mathbf{v})}\left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \right] + \mathbb{E}_{p(\mathbf{v},\mathbf{h})}\left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \right] \end{align}$$
Since both of these are expectations, they can be approximated using Monte Carlo integration:
$$ \frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}} \approx -\frac{1}{N} \sum_{i = 1}^{N} \left[\frac{\partial E(\mathbf{v},\mathbf{h}_i;\mathbf{\theta})}{\partial \mathbf{\theta}} \right] + \frac{1}{M} \sum_{j=1}^{M} \left[\frac{\partial E(\mathbf{v}_j,\mathbf{h}_j;\mathbf{\theta})}{\partial \mathbf{\theta}} \right] $$
The first term can be computed beacuse it is easy to sample from $p(\mathbf{h}|\mathbf{v})$. However, it is difficult to sample from $p(\mathbf{v},\mathbf{h})$ directly, but since it is easy to sample from $p(\mathbf{v}|\mathbf{h})$, then Gibbs sampling is used to sample from both $p(\mathbf{h}|\mathbf{v})$ and $p(\mathbf{v}|\mathbf{h})$ to approximate a sample from $p(\mathbf{v},\mathbf{h})$.
My questions are:

Is my understanding and math correct so far?
In the expression for the gradient of the log-likelihood, can expectations be interchanged with partial derivatives such that:

$$\begin{align} \frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}} &= -\mathbb{E}_{p(\mathbf{h}|\mathbf{v})}\left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \right] + \mathbb{E}_{p(\mathbf{v},\mathbf{h})}\left[\frac{\partial E(\mathbf{v},\mathbf{h};\mathbf{\theta})}{\partial \mathbf{\theta}} \right] \\ &= - \frac{\partial}{\partial \mathbf{\theta}} \mathbb{E}_{p(\mathbf{h}|\mathbf{v})}\left[E(\mathbf{v},\mathbf{h};\mathbf{\theta}) \right] + \frac{\partial}{\partial \mathbf{\theta}} \mathbb{E}_{p(\mathbf{v},\mathbf{h})}\left[E(\mathbf{v},\mathbf{h};\mathbf{\theta}) \right] \\ &= \frac{\partial}{\partial \mathbf{\theta}} \left(\mathbb{E}_{p(\mathbf{v},\mathbf{h})}\left[E(\mathbf{v},\mathbf{h};\mathbf{\theta}) \right] - \mathbb{E}_{p(\mathbf{h}|\mathbf{v})}\left[E(\mathbf{v},\mathbf{h};\mathbf{\theta}) \right] \right) \\ &\approx \frac{\partial}{\partial \mathbf{\theta}} \left(\frac{1}{M} \sum_{j=1}^{M} \left[E(\mathbf{v}_j,\mathbf{h}_j;\mathbf{\theta}) \right] - \frac{1}{N} \sum_{i = 1}^{N} \left[E(\mathbf{v},\mathbf{h}_i;\mathbf{\theta}) \right] \right) \end{align}$$

After approximating the gradient of the log-likelihood, the update rule for the parameter vector $\mathbf{\theta}$ is:

$$\mathbf{\theta}_{t+1} = \mathbf{\theta}_{t} + \epsilon \frac{\partial ln(p(\mathbf{v};\mathbf{\theta}))}{\partial \mathbf{\theta}}$$
Where $\epsilon$ is the learning rate. Is this update rule correct?

Hey @nbro, this is the second question that I ask that contains technical details and receives no answers. I think this is because this question is not what is usually asked on the ai.stackexchange website. After browsing the questions a bit, it seems most of the questions are practical like "How do I choose the optimal batch size?". Which site is more appropriate for theoretical questions like this one? I tried to ask on stat.stackexchange, but I still get no answers because that site is more oriented towards pure probability and statistics and not machine learning. Any suggestions?

Hello. There's is the site Cross Validated SE that also accepts questions related to machine learning. Even if you ask your question there (and often cross-posting is discouraged), in this case, I suggest that you do not delete this post from our site, given that it is on-topic here too, and this is the type of question that I think we should get more here (i.e. technical questions), that's also why I upvoted it. Moreover, note that you may also not receive an answer on CV SE. Unfortunately, right now, I am not familiar with RBMs, otherwise, I would give it try to answer it.

One way to attract more visitors and potentially someone that is able to answer your question is to open a bounty (but I think you still don't have enough reputation to do that) or to edit your post so that it pops up in the active questions of the site and people can regularly see it. Of course, none of these "tricks" will guarantee that you will get an answer.


text:
Are there any rules for choosing batch size?
I am training a CNN with a batch size of 128, but I have some fluctuations in the validation loss, which are greater than one. I want to increase my batch size to 150 or 200, but, in the code examples I have come across, the batch size is always something like 32, 64, 128, or 256. Is it a rule? Can I use other values for it?

related: https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network

Data that has a size of a factor of $2$ (aka $2^n$ for some integer $n$) allows for easier memory management because data can be organized contiguously (without gaps). This allows for faster memory reading and thus faster iteration time in general. From a computational point of view, this is important as it can be taken advantage of by the compiler and speed up iteration loops. This is why batch sizes are chosen as such in practice. However, this does not necessarily imply better training results.
In regards to your question of "Can I use other values for batch size?":
Yes, you can use different values, and, for the most part, you probably won't see the difference in computational performance because of the speed of modern training APIs. So, until training a large model with a lot of compute where this optimization has a larger impact, feel free to experiment :)


text:
What are the state-of-the-art results in OpenAI's gym environments?
What are the state-of-the-art results in OpenAI's gym environments? Is there a link to a paper/article that describes them and how these SOTA results were calculated?

There is the leaderboard page at the gym GitHub repository that contains links to specific implementations that "solve" the different gym environments, where "to solve" means "to reach a certain level of performance", which, given a fixed reward function, is typically measured as the average (episodic) return/reward. For example, in the case of the CartPole environment, you solve it when you get an average reward of $195.0$ over $100$ consecutive trials.


text:
Why do we resize images before using them for object detection?
In object detection, we can resize images by keeping the ratio the same as the original image, which is often known as "letterbox" resize.
My questions are

Why do we need to resize images? If we resize images to have all the same dimensions, given that some original images are too long vertically or horizontally, we will lose a lot of features in those images.

If the "letterbox" method is better than "normal resize" (i.e. without keeping the aspect ratio, e.g. the result of the application of OpenCV's resize function with the parameter interpolation set to cv2.INTER_AREA), why don't people apply it in the classification task?

There are different questions and even different lines of thought here. Let's go through them
On resizing

Why do we need to resize? To fit the network input which is fixed when nets are no Fully Convolutional Networks (FCN)
What if my net is FCN? Still makes sense to resize to bound the dimension of the input features you want to detect (a person on a small image VS big image). Take into account that the kernel sizes do not vary although the image size does.

On keeping aspect ratio (or letterbox as some people like to say)

Why to keep aspect ratio? This is more of a philosophical question. It is believed that keeping the aspect ratio helps the nets to learn the natural variability in object sizes (say a person bounding box can not be super hight and super thin, because that would be a street light).

Why not to keep aspect ratio? If you resize without keeping the aspect ratio and the aspect ratio distortion is not mega super very huge, the networks will still learn. In other words, if your input images don't have crazy aspect ratios, then there is no difference between adding or not a bit of distortion. In fact, sometimes it will even act as a regularization or augmentation.

Conlusion
As long as your application is not too specific and your input images aspect ratios are bounded (this is, if  you train with images from any regular camera), you should not worry too much about this.
When to worry about this? When you train with huge vertical or horizontal images or if you train with images taken from some very specific devices like geophisical, radio, or optical sensors. In these cases you should pay special attention on how to resize or split an image. For example, with a recording of a radio sensor, if you resize with aspect ratio deformation, a wave from a specific frequency would transform to another because of the sine wave warp)

Thank you for your answer, I agree with you at the conclusion, I am trying something like that right now. However, I am not sure when you mention "philosophical" in a technical problem, I think it should be explained by mathematics or at least, logical way. Let see this answer, I will give you a vote, but not the correct answers.

Thanks and no prob man, I write for the community. But just for the record, I work in applied research on AI and most of the research and product we build are first based on some philosophical assumptions, then we model those assumptions in math, and then we run the math in python. AI is still a very open field, that is why I love it, so there is not always a closed answer for everything.

I agree, all of the deep learning ideas that I knew came from the philosophical assumptions, you are doing an amazing job, I will try solving in that direction in the next project.


text:
What is a "center loss"?
I have seen that a center loss is beneficial in computer vision, especially in face recognition. I have tried to understand this concept from the following material

A Discriminative Feature Learning Approach for Deep Face Recognition
https://www.slideshare.net/JisungDavidKim/center-loss-for-face-recognition

However, I could not understand the concept clearly. If someone can explain with the help of an example, that would be appreciated.


text:
Is the 3d convolution associative given that it can be represented as matrix multiplication?
I'm trying to understand if a 3D convolution of the sort performed in a convolutional layer of a CNN is associative. Specifically, is the following true:
$$
X \otimes(W \cdot Q)=(X \otimes W) \cdot Q,
$$
where

$\otimes$ is a convolution,
$X$ is a 3D input to a convolution layer,
$W$ is a 4D weights matrix reshaped into 2 dimensions,
and $Q$ is a PCA transformation matrix.

To elaborate: say I take my 512 convolutional filters of shape ($3 \times 3 \times 512$), flatten across these three dimensions to give a ($4096 \times 512$) matrix $W$, and perform PCA on that matrix, reducing it to say dimensions of ($4096 \times 400$), before reshaping back into ($400$) 3d filters and performing convolution.
Is this the same as when I convolve $X$ with $W$, and then perform PCA on that output using the same transformation matrix as before?
I know that matrix multiplication is associative i.e. $A(BC)=(AB)C$, and I have found that convolution operations can be rewritten as matrix multiplication.
So my question is, if I rewrite the convolution as matrix multiplication, is it associative with respect to the PCA transformation (another matrix multiplication)?
For example, does $X' \cdot (W' \cdot Q) = (X' \cdot W') \cdot Q$, where $X'$ and $W'$ represent the matrices necessary to compute the convolution in matrix multiplication form?
To try and figure it out, I looked to see how convolutions could be represented as matrix multiplications, since I know matrix multiplications are associative. I've seen a few posts/sites explaining how 2D convolutions can be rewritten as matrix multiplication using Toeplitz matrices (e.g. in this Github repository or this AI SE post), however, I'm having trouble expanding on it for my question.
I've also coded out simple convolutions with a $W$ matrix of $4 \times 3$, an $X$ matrix of $4 \times 2$, and using sklearn's PCA to reduce $W$ to $4 \times 2$. If I do this both ways, the output is not the same, leading me to think this kind of associativity does not exist. But how can I explain this with linear algebra?
Can anyone explain whether this is or is not the case, with a linear algebra explanation?


text:
How can reinforcement learning be applied when the goal location or environment is unknown?
I am studying RL. I was thinking whether a new state value or the observation is provided by the environment before the agent actually implements the action.
Take the maze problem as an example. Each state consists of all the available cells information, provided by the environment. But what if the environment is unknown? For example, there is a maze with an unknown destination cell. The agent needs to find the destination cell. The state is 1 or 0, meaning the destination reached or not. But the environment, which is the maze, can only provide the state at cell $i$ which is 0 or 1 only when the agent reaches cell $i$.
Can this still be solved by RL? I am confused about the environment setup.

I think you're using the numbers $0$ and $1$ to denote "reward". Is this correct? Btw, I've changed your title to be what I think is your question. Please, make sure that's really your question. If not, feel free to edit your post again.

When reading about RL and RL agents/algorithms, you always need to keep in mind that, typically, the RL agent/algorithm is trying to maximize the reward (or something equivalent, such as minimizing the regret) in the long run (i.e also the reward that you may receive in the future): that's its (mathematical) goal. Whether that  also corresponds to the high-level goal (e.g. reaching some physical location in some world) that the (human) designer of the so-called reward function (i.e. the function that gives the reward to RL agent) had in mind is a different story.
To maximize the reward, the RL agent interacts with the environment by taking actions, receiving rewards, and moving to other states. Initially, the RL agent does not know which actions lead to more rewards, so it may take random actions (this is known as exploration). Once it starts to understand the dynamics of the environment, it may start to take only the actions that lead to high reward (this is known as exploitation).
To answer your question directly, the RL agent can indeed take actions without knowing the dynamics of the environment. However, initially, it may need to take some random actions (which may lead to low rewards), so that to get more rewards in the long run.


text:
Is which sense was AlphaGo "just given a rule book"?
I was told that AlphaGo (or some related program) was not explicitly taught even the rules of Go -- if it was "just given the rulebook", what does this mean? Literally, a book written in English to read?

Most likely, they refer to an encoding of rules of Go.

okay, how is this encoding done with a neural network?

je ne sais pas :(

it was "just given the rulebook", what does this mean? Literally a book written in English to read?

The program was not given a natural language version of the rules to interpret. That might be an interesting AI challenge in its own way, but none of the current cutting-edge game playing reinforcement learning systems do much in the way of natural language processing.
Instead, "just given the rulebook" is a rough metaphor for what actually happened: The rules of Go were implemented as functions that the game-playing agent could query. The functions can answer things such as "when the board looks like this, what are my valid actions?" and "if the board looks like this, have I won yet?". The board state might be represented by a matrix with stone positions encoded using numbers. Outputs might be a similar matrix of numbers for valid action choices (where the player is allowed to put stones in Go) or perhaps a single number, $0$ for not won yet, $1$ for a move that wins the game.
There may even be further helper functions that help assess moves (e.g. a value for how many enemy stones would be captured if a piece was played in a specific location), but the bare minimum needed is "what moves are valid?" and "has anyone won?". A very common third function, useful for look-ahead planning is "if the board starts like this, and I take that action, what will the board look like next" - with this function, an agent can look ahead to future positions to help search for winning moves.
This approach is common in game playing agents. A learning agent can in theory also learn the rules of the game through trial and error, as long as it receives some feedback when it has broken the rules. However, more often the goal of training the agent is only to play as well as possible - learning the rules from scratch would be extra work for the agent, and maybe not an interesting problem to solve. So the agent is given helper functions by the developers that allow it to explore only valid moves according to the rules.

when u say that a human gives it this information it seems like alphago was partially programmable while the actual strategy was derived completely by the neural net (or whatever alphago is). it  would be cool if natural language processing was indeed the way it could learn any game, by literally just reading a book and that frankly seems like an easier problem than become the uber-world champ in Go.

@releseabe Natural language processing to the level of understanding documentation for a game that was intended for humans is in fact a far far harder problem than becoming a world champion Go player. Cutting edge research into NLP is not even close, and researchers don't even understand the problem fully let alone have an approach that could solve it. This is slightly counter-intuitive since humans seem to take little effort to be able to read a book of rules just fine, whilst becoming a Go master is very hard for a person. DIfficultly of problems in AI is different to human experience.

authors: กลวัชร ไพแก้ว


text:
How do I create a custom gym environment based on an image?
I am trying to create my own gym environment for the A3C algorithm (one implementation is here). The custom environment is a simple login form for any site. I want to create an environment from an image. The idea is to take a screenshot of the web page and create an environment from this screenshot for the A3C algorithm. I know the doc and protocol for creating a custom environment. But I don't understand how to create an environment, exactly, based on a screenshot.
If I do so
self.observation_space = gym.spaces.Box(low=0, high=255, shape=(128, 128, 3), dtype=np.uint8)

I get a new pic.
Here's the algorithm that I am trying to implement (page 39 of the master's thesis Deep Reinforcement Learning in Automated User Interface Testing by Juha Eskonen).

Well, you need to represent the screenshot as an "observation" of your environment. A screenshot is an image, so you can read this image and turn it into an array, which should be your observation. Maybe this code https://github.com/openai/gym/blob/master/gym/wrappers/atari_preprocessing.py can be useful. In any case, I would like to note that our site focuses on theoretical questions, but this is a programming question. Given that this question is about a RL topic, I will not close it as off-topic, given that RL is central to AI. Check this: https://ai.stackexchange.com/help/on-topic.

If you don't want to create the environment, what is your question then?

No, I want) I just uderstand, I reread the paper many times and algorithm and understand that we use a3c for creating a probability map for each element and then we do some action using third part api, like selenium.

Ok. But what is your question? It was "how to create this custom environment from screenshots?", right?

Right)  My question was  "how to create this custom environment from screenshots?"

The question is conceptually wrong, because of misunderstanding of area. Explanation: The idea is to replace open ai gym by something different. For example: web-site or computer game. There is no way to create an environment based on image. If you want to use implemented algorithm for open ai gym and want to change environment for your own, could do something like this: https://towardsdatascience.com/creating-a-custom-openai-gym-environment-for-stock-trading-be532be3910e. If need to change env complitely than it is necessary to rewrite your algorithm for your tasks.


text:
Why does the accuracy drop while the loss decrease, as the number of epochs increases?
I've been trying to find the optimal number of epochs that I should train my neural network (that I just implemented) for.
The visualizations below show the neural network being run with a variable number of epochs. It is quite obvious that the accuracy increases with the number of epochs. However, at 75 epochs, we see a dip before the accuracy continues to rise. What is the cause of this?

You really need to provide more details about your problem, neural network and data. Is your model a simple multi-layer perceptron (i.e. not recurrent connections or convolutions)? How many layers does it have (and how many neurons per layer)? If you are using the MSE loss, I suppose that you are using a regression problem, but you're using the accuracy... So, which problem exactly? How many data points do you have? Please, edit your post to include all these details.

Decrease of loss does not essentially lead to increase of accuracy (most of the time it happens but sometime it may not happen). To know why, you can have a look at this question. The network cares about decreasing the loss and it does not care about the accuracy at all. So it's no surprise to see what you presented.
Additional note: If you use batch approaches to teach your network, or if you choose a big step size you may also see that the loss increases sometimes but in the case of batching, the hope is that its trend is to be decreased.


text:
Why not use the target network in DQN as the predictor after training
Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the stable one, why do we keep using/saving the first model as the predictor instead of the target?
I see in the code everywhere:

Model
Target model
Train model
Copy to target
Get loss between them

At the end, the model is saved and used for prediction and not the target.

The learning model and target model are only different by N steps (typically a few thousand) out the entire taining process. If the process is near complete, they will also be quite similar.
The target model is not inherently more stable in terms of producing "correct" or "better" Q values. Instead it is kept static for a period of time in order to stabilise the temporal difference (TD) target $R_{t+1} + \gamma \text{max}_{a'} \hat{q}(S_{t+1}, a', \theta^{-})$
Due to the copying stage that you listed:

If you return the target model, this is identical to returning the learning model from N steps beforehand.

Which in turn means that there was no point in doing the last N steps of training. You may as well have returned the learning model at the same point as copying.

Or another way of thinking about it: Returning the learning model at the end of the training process is identical to copying it to the target model as normal, then returning the target model immediately.

Maybe one practical suggestion would be to see if learning becomes "unstable" (in some sense) in the last $N$ steps. If that's the case, maybe you can just derive the policy from the target network.

@nbro: I'm not sure how that could be made practical. You can detect that changes are happening, but not why (it might be due to discovery of something real that needs adjusting for, or it might be catastrophic forgetting). The practical approach I take is to separately test candidate networks at snapshot points. I generally ignore the relationship with when the last copying was to the target network. However, due to taking snapshots at e.g. every 10,000 steps, and copying to target every 1,000 steps, often there is no difference between target and learning networks, and it's a non-issue.

Target network is not more stable. Both networks are the same in the regard that no one is more stable than the other. The reason for using a target network is that your current network after each step is updated. So, by not using a target network and using just the current network, after each update the rewards for many states will be modified slightly. So, for a particular state, after each update, the reward will be modified which will lead to an unstable reward. This happens because when you update Q value function for state S, you may also slightly change the reward it predicts for state S' in the next step. So, each update to Q value function changes slightly the rewards for many other states, which makes the predicted rewards in all those states to be slightly unstable, as they are changed very often and you dont have a clear reward in those states.
However, by using a target network, your reward will be more stable(as you dont update your target network at each step, but you only update the current network).
So, by using a target network, the update is more stable, not the individual networks. And because the target network is an older version of the current network, it is sensible to use the current network as it was trained more than the target network

"This happens because when you update Q value function for state S, you may also slightly change the reward it predicts for state S'.", I don't see how this is true, if you are just updating Q(s, a). Maybe you mean that, if at the next iteration/time step, you use the just updated Q(s, a) to update Q(s', a), then you continue doing this, then you will somehow be oscillating.

that is true. By using the current network which was updated for a state to predict the reward for other states, the reward for those states will be different than the reward for those states before updating for a particular state.

Yes, but that's different than what you wrote (in the sense that what you wrote is not fully clear), so I would edit your post to clarify that. I would also edit your post to clarify this sentence "Both networks are the same". Both networks have the same architecture, but they are not the same (i.e. do not have the same parameters' values), apart from when you just copied the main value network to the target network.


text:
Should the exploration rate be updated at the end of the episode or at every step?
My agent uses an $\epsilon$-greedy strategy to learn. The exploration rate (i.e. $\epsilon$) decays throughout the training. I've seen examples where people update $\epsilon$ every time an action is taken, while others update it at the end of the episode. If updated at every action, $\epsilon$ is more continuous. Does it matter? Is there a standard? Is one better than another?

In the long run it would make no difference and both have their benefits. Updating after an episode will take longer to converge but offers more exploration and vice-versa.


text:
How efficient is SCAWI weight initialization method?
I'm currently in the middle of a project (for my thesis) constructing a deep neural network. Since I'm still in the research part, I'm trying to find various ways and techniques to initialize weights. Obviously, every way will be evaluated and we will choose the one that fits best with our data set and our desired outcome.
I'm all familiar with the Xavier initialization, the classic random one, the He initialization, and zeros. Searching through papers I came across the SCAWI one (Statistically Controlled Activation Weight Initialization). If you have used this approach, how efficient is it?
(Also, do you know any good sources to find more of these?)

My 2 cents: this is a method back from 1992, hence I would not expect that has anything meanigful to add in the today's context (heck, ReLU did not even exist back in 1992!).

@desertnaut Something like the ReLU (i.e. that sets to zero everything that is negative) was actually used back in the 80s. See e.g. equation 4 neocognitron paper by Fukushima et al.. Although that is not the ReLU, it's similar. Actually, figure 4 of that paper shows the definition of the ReLU. So, ReLU exists at least since the 1982 (and was used in the context of NNs), but the term "ReLU" or "rectifier" may have been introduced later and became standard as an activation function in neural networks only in the last years.

@nbro thank you for this information. But the spirit of my comment was that rectifier activations were far from standard back then; I am old enough to have used NNs for my BSc thesis in 1991-92, and I cannot recall them being part of the standard toolboxes or introductory exhibitions. On the other hand, the He initialization is built exactly upon the premise of such rectifier activations. (1/2)

@nbro All in all, although assessing the SCAWI scheme in today's context could be a topic of research, worrying about it from a practitioner's viewpoint (such as OP's) sounds like overkill. (2/2)


text:
How to decide if gradients are vanishing?
I am trying to debug a convolutional neural network. I am seeing gradients close to zero.
How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values?
I am getting values close to $4$ decimal places (e.g. $0.0001$) and, in some cases, close to $5$ decimal places (e.g. $0.00001$).

The CNN seems not to be learning since the histogram of weight is also quite similar in all epochs.
I am using the ReLU activation function and Adam optimizer. What could be the reason for the vanishing gradient in the case of the ReLU activation function?
If it is possible, please, point me to some resources that might be helpful.

This question is partially a duplicate of this one, but here you're also asking about vanishing gradients in the context of ReLU, so I will not mark it as a duplicate.


text:
Which reinforcement learning approach to use when there are 2 collaborative agents?
Suppose we are training an environment with 2 collaborative agents with Reinforcement Learning. We define the following example: There is a midfielder and a striker. The midfielder's reward depends on how many goals are scored, which however depends on the attacker's performance. And the striker's performance depends on how good the midfielder is at making his passes.
For this type of problem, what do you recommend to study?

What do you already know about RL? Is there an absolute requirement in this case that the two agents are implemented separately? Could you outline the learning challenges faced by the midfielder and striker - is it really just a matter of accuracy in two specific skills, with no other interdependence?

The performance being dependent on the other agent should not end up mattering that much. They will learn to do the best they can with whatever the other agent is doing and hopefully they will learn to work together. If you are trying to play some form of soccer, then that will need a pretty advanced method because of the complexity. Since you have a continuous state space and probably a mix of continuous and discrete action space (movement is continuous, pass or not pass is discrete), I would suggest an advanced form of policy gradients like PPO.

spinningup.openai.com is a good place to learn about reinforcement learning methods.


text:
How should I define the reward function to solve the Wumpus game with deep Q-learning?
I'm writing a DQN agent for the Wumpus game.
Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?
This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?

The reward function is up to you when you set the goals for the agent.

If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.

If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.

The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.
Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.

The reward function belongs to the environment and it is the only way the agent can explore the world given a state.
If we want agents to do something specific, we must provide rewards to them in such a way that they will achieve our goals. It is thus very important that the reward function accurately indicates the exact behavior.
Depending on your goal you can construct the function such that the agent will try to finish the game as fast as possible or collect the maximum score.
For example, certain reward functions can cause an agent to commit suicide in order to avoid more severe punishment in form of negative reward in the future (e.g. if the step reward is very small). Or it will go the safest way without collecting gold if falling in pits punishment is very big. In other words, you should experiment with your reward function to find a tradeoff.
Check out this video for more intuition behind it.


text:
Is it possible to have a variable-length latent vector in an autoencoder?
I'm trying to have a simple autoencoder but with variable latent length (the network can produce variable latent lengths with respect to the complexity of the input), but I've not seen any related work to get idea from. Have you seen any related work? Do you have any idea to do so?
Actually, I want to use this autoencoder for transmitting the data over a noisy channel, so having a variable-length may help.

What do you mean with  complexity of the input? What is your input? You mean something like texts and the input complexity is the number of words of your texts?

it can be text and in that case the number of words is a good measure of complexity. I've seen an example of doing such a thing in this paper but I think its not a good way to do so. I want a little more general idea, where the model can understand the complexity itself and determine the length of latent with respect to that

If you use RNNs, then I think the solution is to use padding (zero padding) with max sequence length (that is the max number of words in a text) in order to tell your model to skip the zeros when possible. In that way, your model will try to learn a good representation of your input with fixed size. If you do not know this dimension, a solution may be to grid search this hyperparameter.
If you still want to exploit the dimensionality difference, maybe you can train different models with fixed size dimension of the representation dependently of the dimension of the input. That is, for example, use one for small, one for medium and one for large dimensions, but this should surely require to have a large and quite balanced initial dataset.
Another idea could be to use the autoencoder with a fixed latent dimension. Then, do effective clustering on your samples using their latent representation, considering that similar representations should have similar dimensionality requirements (?). After that, you could train your initial dataset on k models, the same number as the clusters. That is, there should be k different latent spaces. The goal is to match each instance to the correct model. At first, you should train them all with each instance, but as the training progresses, you should maybe go with binary search for each instance in order it to find the correct model, assuming that there is total order in the measuring of the dimensionality requirements. Of course, this is just an idea, I don't know if it is going to be really helpful at all.

Unfortunately I can't understand your first paragraph, do you mean zero-padding the latent? can you explain more?

No! In the input space in order to deal with the incosistent dimensionality.

Ok, but how it helps with variable-length latent vector? What I understood from this paragraph will help to have different length input (which RNNs can do with no problem at all)

Your second paragraph is the way they have done in the paper I mentioned before (this one). But training a model for each latent length is not what I want, because I want the network (not me) to measure the complexity of input (useful when the input is image) and be able to produce any latent length (not those I predetermined)

Check my post. I fixed it.

You might want to look at an encoder-decoder sequence to sequence model. This model allows you to input and output data with variable length.

Maybe I asked my question in a bad way. The length of input or output doesn't matter to me (and they can be both fixed). But I want the network to produce variable length latent. Like if it see a complex image, it should use more latent units to reconstruct it, but if the image of the same size is simple (e.g. mostly black), the network should decide to use less latent units and it will be able to get almost the same reconstruction error.


text:
Is my interpretation of the mathematics of the CBOW and Skip-Gram models correct?
I am a mathematics student who is learning NLP, so I have paid a high amount of attention on the mathematics used in the subject, but my interpretations may or may not be right sometimes. Please correct me if any of them are incorrect or do not make sense.
I have learned CBOW and Skip-Gram models.
I think I have understood the CBOW model, and here is my interpretation: First, we fix a number of neighbors of the unknown center word which we would like to predict; let the number be $m$. We then input the original characteristic vectors (vectors of zeros and ones only) of those $2m$ context words. By multiplying those vectors by a matrix, we obtain $2m$ new vectors. Next, we take the average of those $2m$ vectors and this is our hidden layer, namely $v$. We finally multiply $v$ with another matrix, and that is the "empirical" result.
I tried to follow the logic to Skip-Gram similarly, but I have been stuck. I understand that Skip-Gram is kind of a "reversal" of CBOW, but the specific steps have given me a hard time. So, in Skip-Gram, we only have a center word, and based upon that we are trying to predict $2m$ context words. By similar steps, we obtain a hidden layer, which is again a vector. The final process also involves multiplication with a matrix, but I don't know how we can get $2m$ new vectors based upon one, unless we have $2m$ different matrices?

The following figure from this article can be helpful:

This figure represents "Skip-Gram model structure. Current center word is 'passes'".


text:
What is the return-to-go in reinforcement learning?
In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by increasingly smaller numbers, so that the rewards closer to the current time step have a higher weight. You can also have $n$-step returns or $\lambda$-returns.
Recently, I have come across the concept of return-to-go in a few research papers, such as Prioritized Experience Replay (appendix A. Prioritization Variants, p. 12) or Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy (section Theoretical Analysis, p. 3).
What exactly is the return-to-go? How is it mathematically defined? In which situations do we need to care about it? The name suggests that this is the return starting from a certain time step $t$, but wouldn't this be the same thing as the return (which is defined starting from a certain time step $t$ and often denoted as $G_t$ for that same reason)?
There is also the concept of reward-to-go. For example, the reward-to-go is analyzed in the paper Learning the Variance of the Reward-To-Go, which states that the expected reward-to-go is the value function, which seems to be consistent with this explanation of the reward-to-go, where the reward-to-go is defined as
$$\hat{R}_{t} \doteq \sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)$$
We also had a few questions that involve the reward-to-go: for example, this or this. How is the return-to-go related to the reward-to-go? Are they the same thing? For example, in this paper, the return-to-go seems to be used as a synonym for reward-to-go (as used in this article), i.e. they call $R(t)$ the "return to-go" (e.g. on page 2), which should be the return starting from time step $t$, which should actually be the reward-to-go.

"return-to-go" is probably not standard terminology, but I think that some people will also have this question (if they come across the same papers), that's why I asked it (i.e. to serve as a future reference).

so have you figured it out?

Short answer: I the return-to-go is just a synonym of the reward-to-go, which instead is a well known term used to define the basic form of policy gradient, e.g. REINFORCE-like.
Reward-to-go
As you correctly wrote, the reward-to-go is just a sum of rewards collected in a trajectory $\tau$ from a certain timestep $t$ until termination $T$. If we define $R(\tau)=\sum_{t=1}^T r_t^{\tau}$ as the sum of immediate rewards $r_t$ got in trajectory $\tau$, from that we can define the rewards-to-go just by considering a subset of the trajectory, so discarding the past, i.e starting from a time $t$:
$$R_t(\tau)=\sum_{t'=t}r_{t'}^\tau$$
In particular, the quantity $R_t(\tau)$ figures in the (basic) policy gradient formulation:
$$\nabla_\theta J(\theta)\approx \sum_i\Big(\sum_{t=1}^T \nabla_\theta\log \pi_\theta\big(a_t^{(i)}\mid s_t^{(i)}\big) R_t\big(\tau^{(i)}\big)  \Big)$$
Basically, the rewards-to-go score actions within a trajectory $\tau^{(i)}$: the spinning-up rl article you mentioned refers to the fact that the original (although naive) PG formulation uses $R(\tau)$, i.e. sum of rewards, to score the policy's actions but it's useless to consider the immediate rewards $r_k$ gained in the past (so when $k<t$) to score actions happened at time $t$ and onward until the end of the trajectory $T$. In this regard, both $R(\tau)$ and $R_t(\tau)$ are empirical estimates of the value of an action $a_t$ from a given state $s_t$ - think of it as a single-sample estimate of the action-value function, $Q(s_t,a_t)\approx R_t$ - but $R(\tau)$ has a higher variance due the additional sums.
This is reasonable, since in more recent PG formulations we subtract a baseline function $b(s)$ to the rewards-to-go to reduce the variance of the gradient estimate: $\nabla_\theta\log\pi_\theta(a_t\mid s_t)(R_t - b(s_t))$. If $b(s)=V(s)$ is chosen to be the value function, then $R_t - b(s_t)$ is an estimate of the advantage function $A(s_t, a_t) = Q(s_t,a_t) - V(s_t)$ since $R_t$ is the action-value gathered in the current trajectory (but not expected, as in the Q-function.) Formulations that use $A(s,a)$ yield the gradient with lowest variance: A2C uses this.

In which situations do we need to care about it?

We talk about rewards-to-go mainly when dealing with the PG estimation, and defining new baseline functions to improve the gradient estimate: otherwise the term return is largely preferred. Note: it is demonstrated that subtracting a baseline does not alter the gradient formulation, and if correctly picked it does not either add bias to the estimate.

the reward-to-go is analyzed in the paper which states that the expected reward-to-go is the value function

The rewards-to-go are more appropriate as action-value estimates rather than state-value estimates (i.e. the return $G_t$) since we start from a state-action pair. This is a technicality that does not hold in practice, since the value function is also estimated with empirical returns (i.e. the return of a single trajectory $\tau$), i.e. $G_t(\tau) = r_t^\tau + r_{t+1}^\tau +\cdots + r_T^\tau$ that correspond to the rewards-to-go $R_t(\tau)$: think about learning a value function in deep RL, we just fit the neural-net to empirical returns than MSE + GD does the magic. Now, if we average the collected rewards or returns over all possible trajectories we obtain the expected return that is the value function: $V(s_t) = \mathbb{E}_\tau[G_t(\tau)] = \mathbb{E}_\tau[R_t(\tau)] = \mathbb{E}_\tau[r_t^\tau + r_{t+1}^\tau + \cdots + r_{T}^\tau\mid s_t]$.

How is the return-to-go related to the reward-to-go?

They are same thing. I'd say to avoid "return-to-go" and use return, instead (or at most reward-to-go when talking about PG.)

As a side note: none of both terms are mentioned in the famous book "An introduction to reinforcement learning" by Sutton and Barto, so I guess it's something introduced in modern deep RL.


text:
What is the right way to train a generator in a GAN?
I am not fully understanding how to train a GAN's generator. I have a few questions below, but let me first describe what I am doing.
I am using the MNIST dataset.

I generate a batch of random images (the faked ones) with the generator.

I train the discriminator with the set composed of faked images and real MNIST images.

After the training phase, the discriminator modifies the weights in the direction of recognizing fake (probability 0) from real (probability 1) ones.

At this point, I have to consider the combined model of generator and discriminator (keep untrainable the discriminator) and put in the generator as input the faked images with the tag of 1s (as was real one).

My questions are:
Why do I have to set to real these fake images, and what fake images are these? The one generated in the first round from the generator itself? Or only the one classified as faked by the discriminator? (Then they could be both real images classified wrongly or fake images classified in the right way). Finally, what the generator does to these faked images?

Why I have to set to real these fake images and what fake images are these?

You set them to "real" label for the discriminator when training the generator, because that is the goal of the generator, to produce an output of 1 (probability of being a real image) when tested.
Usually you will generate a new batch of generated images for this step in training. You just used the last generated mini-batch to train the discriminator, so you expect them to score worse. Sending the exact same images again will cause correlation between the two minibatches that you want to avoid. It would not be a disaster, but training GANs can be quite difficult and sensitive to details like this, so it is better to keep generating new images and not re-use the previous ones.

The one generated in the first round from the generator itself?

No. New images generated just for training the generator.

Or only the one classified as faked by the discriminator? (then they could be both real images classified wrongly or fake images classified in the right way).

No. New images generated just for training the generator.
Out of interest though, if the discriminator classifies a fake image as 100% real (with a probability close to 1), then the generator will not learn anything from that. The gradients would all be zero.

Finally what the generator does to these faked images?

Nothing is done to the images themselves - unless perhaps you are keeping some copies to render and monitor training progress etc. The images occur within the combined generator/discriminator network, effectively as a hidden layer. The images are represented as artificial neuron output, so they are involved in backpropagation calculations for that layer (with no difference to any other hidden layer in a CNN), but are not changed directly.
The generator uses the gradients calculated from the combined discriminator/generator network to update its weights using gradient descent. Importantly in this phase of the updates, the discriminator weights are not changed.
In terms of training the generator/discriminator combined network to update the generator:

The input to the combined network is some new random input vectors (typically a vector with independent truncated normal distribution for each element).

The "ground truth" label is 1 for every item.

The discriminator parameters must be "frozen", somehow excluded from being updated.

Run the minibatch forward to get loss and backpropagate to get gradients for the whole network including the generator.

Apply a gradient step (usually via some optimiser, such as Adam).

Many thanks for the prompt answer! Said that, how the algorithm works after that? We have the generator trained with this batch and now it is the discriminator turn right? So I use a new generated batch from the generator + real data set, and then inject in the discriminator alone, that does the new update on weights and so on,  (it follows then another round of combined gen and discr to train generator with fix discriminator.....) until the nash equilibrium if obtained. Am I right?

@AntonYellow Yes then you go back to (1) in your description and repeat. I don't think it is really a Nash equilibrium - although there is a game-like feel to it, the networks are not trying to optimise decisions or actions, each only has one action and is trying to improve their chance of success. The equilibrium point depends critically on how well the training data of real images covers the population of all possible images of the thing you want the generator to learn, plus the limits of sophistication of the dsicriminator

[Run the minibatch forward to get loss and backpropagate to get gradients for the whole network including the generator.
Apply a gradient step (usually via some optimiser, such as Adam).]

Ok good for the Nash eq. Get back on what we were, the last two bullet points in your answer mean that we run the minibatch and get the loss respect the overall cost function (gen+discriminator) and then backpropagate the signal towards the generator, but the discriminator is not touched by this update, right? If it is the case, why use a combined model and don't stick only with the separated generator. I am near the answer but I need a further push:-) to understand why this interconnection is needed

@AntonYellow: You need the discriminator included in order to have a measurable loss value for the generator's output, plus the gradients associated with that loss. The generator cannot compare to the goal of creating a realistic fake by itself - it outputs an image, and has no way to get a score for that image. If this was not a GAN, and if you expected a specific/precise image then you could get a score, e.g. if you knew what exact image a generator should make given the input, then you could measure the difference and get a gradient. But with a GAN you cannot, the image is random.

I'd like to add some details to the Neil Slater's answer.
In order to generate data, we want to find some unknown distribution. Since we do not know anything about a real distribution, we can approximate it using GAN. It was shown that optimizing the loss function of the original GAN is equivalent to
minimizing Jensen-Shannon divergence between the real distribution and the estimated (generator) distribution.

where q is our initial distribution and p is the target distribution.
While training the Discriminator as a normal classification model (which learns to distinguish between generated and real samples), we want to push the Generator towards the real data distribution. To make the generated samples look realistic, we pass true labels to calculate the gradient in that direction. Here is the intuition behind generator training:


text:
Why would the reward of A3C with LSTM suddenly drop off after many episodes?
I am training an A3C with stacked LSTM.
During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. Is it because of LSTM?
Is it normal?
Should I expect it to work after the training is over or just terminate the training and increase the density of my network?

The thing you're explaining is not impossible for a RL model, but it's rare. That's a known thing that some RL algorithms work or don't work depending on a random seed. I implemented the same model once to play KungFuMaster-v0. It was during a university RL course and the code seemed fine (actually 2 people including teacher looked at it very carefully and they didn't find any bugs). I remember calling it 10 times a row and one time out of ten it showed that nasty behavior. There were teacher's notes in the task: if the reward suddenly drops to 0 and stay there for a long time, check your code, there's a high probability of a bug. So I'd say if your net works just fine 9 times out of 10, probably there are no bugs, otherwise if I were you, I'd carefully check the code.


text:
How to perform prediction when some features have missing values?
Sorry if this is too noob question, I'm just a beginner.
I have a data set with companies' info. There are 2 kinds of features: financial (revenue and so on) and general info (like the number of employees and date of registration)
I have to predict the probability of default. And the data has gaps: about the half of the companies have no financial data at all. But general features are 100% filled.
What is the best practice for such a situation?
Will be great if you can give some example links to read.

May help https://stats.stackexchange.com/questions/98953/why-doesnt-random-forest-handle-missing-values-in-predictors

You should look into "missing values". This is an entire research field in itself.
First, you need to identify the type of missing values:

They can be missing purely at random.
Whether they are missing or not is itself a useful feature, and should be treated as a class of its own.

(Those two are the best case scenarios.)

Whether they are missing or not depends on the underlying (unknown) value. For example, a thermometer might fail occasionally if the temperatures get too high. In your case, certain types of companies might be more likely to not share their information.
Information might be missing specifically to mislead you, the data analyst. This is the worst possible scenario, and there is not much you can do.

So, what do you do about it? A few typical options:

Throw out all the rows with missing data: we do not have enough information about these companies.
Throw out all the columns with missing data: this field is not reliably measurable and we shouldn't use it.
Try to guess the missing values. This can be done if the amount of missing data is small. Either you train a predictive model based on the non-missing data, or you fill in the median for that type of row, or you fill in the value of the "closest" matching row. This can be dangerous.
Some algorithms are OK with missing data. Check the documentation for your models and algorithms to see how they deal with missing values.

Thanks for an answer! I've got this data in some competition, so it is possible that missing values were selected randomly (but on purpose) by orgs. It looks like a real situation: some companies - are already clients, and some are new so we don't know much. I think that in my case throw all columns is bad - we'll miss too much data. And throw out all this rows also not an option, because test data has the same structure - half of record without financial info.

I think it will be good to find out whether the fact of missing finance is a feature. But can i make (and how) more than 2 classes on presented data? How can i categorise several features into a new feature ? %)

I'd introduce a new boolean feature: "is_missing". Then you can see if it correlates with any of the known features. To see if it can be predicted, take the part of the data set where it is available, and train a predictor on that particular feature. Depending on how well it does (validation score) you can try to predict the missing ones.

Essentially you're treating the possibly missing value as the target of a prediction problem.