text:
How is AI helping humanity?

There was a lot of Negative news on Artificial Intelligence. Most people were first exposed to the idea of artificial intelligence from Hollywood movies, long before they ever started seeing it in their day-to-day lives. This means that many people misunderstand the technology. When they think about common examples that they’ve seen in movies or television shows, they may not realize that the killer robots they’ve seen were created to sell emotional storylines and drive the entertainment industry, rather than to reflect the actual state of AI technology.

There are few questions on our SE on how AI impacts/harms humankind. For example, How could artificial intelligence harm us? and Could artificial general intelligence harm humanity?
However, now, I'm looking for the positive impacts of AI on humans. How could AI help humankind?

I am not sure there is a consensus defining AI (e.g. two persons would disagree on what is artificial intelligence, and what was AI 20 years ago is no more AI in 2020). But you could look into this blog (of Jacques Pitrat, one of the first French AI researcher, he died in oct. 2019). You might be interested in RefPerSys

For good or bad, AI is the next step in automation. The impact which is already visible, and trends show will continue in the future, is the eradication of repetitive and body-straining labor.
Hopefully, the transformation will be gradual enough for the global labor market to re-adjust, otherwise, we'll face a problem of growing unemployment. It seems to me that we've become aware enough to foresee bad outcomes of our inventions, hence in almost every dimension affected by AI, a plausible and either positive or negative future can be presented, depending on the sentiment of the storyteller.
Regardless of what different experts and sci-fi writers tell us about the future, actually predicting it is a futile endeavour. Considering that predictions made for a dynamic system, even when we have a lot of data and good models (like about the weather), become unreliable just for a few weeks ahead.

We've already seen significant progress in fields that we could not even come close to prior to the explosion in AI research. For example, the automated identification of cancerous tumours in lung tissues could save countless lives, as a computer never tires and has no bias. Incredible advances in speech synthesis can allow people who lost or never had a voice to have their own human-sounding voice. Personally, I also predict soon we will be able to synthesise music to an individual's tastes.
Advances in rapid image processing, thanks to the polynomial complexity of machine learning techniques, allows for advanced guidance systems and autonomous cars that are only improving. Ranking algorithms for listings allow for finely tuned results for a user's search. Unsupervised techniques can help us separate suspicious activity in banking records to catch fraudulent transactions.
There is a very, very long list here and this only scratches the surface. AI is already benefiting us enormously, to the point where very soon, if not already, I would say we will become so dependant on learning techniques that if they were outlawed, entire systems may collapse.

You say "as a computer never tires and has no bias", but this may not be true. Often, machine learning systems are trained with data annotated or produced by or about humans, so this data can have biases. This is already an issue and reality in current ML systems.

@nbro This is true, but inherently can never be worse than the bias in the data the humans provided. I guess a better way of wording it would be that a computer can only be as biasd as the human creates it.


text:
What is meant by degrees of freedom of latent variables?

...Designing such a likelihood function is typically challenging; however, we observe that features like spectrogram are effective when latent variables have limited degrees of freedom. This motivates us to infer latent variables via methods like Gibbs sampling, where we focus on approximating the conditional probability of a single variable given the others.

Above is an excerpt from a paper I've been reading, and I don't understand what the author means by degrees of freedom of latent variables. Could someone please explain with an example, or add more details?

References
Shape and Material from Sound (31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA)

A good example is the degree of freedom in Student's distribution:

 The degrees of freedom refers to the number of independent observations in a set of data.

For example:

When estimating a mean score or a proportion from a single sample, the number of independent observations is equal to the sample size minus one.

e.g, if we have 100 observation $X_1, \ldots, X_{100}$ and we want to estimate their mean $\bar{X}$, as $\bar{X} = \frac{X_1 + \cdots + X_{100}}{100}$, if we know the mean, just we need to find the value of 99 variables from $X_1, \ldots, X_{100}$. Hence, here the degree of freedom is 99.
Your referenced paragraph is a general explanation in the paper as well. However, base on the above example, the degree of freedom in the paragraph depends on the likelihood function and number of observations that we have from spectrograms.
Now, as the DoF of latent variables is not high, using Gibbs sampling we will approximate some observations, and then using them we will compute the value of the latent variables.

This seems to be correct, but maybe you should explain what would be the degrees of freedom of a latent variable in the specific context of that paper.

How does Gibbs sampling help in this context?

@cogito_ai Gibbs sampling helps us to approximate some observations from the distribution and compute the value of latent variables.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that notation:
$$\frac{\varepsilon}{|\mathcal{A}(s)|} \sum_{a} Q^{\pi}(s, a)+(1-\varepsilon) \max _{a} Q^{\pi}(s, a)$$
Here is the source of the formula.
I also want to clarify that I understand the idea behind the $\epsilon$-greedy approach and the motivation behind the on-policy methods. I just had a problem understanding this notation (and also some other minor things). The author there omitted some stuff, so I feel like there was a continuity jump, which is why I didn't get the notation, etc. I'd be more than glad if I can be pointed towards a better source where this is detailed.

From the pseudo code, it is pretty clear that $A(s)$ refers to the set of all possible actions, since in step c) the algorithm iterates through all actions ($a$) (taken from that set). That it is about the actions becomes apparent from the use of $a$.

Yes I realized that I was talking more about the notation $|A(s)|$ but I get it now. Thanks.

This expression: $|\mathcal{A}(s)|$ means

$|\quad|$ the size of

$\mathcal{A}(s)$ the set of actions in state $s$

or more simply the number of actions allowed in the state.
This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.

Comments are not for extended discussion; this conversation has been moved to chat.

authors: Роберт Фитагдинов

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Doesn't the number of explored nodes with IDA* increase linearly?
I think I'm misunderstanding the description of IDA* and want to clarify.
IDA* works as follows (quoting from Wiki):

At each iteration, perform a depth-first search, cutting off a branch when its total cost exceeds a given threshold. This threshold starts at the estimate of the cost at the initial state, and increases for each iteration of the algorithm. At each iteration, the threshold used for the next iteration is the minimum cost of all values that exceeded the current threshold.

Suppose that we have the following tree:

branching factor = 5
all cost are different

Say we have expanded 1000 nodes. We pick the lowest cost of the nodes that we 'touched' but didn't expand. Since all costs are unique, there is now only one more node which satisfies this new cost bound, and so we expand 1001 nodes, and 'touch' 5 new ones. We now pick the smallest of these weights, and starting from the root expand 1002 nodes, and so on and so forth, 1003, 1004...
I must be doing something wrong here right? If not, the complexity is $n^2$, where n is the number of nodes with cost smaller than the optimum, compared to n for normal A*.
Someone pointing out my misunderstanding would be greatly appreciated.

I think you only restart the search from the root note if you have reached the bound and not found the goal node. I think this can be seen from the pseudocode. I am not sure if this answers your question (and if this is completely correct, because it's been a while since I had to do something with IDA*), but it seems like you are assuming that you restart the search from the root everytime you expand a node. Anyway, I will maybe give you a more complete answer later (if I don't forget about this).

@nbro I believe they were saying that the search starts from the root every time you expand a node because each node has a different cost. If you choose each new bound to be the minimum cost of the nodes that were cut off, and no node has the same cost, each restart only results in one extra node being expanded: in effect, we restart from the root every time we expand a node.

No misunderstanding. IDA* is indeed n^2 vs n for A* in certain situations.
See
https://pdfs.semanticscholar.org/388c/0a934934a9e60da1c22c050566dbcd995702.pdf
for a reference.
IDA* only works in certain (not uncommon) situations. If there are many states with the same costs, and this number grows exponentially with the cost, then IDA* will be O(n). An example of such a problem is the 8 sliding piece puzzle.
IDA* will however, as noted, become exponential when all costs differ. Examples include continuous costs, or discrete values spanning a large range.
If you really wanted to apply IDA* to e.g. a map, a workable approach would likely be to round all values - or perform a log+floor operation.

Note that there is a replacement for IDA*, BTS, which has much better worst-case performance: O(n log(C*)) where C* is the optimal solution cost.


text:
Can rewards be decomposed into components?
I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like  reward = distance_x + distance_y + standing_up_straight, and then it adds this reward to the replay buffer. However, I think that it would be more efficient if it can break the reward down by category, so it can figure out "that action gave me a good distance distance_x, but I still need work on distance_y and standing_up_straight".
Are there any existing algorithms that add rewards this way? Or have these been tested and proven not to be effective?

If I understood correctly you're looking at a Multi-Objective Reinforcement Learning (MORL). Keep in mind however  that many scientist will often follow the reward hypothesis (Sutton and Barto) which says that

All of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

The argument for a scalar reward could be that even if you define your policy using some objective vector (as in MORL) - you will find a pareto bound of optimal policies, some of which favour one component of the objective over the other - leaving you (the scientist) responsible for making the ultimate decision concerning the objectives' tradeoff - thus eventually degenerating the reward objective into scalar.
In your example there might be two different "optimal" policies - one which results in a very high value of distance_x but relatively poor distance_y and a one that favours distance_y instead. It'll be up to you to find the sweet spot and collapse a reward function back to a scalar.

I agree with Tomasz that the approach you are describing falls within the field of MORL. For a solid introduction MORL I would recommend the survey by Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.
https://www.jair.org/index.php/jair/article/view/10836 (disclaimer: I'm an author in this, but I genuinely believe it will be useful to you).
Our survey provides arguments for the need for multiobjective methods by describing three scenarios where agents using single-objective RL may be unable to provide a satisfactory solution which matches the needs of the user. Briefly these are (a) the unknown weights scenario where the required trade-off between the objectives isn't known in advance, and so to be effective the agent must learn multiple policies corresponding to different trade-offs and then at run-time select the one which matches the current preferences (eg this can arise when the objectives correspond to different costs which vary in relative price over time; (b) the decision support scenario where scalarization of a reward vector is not viable (for example, in the case of subjective preferences which defy explicit quantification), so the agent needs to learn a set of policies, and then present these to a user who will select their preferred option, and (c) the known weights scenario where the desired trade-off between objectives is known but its nature is such that the returns are non-additive (ie if the user's utility function is non-linear) and therefore standard single-objective methods based on the Bellman equation can't be directly applied.
We propose a taxonomy of MORL problems in terms of the number of policies they require (single or multi-policy), the form of utility/scalarization function supported (linear or non-linear), and whether deterministic or stochastic policies are allowed, and relate this to the nature of the set of solutions which the MO algorithm needs to output. This taxonomy is then used to categorise existing MO planning and MORL methods.
One final important contribution is identifying the distinction between maximising Expected Scalarised Return (ESR) or Scalarised Expected Return (SER). The former is appropriate in cases where we are concerned about the results within each individual episode (for example, when treating a patient - that patient will only care about their own individual experience), while SER is appropriate if we care about the average return over multiple episodes. This has turned out to be a much more important issue than I anticipated at the time of the survey, and Diederik Roijers and his colleagues have examined it more closely since then (eg http://roijers.info/pub/esr_paper.pdf)

Given that this answer is basically a copy-and-paste of your other answer, I would actually suggest that, in this case, you just give a little introduction and ask the OP to read your other answer. In this case, it's fine because the answer is on the site (and this is really just a copy and paste of the other).


text:
How can I classify policy gradient methods in RL?
In the book of Barto and Sutton, there are 3 methods presented that solve an RL problem: DP, Monte Carlo, and TD. But which category does policy gradient methods (or actor-only methods) classify in? Should I classify them as the 4th method of solving a reinforcement learning problem?

What category do you consider that applies to DP, Monte Carlo and TD? You seem happy that these are categorised. For someone to answer, they will need to understand better how you are thinking about this. For instance, do you consider DP, Monte Carlo and TD to be categories of RL and you want to know which of these you should consider PG methods as?

DP, Monte Carlo, and TD are methods of estimating returns. Policy gradient describes methods of learning a policy. So policy gradients serve a different purpose than the other things you mentioned. For clarity, you can use Monte Carlo or TD methods to estimate returns to construct the loss that you get your policy gradient from.


text:
Should forecasting with neural networks only be treated as a supervised learning (regression) problem?
I have recently made a work about the application of neural networks to time series forecasting, and I treated this as a supervised learning (regression) problem. I have come across the suggestion of treating this problem as an unsupervised, semi-supervised, or reinforcement learning problem. The ones that made this suggestion didn't know how to explain this approach and I haven't found any paper of this. So I found myself now trying to figure it out without any success. To my understanding:
Unsupervised learning problems (clustering and segmentation reduction) and semi-supervised learning problems (semi-supervised clustering and semi-supervised classification) can be used to decompose the time series but not forecast it.
Reinforcement learning problems (model-based and non-model-based on/off-policy) is to decision taken problems, not to forecast.
It is possible to treat forecasting time series with neural networks as an unsupervised, semi-supervised, or reinforcement learning problem? How it is done?

I think the choice of technique strongly depends on how fine-grained your forecast-predictions need to be.
When it comes to forecasting by Reinforcement Learning (RL), one prominent example is the stock-trading RL agent. The agent must decide which stock to buy or sell, thereby drawing upon predictions concerting the expected future development of some stock.
Given this approach, you would not necessarily let the RL agent explicitly generate estimates of how stock prices are gonna develop at any point, but instead you would only observe the predicted decision concerning whether to buy or sell etc.
But if you think hard enough, I am certain that you could come up with setups of RL agents that would allow you to explicitly generate future estimates of values to be forecast. In this case, the final buy/sell decision would have to depend on the explicit future stock price predictions to enforce accurate predictions.
Concerning unsupervised learning, you could cluster data points (training samples) with respect to how some value(s) of interest changed $t$ time steps in the future (after having observed the training sample). Then, you could associate clusters with rough forecast-estimates. After all, you would treat the forecast value as a label associated with data points.
Afterwards, you could use some kind of nearest neighbor approach to determine which cluster is closest to some novel data sample. Then, you take as a prediction for the new data sample the forecast prediction (i.e. label) that is associated with the closest cluster/prototype etc. But strictly speaking, as soon as you start turning forecast values (which were previously part of some unlabeled time-series dataset) into labels, you turn the training procedure of course into a supervised technique again.
How well especially the latter training approach would work, I can't tell since I have never heard anyone using this method. But if training data is too scarce to employ some deep learning method, why not giving it at least a try if accuracy doesn't have to be too precise?
After all, it's just a matter of creativity and testing which method works best given your specific machine learning problem at hand.


text:
Do I need to rotate the masks, if I also rotate the images and the masks are generated from the input?
I am training a neural network that takes an input (H, W, 3) and has the output of size (H', W', C). Now, to augment my dataset, since I only have 45k images, I am using the following in my custom data generator
def Generator():
img=cv2.imread(trainDir+'\'+imgpath)
img=tf.keras.preprocessing.image.random_rotation(img,20m,row_axis=0,col_axis=1,channel_axis=2)

output_mask=np.load(trainDir+'\'+maskpath)

yield(img/255-.5,output_mask)

Since I am rotating my input images and the output_masks are generated from information about the input (specifically, heat maps around the joint locations) do I also need to rotate the masks as well?

Yes! This is crucial.
If you rotate your input images for segmentation, you need to rotate the output masks as well. Otherwise the loss of your network will not be correctly calculated and your network will not learn how to generalize to rotated input images.
If you use keras, you can use two ImageDataGenerator classes, one for the images and one for the masks, with the same random seed and augmentation parameters. It looks something like this:
data_gen_args = dict(rotation_range=45)

image_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(**data_gen_args)

image_generator = image_datagen.flow(train,seed=SEED)
mask_generator = mask_datagen.flow(y_train,seed=SEED)
train_generator = zip(image_generator, mask_generator)

model.compile(...)
model.fit_generator(train_generator, ...)

This is great thank you! So what I ended up doing was resizing the masks and concatenating the images with the masks and then
tf.keras.preprocessing.images.random_shear((images,masks),*args) and then resizing the masks after. Would setting the same seed be sufficient to make sure the rotations or shearing agreed on both.
Lastly, is there anyway to use ImageDataGenerator with a custom generator also?


text:
How does batch normalisation actually work?
I actually went through the Keras' batch normalization tutorial and the description there puzzled me more.
Here are some facts about batch normalization that I read recently and want a deep explanation on it.

If you froze all layers of neural networks to their random initialized weights, except for batch normalization layers, you can still get 83% accuracy on CIFAR10.

When setting the trainable layer of batch normalization to false, it will run in inference mode and will not update its mean and variance statistics.

Which tutorial are you exactly referring to? Can you please edit your post to include the link to the tutorial that states those facts?

I'm not sure how just training the batch normalisation layer, you can get an accuracy of 83%. The batch normalisation layer parameters $\gamma^{(k)}$ and $\beta^{(k)}$, are used to scale and shift the normalised batch outputs. These parameters are learnt during the back-propagation step. For the $k$th layer, $$y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)}$$ The scaling and shifting are done to ensure a non - linear activation being outputted by each layer. Because batch normalisation scales outputs between 0-1, some activation functions are linear within that range (Eg. $tahh$ and $sigmoid$)
Regarding the second fact however, the difference between training and inference mode is this. During training mode, the statistics of each batch norm layer $\mu_B$ and $\sigma^2_B$ is computed. This statistic is used in scaling and normalising the outputs of the batch norm layer to have 0 mean and unit variance. At the same time, the current batch statistic computed is also used to update the running mean and running variance of the population. $\mu_B[t]$ represents the current batch mean, $\sigma^2_B[t]$ represents the current batch variance, while $\mu'_B[t]$ and $\sigma'_B[t]$ represent the accumulated means and variance from the previous batches. The running mean and variance of the population is then updated as $$\mu'_B[t]=\mu'_B[t]× momentum+ \mu_B[t]×(1−momentum)$$
$$\sigma'^2_B[t]=\sigma'^2_B[t] × momentum + \sigma^2_B[t]×(1−momentum)$$
In inference mode, the batch normalisation uses the running mean and variance computed during training mode to scale and normalise inputs in the batch norm layer instead of the current batch mean and variance.

A batch normalisation layer is like a standard FC layer but instead of learning weights and bias', you learn means and variances and scale the whole layer by said means and variances.
Fact 1:
Because it behaves just like a normal layer, and can learn, with the right structure it will learn to get a high enough accuracy.
Fact 2
Disabling learning on a batch norm layer is just like disabling learning on any other layer. It will not updated any of its parameters, and in this case the parameters are the means and variances, and so these will not be updated.

The original paper by Sergey Ioffe and Christian Szegedy; https://arxiv.org/abs/1502.03167 "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" is very good. Make sure to go through the paper slowly and make annotations to truly understand it.


text:
When is adding a feature useless?
I'm building a model, where, from a feature set A, I want to predict a target set C. I need to understand if another feature set B, together with A, can improve my model performances, instead of using only A.
Now I want to check if I can predict B directly from A, since, in my understanding, this would mean that info on B is already inside A.
If I get good predictions when testing the model A -> B, is it true then that adding B to A in predicting C is completely useless?
And furthermore, are there smarter ways to decide if/when a feature is useless?

Now I want to check if I can predict B directly from A, since, in my understanding, this would mean that info on B is already inside A.

This will help inform you how much redundancy there is between A and B. However, even if you can predict B with 100% accuracy from A, you may still be better off using A+B (instead of A alone) to predict C.

If I get good predictions when testing the model A -> B, is it true then that adding B to A in predicting C is completely useless ?

It is an indicator that adding B probably won't make great improvements to your prediction of C.
The only way to be sure is to make the model that uses A+B and compare its performance against a model that uses only A. If collecting B costs time or other resources, then perform this check by limiting both models to only learn from the subset of data where you have all of A, B, C available.

And furthermore, are there smarter ways to decide if/when a feature is useless ?

Another thing you can do is to try and predict C from B alone. It doesn't need to score well, but may indicate that something useful is in the data if you get better than chance results. Scoring badly unfortunately doesn't rule it out for working well in combination with A.
Generally, if you cannot reason it clearly one way or another from theory, the accepted method is to build variations of your model with and without a feature and test them. You do have to be aware of the chances of spurious correlation though, so usually you need to have some basis or motivation for considering any new feature, from your domain knowledge that applies to the model.


text:
Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?
Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?
Here's the update formula.

Removing the learning rate will likely yield poor convergence to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.
Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and gradually until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $\alpha = 0.1$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their book, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.

May be worth noting there are some situations in which a learning rate of 1 works quite well. That includes quite a few simple scenarios use to teach concepts, like deterministic grid worlds etc.

@NeilSlater Since what you said is great information that would help the answer, am I supposed to edit my answer to include the info, or do I leave it as a comment? If I edit my answer to include it, do I give you attribution? I'm still a little new here and am trying to figure out the unwritten rules - I didn't see this explicitly mentioned in the rules when I just check them.

The comments attached to a question or answer are generally supposed to be suggestions for improvement. They also may be deleted for any reason. So please do take my suggestion and edit something appropriate into the answer to cover it, if you agree. I don't require attribution. I also think for most cases this spoils the flow of the answer. The ideal is that the new parts are edited in so that the answer stands by itself, so no need to flag changes either. Some Stack Exchange sites do have a different culture around this, and some writers like to credit sources. Up to you here


text:
If the normal equation works, why do we need gradient descent?
Recently, I followed the open course CS229,
http://cs229.stanford.edu/notes/cs229-notes1.pdf
This lecturer introduces an alternative approach to gradient descent that is called "Normal Equation" and the equation is as follows:
$$\theta=\left(X^{T} X\right)^{-1} X^{T} \vec{y}$$
The normal equation can directly compute the $\theta$.
If the normal equation works, why do we need gradient descent? What is the trade-off between these two methods?

That normal equation is sometimes called the closed-form solution.
The short answer to your question is that the closed-form solution may be impractical or unavailable in certain cases or the iterative numerical method (such as gradient descent) may be more efficient (in terms of resources).
This answer gives you more details and an example.


text:
What does it mean when a model "statistically outperforms" another?
I was reading this paper where they are stating the following:

We also use the T-Test to test the significance of GMAN in 1 hour ahead prediction compared to Graph WaveNet. The p-value is less than 0.01, which demonstrates that GMAN statistically outperforms Graph WaveNet.

What does "Model A statistically outperforms B" mean in this context? And how should the p-value threshold be selected?

You should read about hypothesis testing, null hypothesis, and p-values. These are basic statistical concepts. Maybe Khan Academy is a good place to start. The value of $p$ is typically a hyper-parameter and needs to be chosen before the experiment. To understand why, you should first understand the meaning of the p-value, which actually takes a little bit of time to grasp.

Most model-fitting is stochastic, so you get different parameters every time you train, and you usually can't say that one algorithm will always give you a better-performing model.
However, since you can retrain many times to get a distribution of models, you can use a statistical test like the T-Test to say "algorithm A usually produces a better model than algorithm B," which is what they mean by "statistically outperforms."
p-value is usually set by consensus in the field. The higher the p-value, the less confidence you have that there's a statistical difference between the distribution of values being compared. 0.1 might be normal in a field where data is very expensive to collect (like risky, long-term studies of humans), but in machine learning, it's usually easy enough to retrain a model that 0.01 is common, and demonstrates very high confidence. To know more about selecting and interpreting the values, I recommend Wikipedia's page on statistical significance.


text:
Are there any agents that are based on quantum computing?
Assuming the definition of an agent to be:

An entity that perceives its environment, processes the perceived information, and acts on the environment such that some goal is fulfilled.

Are there any agents that are based on quantum processing/computing (i.e. implemented by a network of quantum gates)?
Is there any work done towards this end? If so, could someone provide references?

I think you are looking for quantum machine learning (QML), which is a relatively new field that sits at the intersection of quantum computing and machine learning.
If you are not familiar with quantum computing (QC) and you are interested in QML, I suggest that you follow this course by prof. Umesh Vazirani and read the book Quantum Computing for Computer Scientists (2008) by Yanofsky and Mannucci. If you have a solid knowledge of linear algebra, you should not encounter big problems while learning QC. Be prepared to deal with the weirdness and beauty of qubits, quantum entanglement, and so on.
If you want to directly dive into QML (although I am not familiar with the details of QML, I suggest that you first get familiar with the basics of QC, which I am familiar with) there are already several courses, papers, overviews, and libraries (such as TensorFlow Quantum) on quantum machine learning.
If you are interested in quantum reinforcement learning, maybe have a look at the paper Quantum Reinforcement Learning (2008) by Daoyi Dong et al.


text:
Why do some DQN implementations not require random exploration but instead emulate all actions?
I've found online some DQN algorithms that (in a problem with a continuous state space and few actions, let's say 2 or 3), at each time step, compute and store (in the memory used for updating) all the possible actions (so all the possible rewards). For example, on page 5 of the paper Deep Q-trading, they say

This means that we don't need a random exploration to sample an action as in many reinforcement learning tasks; instead we can emulate all the three actions to update the Q-network.

How can this be compatible with the exploration-exploitation dilemma, which states that you have to balance the time steps of exploring with the ones of exploiting?

The example that you linked is using a model (emulation) in order to look ahead at all possible actions from any state. It essentially explores off-policy and offline using that model. This is not an option that is available in all environments, but if possible it resolves the exploration/eploitation dilemma for a single time step nicely by investigating all options.
Longer term the agent proposed by the link does not sufficiently explore for general use in my opinion. It appears to always choose a single action deterministically based on maximising action value. In other words it always attempts to exploit the training data so far, even though it augments the training data with short-term knowledge about exploration. However, this appears to be sufficient in the problem domain that it is used in. I suspect this is for a couple of reasons:

The environment is non-stationary, making long-term state predictions unreliable in any case. An agent that learns to exploit in the short term (i.e. over only a few time steps into the future) is likely to be approximately optimal already.

State transitions may be highly stochastic, meaning that state space will still be adequately explored even using a deterministic policy. This feature of the environment is also used by other well-known Q learning approaches with deterministic behaviour policies, such as TD Gammon

I think you have correctly identified a weakness of the approach used in the linked paper that means it may not make a strong general algorithm. The algorithm avoids addressing the exploration/exploitation balance in full, and instead relies on features of the environment to work well despite this. If you find yourself working in similar environments for your own projects, then it may well be worth trying the same approach. However, if you find yourself working in a more deterministic environment with more stationary behaviour and sparse rewards, the lack of state space exploration would be a serious limitation.


text:
How exactly does nested cross-validation work?
I have trouble understanding how nested cross-validation works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but why are they nested?
From what I understood, we need to select the model before training it, which points toward non-nested loops.
Could someone please explain what's wrong (or right?) with my line of reasoning, and also explain nested cross-validation in greater detail? A representative example would be great.

"Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.
Consider the example in the link. First you like to select the best hyperparameters of your svm model by GridSearchCV(). This is done by 4-fold CV. Now the clf.best_score_ will be the mean cross-validated score of the best estimator (the model with the best hyperparameters). However now you used the same data for training and reporting the performance, although you used CV. Keep in mind that the folds are not independent. Therefore the hyperparameters might be too data specific, i.e. your generalization error estimates are too optimistic. Therefore we like to evaluate our final model performance outside / independent of the hyperparameter selection loop / process (the call cross_val_score()).
In the provided plot, you can clearly see that the reported performance by GridSearchCV() is most of the time better than the performance reported by cross_val_score().


text:
How does training for DQN work if messing up in the environment in costly?
Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.
Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.

Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this?

Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that).

You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question).

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.
With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.

Ah, you're right, my mistake. Edited my response to address this.


text:
On what basis is MATLAB "inflexible" to perform ML/AI research on it?
During a course review, I have provided my opinion on the course overall. I stated that MATLAB is also a great environment to program and do research for ML/AI, but my professor seemed to have taken my comments as a joke and told me "If you take a look at the statistics, then you'll see that MATLAB is not a feasible environment to innovate and research topics in ML/AI".
As an undergraduate who is new to machine learning, I would hope to understand more and not debate on which is better (MATLAB vs Python), but rather to know whether there is a bias against MATLAB or there are actually reasons that make MATLAB not a good environment to research and program ML topics.

I think it boils down to the fact that most deep learning research is conducted using the libraries in python (pyrotechnics, tesnorflow, etc.) and so there’s a lot of existing support for it already out and any new methods are likely to be built in these libraries.

(This question is could be considered off-topic or opinion-based, but I will answer it by providing facts that could hinder the adoption of MATLAB by AI researchers).
There are 2 main reasons why MATLAB may not be the "best" programming language/environment for research (in AI and other areas too)

It's closed source (i.e. it's more difficult to extend it; the most similar open-source alternative to MATLAB would be Octave)
It's owned by a company (i.e., it's proprietary, so, for you to use MATLAB, you need a license, which could be given to you by your university, which is my case, or you need to buy it, but not everyone can afford this).

In my view, these are clear obstacles for people that want to share their research and make sure that people can reproduce their results or build new ideas on top of them. However, in some cases, using proprietary software for research may be the best (or even only) option (see e.g. this for more info). In fact, I have also seen MATLAB being used for research in artificial intelligence (here is an example), but the choice of MATLAB might have been due to the familiarity of the authors with the language/environment.


text:
Non-Neural Network algorithms for large state space in zero sum games
I was reading online that tic-tac-toe has a state space of $3^9 = 19,683$. From my basic understanding, this sounds too large to use tabular Q-learning, as the Q table would be huge. Is this correct?
If that is the case, can you suggest other (non-NN) algorithms I could use to create a TTT bot to play against a human player?

I was reading online that tic-tac-toe has a state space of $3^9 = 19,683$. From my basic understanding, this sounds too large to use tabular Q-learning, as the Q table would be huge. Is this correct?

That is a relatively small number of states that can easily be represented in a table on a modern computer. For a Q table, you would multiply by the number of moves possible in each state, but this is still a small amount of memory. Even with the most naive implementation that tracked impossible states and state/action pairs, using string state representations (so 10 bytes for each state/action key), and using double precision floating point for each action value (another 8 bytes), the full table would be around 3 megabytes in size.
So it is definitely possible to use tabular Q learning here. I have done just that whilst learning about RL - my Q-learning Tic Tac Toe agent is written in Python and available on Github. There are many ways to optimise the required space, e.g. only representing reachable states. Also in games with perfect control (moves directly result in a new state), it is common to use afterstate values instead of action values.

If that is the case, can you suggest other (non-NN) algorithms I could use to create a tic-tac-toe agent to play against a human player?

It is not the case, as explained. However, there are two basic approaches at the top level that are worth learning about for working with game trees:

Forward planning or search algorithms, as mentioned by skillsmuggler in their answer. These use game rules plus a heuristic (measure of likely success) to explore the future states of the game and pick the search result with the best heuristic. Minimax with a heuristic only based on win/lose at the end is a very basic approach, but would cope just fine with Tic Tac Toe.

Other planning algorithms include Negamax (a minor variation of Minimax for zero-sum games) and MCTS (famously used in Alpha Go).

Policy function improvement, which generate policies - maps of current state to actions - and have ways to assess them and select better policies.

Q learning, and the many algorithms of reinforcement learning are in this category, as well as genetic algorithms.
Tic Tac Toe is simple enough that you could hard-code a policy function that mapped any state to the next action. The xkcd webcomic encoded this policy into a diagram.

These two approaches are complementary, in that both can be used together to solve more sophisticated problems. For instance, value-based reinforcement learning algorithms - including Q learning - can be used to provide learned heuristics for a search algorithm. The categories I suggest above are not strict either, in that some algorithms are not clearly one thing or another.

In the case of TicTacToe, you can make use of game theory. The entire search space can be denoted by a game tree. You bot must now be able to maximize the chance of winning.
You can make use of the Maximin algorithm. This is still computationally intensive on large search spaces. To improve the efficiency Alpha-Beta pruning can be applied to reduce the number of nodes in the Game tree.
These are core AI concepts and will always perform better than neural networks on dully defined and relatively smaller search spaces. Neural networks perform better when it's too difficult to compute all the possible combinations of a game at a certain state.
You can have a look at this to build a TicTacToe bot.

Thank you. Would you also suggest the Maximin algorithm for games with even larger state space; for example, Chess?

No. The Miaximin algorithm requires the entire search space to be properly defined. Incase of chess this is not possible as there are too many possibilities. This is also due to the fact that the different pieces follow different movement rules. This is the main reason why developing chess agents is very challenging.

Thank you. When you say too many possibilities, do you mean too many states / too large of a state space? Then, if the different pieces with their respective movement rules causes too many states, how about a game like Connect Four, where the pieces all follow the same movement rules? Could Minimax be used for that, or is the state space too large for that too? If so, what alternatives could be used?

In the case of chess, yes the search space is too large. Both in terms of space complexity and time complexity. Have a look at this should give you an idea of the complexity of the task.

For connect 4, you can consider it to be a relatively small search space. This again depends on the computing resources. For connect 4 you can make use of Minimax.

Thank you. How do you know what Minimax can and can't be used for? What's the state space 'cut off' for Minimax?

It's an algorithm. In theory it can be used for all 0/1 games. The threshold is determined by your computing resources. A larger state space requires more resources. If you have them available, you can make use of Minimax. In case of chess, even super computers do not have enough computing power to implement Minimax.


text:
What would be the importance sampling ratio for off-policy TD learning control using Q values?
The off-policy TD learning control using state value function from page 34 of David Silver's RL lecture is:
$$ V(S_t) \leftarrow V(S_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \gamma V(S_{t+1})) - V(S_t) \right). $$
I'd like to change this update rule to action value function Q, something like:
$$ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (R_{t+1} + \gamma Q(S_{t+1},A_{t+1})) - Q(S_t,A_t) \right). $$
Then what is the corresponding importance sampling ratio?
Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not?

Since $A_t$ is already determined (because we are calculating $Q(S_t,A_t)$), I think $\pi(A_t|S_t)$ is definitely 1. But what about $\mu (A_t|S_t)$? Is it 1 or not?

You could assign values of 1 to each to get the right answer, but the situation is different. You can see that more clearly in the definition of action value, $q(s,a)$:
$$q_{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1}|S_t=s, A_t=a]$$
The condition on the expectation $|S_t=s, A_t=a$ means that the value of $A_t$ is already assumed. There is no need to know or use the associated probability, and no need to adjust between probabilities of different policies when estimating returns for the first action choice.
In addition, the last action choice used to bootstrap in Q learning is always the maximising action over current Q values. That is an on-policy choice with respect to the target policy $\pi$, so does not need to be adjusted for.
For single-step Q learning, there is no need to use importance sampling to adjust for policy differences between behaviour and target policies. The correct update equation rule is:
$$ Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha \left( R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t,A_t) \right). $$
This does change for n-step updates, where you do need to take account of differences in action choice between target and behaviour policies for $A_{t+1}, A_{t+2}$ up to $A_{t+n-1}$. However, you should bear in mind that $\pi(a|s)$ is zero for any non-maximising action - if you use weighted importance sampling then any trajectory with an exploring action before calculating the TD update contributes zero update.
So, if you are calculating over longer trajectories, in e.g. Q($\lambda$), it is common to see this simplified to a logical test and some kind of shortcut update. You rarely see explicit use of importance sampling - with probability calculations and ratios - in Q learning.


text:
Is it a good practice to pad signal before feature extraction?
Is padding, before feature extraction with VGGish, a good practice?
Our padding technique is to find the longest signal (which is loaded .wav signal), and then, in every shorter signal, put zeros to the size of the longest one. We need to use it because one size of input data is desirable.
Perhaps there is any other techniques you recommend?
The difference between padding before and after the features extraction by accuracy is quite big - more than 20%. Using padding before extraction gives 97% accuracy.
I'd be glad to read your feedback, and explain me why that happens, and tell me if that kind of padding is correct action or is there a better solution.

Padding is a common practice both in image-processing (typically via CNNs) and in sequence-processing tasks (RNNs, Transformers).
For CNNs all the standard convolutional layers - Conv1D, Conv2D and Conv3D,- have the padding argument. The padding values can be valid or same for 2d and 3d convolutions. And extra causal type of padding is possible for 1d convolutions and the documentation refers to this paper: WaveNet: A Generative Model for Raw Audio -  which sounds quite close to what you are interested in.
This animations might be useful to get a bit more intuition about the convolutions and strides/padding. The general consensus is that using same padding is advantageous for model performance - your network gets more information about the borders of your inputs (and deeper networks are possible).
For sequential models padding is even more important. Training samples are usually  of unequal lengths, so you have to pad them with a special token (usually called [PAD] that gets encoded as 0). Here are some examples of this mentioned in tensorflow docs, huggingface.transformers or BERT tutorial.


text:
Is the final model scaling done on the full training set?
We have our training set and our test set. When we scale our data we "fit" the scaler transform to the training set and then we scale both the training set and test set using this scaler object. Using splitting and cross-validation techniques, one can use the training set as training and validation. Finally, reporting on the test set.
Now, if I want to use a model in a real-life environment, it's common to use the entire dataset (training and test) to train our already optimized model to obtain a final ready for the production model.
My question is regarding scaling. Should we fit the scaler to the entire set and then scale? Or can we simply append the scaled training set and scaled test set (both have been scaled using the training set's scaling parameters)?
I am making use of sklearn.preprocessing.PowerTransformer. Using "Yeo-Johnson's" power transform and also standardizing the data.

The short answer is yes.
When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more data leads (generally) to better models" is valid to the scaling to the same extent it's valid for the model itself.


text:
In variational autoencoders, why do people use MSE for the loss?
In VAEs, we try to maximize the ELBO = $\mathbb{E}_q [\log\ p(x|z)] + D_{KL}(q(z \mid x), p(z))$, but I see that many implement the first term as the MSE of the image and its reconstruction. Here's a paper (section 5) that seems to do that: Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse (2019) by James Lucas et al. Is this mathematically sound?

It may be a good idea to provide 1-2 examples where you saw this because the MSE is not always used. For example, here they use the cross-entropy.

I recently read that MSE loss optimization is equivalent to minimizing Pearson $\chi^{2}$ divergence. Kullback–Leibler divergence (and also cross-entropy) has its drawbacks. Here is explanations of Least Squares loss for GAN

As you mentioned, MSE is used to measure the difference between the original and generated images. This encourages the model to preserve the original content. MSE loss can be used as an additional term, which is done in CycleGAN, where the authors use LSGAN loss and cycle-consistent loss, which is MSE-like loss.

@nbro, it is not clear why they use BCE there... In fact, that implementation doesn't seem to sample between the encoder and decoder, so even more strange. Looks like they treat the distribution parameters as the input to the decoder

What do you mean by "sample between the encoder and decoder"? Yes, the input to the decoder is a sample from the latent space, so I am not sure what you mean.

@nbro, I don't see in the implementation how they sample. But my main question is how BCE is achieved on data points (instead of distribution)

@IttayD Please, provide the link to the examples of the VAE implementations that use the MSE. To give a proper answer, I would need more context.

@nbro, MSE or BCE are the same thing. The VAE paper doesn't talk about comparing the reconstructed image with the original one. Just about optimizing PDFs. See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE

@nbro, from the article: "for the decoder we used MLPs with
either Gaussian or Bernoulli output". That is, the output of the decoder are the parameters of a distribution, not a reconstructed image. We sample from it to get the image.

If $p(x|z) \sim \mathcal{N}(f(z), I)$, then
\begin{align}
\log\ p(x|z)
&\sim \log\ \exp(-(x-f(z))^2) \\
&\sim -(x-f(z))^2 \\
&= -(x-\hat{x})^2,
\end{align}
where $\hat{x}$, the reconstructed image, is just the distribution mean $f(z)$.
It also makes sense to use the distribution mean when using the decoder (vs. just when training), as it is the one with the highest pdf value. So, the decoder produces a distribution from which we take the mean as our result.

This seems correct to me (and this is what I had in mind when I wrote my answer above, although I didn't write the derivation), but note that the second $\sim$ is exactly $=$. The first one is correct because you're omitting the denominator of the Gaussian density (which becomes a constant if you optimize wrt $\theta$), so you use approximately equal, which is correct. Moreover, it might be a good idea to add a underscript to $f$, i.e. $f_\theta$ to emphasize that you will be optimizing with respect to those parameters, although you jointly optimize them with $\phi$ (the encoder parameters).

On page 5 of the VAE paper, it's clearly stated

We let $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ be a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data) whose distribution parameters are computed from $\mathbf{z}$ with a MLP (a fully-connected neural network with a single hidden layer, see appendix $\mathrm{C}$ ).
...
As explained above and in appendix $\mathrm{C}$, the decoding term $\log p_{\boldsymbol{\theta}}\left(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i, l)}\right)$ is a Bernoulli or Gaussian MLP, depending on the type of data we are modelling.

So, if you are trying to predict real numbers (in the case of images, these can be the RGB values in the range $[0, 1]$), then you can assume $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ is a Gaussian.
It turns out that maximising the Gaussian likelihood is equivalent to minimising the MSE between the prediction of the decoder and the real image. You can easily show this: just replace $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$ with the Gaussian pdf, then maximise that wrt the parameters, and you should end up with something that resembles the MSE. G. Hinton shows this in this video lesson. See also this related answer.
So, yes, minimizing the MSE is theoretically founded, provided that you're trying to predict some real number.
When the binary cross-entropy (instead of the MSE) is used (e.g. here), the assumption is that you're maximizing a Bernoulli likelihood (instead of a Gaussian) - this can also be easily shown.

the pdf is $\frac{1}{\sigma_z \sqrt{2\pi} } e^{-\frac{1}{2}\left(\frac{x-\mu_z}{\sigma_z}\right)^2}$. MSE is $|\hat{x}-x|^2$. I don't see the connection other than the fact there's an $(x-\mu_z)^2$ at the power of e. Note that in p(x|z), the original x is used, not the reconstruction.

See section C.2 in the original paper where they calculate p(x|z) with no MSE / BCE

Note sure why you say "Note that in p(x|z), the original x is used, not the reconstruction.". Both the original and the reconstructed images are used. z is used to compute the reconstructed image (or pixel).


text:
How does the Alpha Zero's move encoding work?
I am a beginner in AI. I'm trying to train a multi-agent RL algorithm to play chess. One issue that I ran into was representing the action space (legal moves/or honestly just moves in general) numerically. I looked up how Alpha Zero represented it, and they used an 8x8x73 array to encode all possible moves. I was wondering how it actually works since I got a bit confused in their explanation:

A move in chess may be described in two parts: selecting the piece to move, and then selecting among the legal moves for that piece. We represent the policy $\pi(a \mid s)$ by a $8 \times 8 \times 73$ stack of planes encoding a probability distribution over 4,672 possible moves. Each of the $8 \times 8$ positions identifies the square from which to "pick up" a piece. The first 56 planes encode possible "queen moves" for any piece: a number of squares $[1..7]$ in which the piece will be moved, along one of eight relative compass directions {N, NE, E, SE, S, SW, W, NW}. The next 8 planes encode possible knight moves for that piece. The final 9 planes encode possible under-promotions for pawn moves or captures in two possible diagonals, to knight, bishop or rook respectively. Other pawn moves or captures from the seventh rank are promoted to a queen.

How would one numerically represent the move 1. e4 or 1. NF3 (and how would the integer for 1. NF3 differ from 1. f3) for example? How do you tell what integer corresponds to which move? This is what I'm essentially asking.

Let's do the code, so all the details are down.
Encoding dictionary:
codes, i = {}, 0
for nSquares in range(1,8):
    for direction in ["N", "NE", "E", "SE", "S", "SW", "W", "NW"]:
        codes[(nSquares,direction)] = i
        i += 1

You'll see that the codes dictionary will have 56 entries in it for each (nSquares,direction) pair.
The knight moves we'll encode as the long "two"-cell edge move first and the short "one"-cell edge second:
for two in ["N","S"]:
    for one in ["E","W"]:
        codes[("knight", two, one)] , i = i , i + 1
for two in ["E","W"]:
    for one in ["N","S"]:
        codes[("knight", two, one)] , i = i , i + 1

Now we should have 64 codes. As I understand, the final 9 moves are when a pawn reaches the final rank and chosen to be underpromoted. It can reach teh final rank either by moving N, or by capturing NE, NW. Underpromotion is possible to three pieces. Writing the code:
for move in ["N","NW","NE"]:
    for promote_to in ["Rook","Knight","Bishop"]:
        codes[("underpromotion", move, promote_to)] , i = i , i + 1

We get 73 codes as described.
Policy
The distribution over actions is a (8,8,73) tensor (it is not formally a "policy", since policy should also depend on state, but lets cut this corner for this discussion):
policy = np.zeros((8,8,73))

Let's also do codes for columns for convenience:
columns = { k:v for v,k in enumerate("abcdefgh")}

How would one numerically represent the move 1. e4

The first two dimensions choose the figure you are moving. So, that'd be e2 pawn. And we move north 'N' by 2 cells.
So, we put 1 into the tensor at the appropriate indices. Note that you have to subtract 1 from the row index, to make it zero-based.
e4policy = np.zeros((8,8,73))
e4policy[ columns['e'] , 2 - 1 , codes[(2 , "N")]] = 1

How would one numerically represent the move 1. NF3

The first two dimensions choose the figure you are moving. So, that'd be g1 knight. And we perform north-west N,W knight jump.
NF3policy = np.zeros((8,8,73))
NF3policy[ columns['g'] , 1 - 1 , codes[("knight", 'N' , 'W')]] = 1

Generally, the policy is a probability distribution over all possible moves, so the policy tensor would have several non-zero probability values in it. For example an opening policy that does 1.e4 or 1.Nf3 with 50/50 probability would be:
openingPolicy = (e4policy + NF3policy) / 2

Hope this clears things up.

Thanks for such a detailed explanation! I assume in order to make this more appropriate to sample, we could flatten the 8x8x73 array into a 1d 4762 array with each integer representing it's respective move? This would make more sense in the context of training via self play, correct?

@Akshay There are quite a lot of details there. This tensor is an output of  Policy Network, that gets trained by the whole MCTS machinery. You also have to make sure that probabilities of invalid moves are zero and that all probabilities sum to one. For more detailed explanation I suggest studying the paper and asking more questions.

thanks for great explanation for encoding the output of AlphaZero, however I struggle with decoding from move probabilities to real move. Do you have idea how to make it?


text:
In variational autoencoders, what does p(x|z) mean?
If $x \sim \mathcal{N}(\mu,\,\sigma^{2})$, then it is a continuous variable, and therefore $P(x) = 0$ for any x. One can only consider things like $P(x<X)$ to get a probability greater than 0.
So what is the meaning of probabilities such as $P(x|z)$ in variational autoencoders? I can't think of $P(x|z)$ as meaning $P(x<X|z)$, if $x$ is an image, since $x<X$ don't really make sense (all images smaller than a given one?)

Whilst you're right that for any continuous distribution $P(X = x) = 0 \;; \forall x \in \mathcal{X}$ where $\mathcal{X}$ is there support of the distribution, they are not referring to probabilities here, rather they are referring to density functions (though this should really be denoted with a lower case $p$ to avoid confusion such as this).
$p(x|z)$ is a conditional distribution, which is also allowed in the continuous case -- you can also 'mix and match', i.e. $x$ could be continuous and $z$ could be discrete, and vice-versa.
In the paper, all the authors are meaning when they write $p(x|z)$ is the density of $x$ conditioned on $z$; in VAE's with an image application this is the conditional density of the image $x$ given your latent vector $z$.

In VAE's, we want to model the distribution of images $x$ with some latent variable $z$. Because $x$ is a random variable, You can think of $P(x|z)$ as the distribution of images $x$ conditioned on the random variable $z$. So given a particular value of $z$, we can generate a distribution over images $x$.
VAE's try to model images, which are themselves high dimensional 2D data. Given a 28x28 image, we already have 784 latent variables to model. We cannot visualise the distribution over all images $x$. Your notation $P(x < X|z)$ makes sense in a 1D case with a scalar value. However when considering 2D and higher, we have a problem with how we consider what is less then. if $x = (y_1,y_2)$ and $X = (y_3,y_4)$, then is $x < X$ if both $y_1 < y_3$ and $y_2 < y_4$? (I.e all dimensions have to be less than or if only one dimension needs to be less than). When talking about high dimensional space therefore, it is not very useful to denote $P(x < X|z)$ because of difficulty in interpreting the results.

I didn't think too much about notation. The point is that P(X), that is, the probability of a single point (in, maybe, a multi dimensional space), in a continuous distribution, is 0.


text:
What does "semantic gap" mean?
I was reading DT-LET: Deep transfer learning by exploring where to transfer, and it contains the following:

It should be noted direct use of labeled source domain data on a new scene of target domain would result in poor performance due to the semantic gap between the two domains, even they are representing the same objects.

Can someone please explain what the semantic gap is?

In terms of transfer learning, semantic gap means different meanings and purposes behind the same syntax between two or more domains. For example, suppose that we have a deep learning application to detect and label a sequence of actions/words $a_1, a_2, \ldots, a_n$ in a video/text as a "greeting" in a society A. However, this knowledge in Society A cannot be transferred to another society B that the same sequence of actions in that society means "criticizing"! Although the example is very abstract, it shows the semantic gap between the two domains. You can see the different meanings behind the same syntax or sequence of actions in two domains: Societies A and B. This phenomenon is called the "semantic gap".

The wiki has a concise quote by Andreas Hein, where the gap is defined by "the difference in meaning between constructs formed within different representation systems".  This connotes the core problem of translating meaning between an informal language (typically natural language) and a formal language (programing language or other formal symbolic system).
Informally, the problem could be defined as "the gap in meaning between two different contexts", and we might observe this in something as simple as gestures having different meanings in different cultures.  It would be less of a problem translating meaning between two formal systems.
At its core, "Semantic gap" seems to relate to the difficulty in formalizing certain fuzzy concepts in symbolic systems.


text:
What's the difference between a 1d tensor and a 2d tensor with 1 dimension?
I'm doing a TensorFlow tutorial, where they convert an array of the numbers [1,2,3] to a tensor like this:
const xs = tf.tensor2d([1, 2, 3], [3, 1])

The shape is [3,1] because there is one row with 3 numbers.
My question is, why would they use a 2D tensor, isn't this just exactly the same as:
const xs = tf.tensor1d([1, 2, 3])

This seems to be just a programming question/issue. Am I right? If yes, then you should ask this question on Stack Overflow, as programming questions are generally off-topic here. Please, if you have some time, take a look at our on-topic page, where we describe what kind of question you can ask here.

I posted it here because it's not really about code but about the difference in 1d or 2d tensors. Unless that is also considered code?

To give a proper answer, I think we will need more context, i.e. please provide the link to the tutorial where the first version of the code is used.

The required shape of the tensor $T$ depends on the shape of other tensors that are involved in the same operations of that same tensor $T$ and the required/desired shape of the resulting tensor, in the same way that the number of columns of the matrix $M \in \mathbb{R}^{n \times m}$ needs to match the number of rows of the matrix $M' \in \mathbb{R}^{n' \times m'}$ when you perform the matrix multiplication $M M'$, i.e. in order for $M M'$ to be well-defined, $m = n'$.
So, in your case, although the tensors contain the same elements, it might not be possible to use both in the same operations. Without more context/details, I cannot specifically answer why the first tensor was used in the tutorial you're mentioning.


text:
How to improve a trained model over time (i.e. with more predictions)?
I built a model using the tutorial on the TensorFlow site. It was a simple image classification neural network.  I trained it and saved the model and weights together on a .h5 file.
Recently, I have been reading about backpropagation. From what I understand, it's basically a way to tell the neural network whether if it's identified the correct output and that it is applied during training data only.
So, I was wondering if there is a way for the model to 'improve' over time as it makes more and more predictions. Or is that not how it would work with Neural Networks?

That is exactly a neural network works like.
Suppose you have a 1000 examples. How you train a network is: First, you divide these 1000 into maybe 100 batches (10 each). After that's done, you feed a batch to the network get its output and compare it with the ground truth, whatever is the error gets backpropagated. Then, for the next batch and then another. Once all these batches are done, you say an epoch is over. So, the number of epoch is effectively the number of times the network has seen the whole data.
This is how a neural network gets better.

Oh ok, but how it work if I would like the neural network to increase it's speed in which it can classify an a particular pre-defined image input.

If you want to reduce inference time, use a smaller model. Distill the knowledge of bigger one into smaller one. Use quantized form of the network.

Hey, but could there be a way such that inference time is reduced with practice. So the inference time is reduced over time due to it being able to process faster. or is that not the case. I am trying to link it to how humans get better with practice and such.. thx

The number of multiplications don't change. In case of humans, things get cached in basal ganglia over time because of repeated actions. If you are giving the same inputs to the network, then, you can cache just like humans do in form of habits.

Oo okay, so you suggest I should add a way of caching new inputs, and have the neural network store them? Is there a simple way to do this with the Tensorflow and Keras libraries. Thx


text:
Does adding a model complexity penalty to the loss function allow you to skip cross-validation?
It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.
Do I have this right?
It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

It's my understanding that selecting for small models, i.e. having a multi-objective function where you're optimizing for both model accuracy and simplicity, automatically takes care of the danger of overfitting the data.

Sort of. A secondary objective function often works as a form of regularisation, and can work to reduce overfit.
However, this regularisation is not a magic bullet. The degree of regularisation that you achieve will vary, it may depend on hyper-parameters in the regularisation technique. In your case the relative weightings of objective functions for the accuracy and simplicity of the model will have a sweet spot that minimises overfit without compromising too much and under-fitting instead.

It would be very convenient for my use case to be able to skip lengthy cross-validation procedures.

Typically using regularisation requires cross-validation in order to find good values for your regularisation hyperparameters. If the regularisation technique you have chosen is a good fit for your problem, then you may not have to search too much - there may be a broad set of values that work well for you.
In turn, a good choice of regularisation may mean that your model's accuracy is less sensitive to other hyperparameters of your model, so searching for an accurate model becomes a little easier.
However:

Does adding a model complexity penalty to the loss function allow you to skip cross-validation?

No. Assuming you want to find the best performing model, you still have to perform cross-validation. At best you may have to perform a little less than without the added objective because it has stabilised your model against other factors that can affect generalisation. However, you might have to perform more cross-validation, at least initially, in order to establish useful values of the new relative weighting hyperparameters you added with the secondary objectives. In addition the new simplicity objective function will likely change the best choices for other hyperparameters, such as number of free parameters, learning rate and length of training time.
If you were previously performing cross-validation after every epoch of training and picking the best model after the epoch that gave the best accuracy on the cv set, then that is often considered a different form of regularisation called early stopping. You may find you could relax this and test less often during training, because with regularisation based on complexity objectives, the training will tend towards a more stable end point and be less likely to overfit through additional training epochs. Although in my experience cross-validation during training is usually left on by default, in order to plot learning curves and ensure this stability really holds.


text:
Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?
I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing the algorithm helped me to understand the math behind the algorithm.
Unfortunately, I still have a question.
When the action space is continuous, I am using the normal distribution (same as in the PPO implementation by Spinning up). In the mentioned implementation, the logarithm of the standard deviation is used initially to give the same probability to all of the possible action, then they use the standard deviation when choosing an action. Why do we use the logarithm? Why not directly use simply the standard deviation?
I know that the logarithm is easier when it comes to the computations, but I can not see the benefits of the logarithm in the Spinning up implementation.
class MLPGaussianActor(Actor):

    def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
        super().__init__()
        log_std = -0.5 * np.ones(act_dim, dtype=np.float32)
        self.log_std = torch.nn.Parameter(torch.as_tensor(log_std))
        self.mu_net = mlp([obs_dim] + list(hidden_sizes) + [act_dim], activation)

    def _distribution(self, obs):
        mu = self.mu_net(obs)
        std = torch.exp(self.log_std)
        return Normal(mu, std)

    def _log_prob_from_distribution(self, pi, act):
        return pi.log_prob(act).sum(axis=-1)    # Last axis sum needed for Torch Normal distribution

As you have mentioned using log is nicer because it makes multiplications to additions etc etc(it helps in numerical stability issues). But I think over here, the reason they are doing it like that is because of enforcing a simple constraint in a much more simpler way. In the __init__ we are noticing that the log_std is being formulated instead of the std itself. Would it be wrong if you formulated the std itself? No. But it would be a bit messy to have some constrained imposed on it. For example, if we modeled std it could have been possible for the std to become 0 or negative. Well that's not a correct values of a std. But over here that is being enforced automatically, this kind of little things makes the learning for the model easier. Even if the log_std becomes negative or zero it does not matter because the exp will take care of it by exponentiating it to a positive number.

thank you for the answer and also mentioning that using std is also not wrong!


text:
With Monte Carlo off-policy learning what do we correct by using importance sampling?
I do not understand the link of importance sampling to Monte Carlo off-policy learning.
We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.
So, it is possible that in the target policy, we could have state values (or state action values) coming from different trajectories.
If the above is true, and if the values depend on the subsequent actions (the behavior policy), there is something wrong there, or else, better, something I do not understand.
Linking this question with importance sampling, do we use this ro value to correct this inconsistency?
Any clarification is welcome.

Recall that the definition of a value function is
$$v_\pi(s) = \mathbb{E}\left[G_t | S_t = s\right]\;.$$
That is, the expected future returns given from state $s$ at time $t$ when we follow our policy $\pi$ -- i.e. our trajectory is generated according to $\pi$.
Using Monte Carlo methods we typically will estimate our value function by looking at the empirical mean of rewards we see throughout many training episodes, i.e. we will generate many episodes, keep track of all the rewards we see from state $s$ onwards across all of our episodes (this may be the first visit method or the all visit methods) and use these to approximate the expectation that is our value function.
The key here is that to approximate the value function in this way, then the episodes must be generated according to our policy $\pi$. If we choose the actions in an episode according to some other policy $\beta$ then we cannot use these episodes to approximate the expectation directly. As an example, this would be like trying to approximate the mean of a Normal(0, 1) distribution with data drawn from a Normal(10, 1) distribution.
To account for the fact that the actions came from a different distribution, we have to reweight the returns according to an importance sampling ratio. To see why we need importance sampling, see this question/answer.

Ok. What we re-weight are the returns, so the values of each state value or state value action. Thank you @David Ireland

We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.

The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,a)$) of a state (or a state action pair). These value functions are always defined with respect to some policy $\pi(a|s)$ and is given the notation of $V^{\pi}(s)$ (or $Q^{\pi}(s,a)$).
The off-policy learning problems are arising when you have two policies: the generation policy $\mu(a|s)$ and the target policy $\pi(a|s)$. Your MC sampling data came from an agent following $\mu$, while you want to improve your target policy $\pi$. It is pretty straightforward from here that you'd need to weight your calculations with factors like $\frac{\pi(a_i|s_i)}{\mu(a_i|s_i)}$ - that's what importance sampling is.


text:
What Constitutes Messages in Junction Tree Algorithm?
I'm currently studying the Junction Tree Algorithm: I'm referring to the process of transforming a Bayesian Network into a Junction Tree in order to apply inference. I understand how you build the Junction Tree, but I'm stuck on the idea of message passing.
What exactly are these messages? Are they numbers, or vectors?
If any of you could direct me to a numerical example that would be very appreciated.

I think an example could make you understand better.
Suppose you want to calculate $P(X|S)$, you now need to put evidence on $S$(so you have to change all tables where $S$ appears making probability zero where $S$ does not appear).
At this point, you can proceed with the collect and distribute method in order to propagate the evidence throughout the graph, in order to maintain the global consistency property. Now you can choose any $cluster\space V$ that contains $X$, marginalize with respect to $S$, and the job is done.

Thanks, now I finally understand where my confusion came from!


text:
What is the difference between environment states and agent states in terms of Markov property?
I'm going through the David Silver RL course on YouTube. He talks about environment internal state $S^e_t$, and agent internal state $S^a_t$.
We know that state $s$ is Markov if
$$\mathbb{P}\{S_t=s|S_{t-1}=s_{t-1},...,S_1=s_1\}=\mathbb{P}\{S_t=s|S_{t-1}=s_{t-1}\}.$$
When we say that Decision Process is Markov Decision Process, does that mean:

All environment states must be Markov states
All agent states must be Markov states
Both (All environment states and all agent states must be Markov states)

and according to this, if we specify corresponding MDP as $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma, T)$, is $\mathcal{S}$ the state space of environment states or agent states?
Why I'm confused by this? He claims that environment states are Markov (I'm also confused why, but I'll make another post for this), and then claims that if the agent can directly see environment internal state $S^e_t$, then observations $O_t=S^e_t$, and agent constructs its state trivially as $S^a_t=O_t=S^e_t$. Now, both environment and agent states are Markov (since they are the same), so this makes sense. If we specify MDP as $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma, T)$, it's clear that state space $\mathcal{S}$ is state space of both agent internal states and environment internal states (again, since they are the same).
Now consider the case when the environment is not fully observable. Now $O_t\ne S^e_t$, and agent must construct it's state $S^a_{t}=f(S^a_{t-1}, H_t)$, where $H_t=(O_0, A_0, R_1, O_1,...,O_{t-1}, A_{t-1}, R_t, O_t)$ is history until time step $t$, and $f$ is some function (such as Recurrent Neural Network for example). In the case of $f$ being a recurrent neural network, we have that both environment internal states are Markov (by this hypothesis), and agent internal states are Markov (approximately), so again, the process is an MDP $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma, T)$, but state space of agent states is different to that of environment states, so I'm confused about what is $\mathcal{S}$ here. Is it environment state-space or agent state space?
Lastly, what if $f(S^a_t, H_t)=O_t$? That is, the agent's internal state is simply the last observation. Considering that environment states are always Markovian (again, don't know why we can claim this), but agent states are not, this is the case of POMDP. Even here I don't know what $\mathcal{S}$ stands for in specification of POMDP. Is it environment state-space or action state space?

To have more context, could you please provide the link to the specific video lesson where D. Silver talks about this topic (maybe also the specific part of the video lesson, minute, where he says that)?

$\mathcal S$ is just a set of all possible states. It doesn't matter if it's agents perceived state or true environment state, they are within the same set of states. Agent cannot perceive itself to be in some "middle" state that's not in $\mathcal S$, it might think that's in the state that's not the actual environment state but that state is also in set of all states.
To give an example, if the car can be blue or red, then the agent might think that the state is blue car or red car, but it cannot think that car is purple because that's not one of the possible states. It might wrongly think that the car is blue when the actual car is red, but that's ok because blue is one of the possible car states. Of course, it might also correctly think that the car state is red.

Ok, so what you are saying is that all states that an agent can be in are embedded in the environment state itself. For example, if the agent state is a hidden state of RNN, then this internal agent state vector is part of the environment state (as the agent is part of the environment as well)? If this is the case, then $\mathcal{S}$ is set of all environment states (as environment state space contains agent state space in one of it's subspaces)? After reading your answer, this is my conclusion. Did I get it right?

@StankoKovacevic whenever we talk of MDP or POMDP, we are always concerned about the agent's internal state, not the environment's. I don't know what the answer above means when it talking of S, but it is possible for the set of possible environment state and the set of possible agent state to be completely disjoint. Consider the case of an Atari game. Then the agent's state can be the screen while the environment state is the memory (which includes the code itself) in the emulator. So we are always talking about the agent's state. It's quite late but hope it helps.

@StankoKovacevic As to the reason why all environments are said to be Markov, I am not entirely certain, but I can guess. For any agent that is run on a PC entirely (like Atari), we can easily say that the environment state is the entire memory and it becomes Markov. In other cases, like a Robot trying to learn how to walk, the environments state is the entire universe, including the robot of course. Classically, if we know the mass, location, velocity and acceleration of every particle, then we can predict what will happen at every future instant - again it becomes an MDP.

I don't know enough about quantum mechanics to say anything about it.

@QuantumSphinx This answers my question perfectly.

@StankoKovacevic If this answers your question perfectly, you should accept it.

@StankoKovacevic You may also consider upvoting this answer if you found it useful and you think it's orrect.

When you say "Agent cannot perceive itself to be in some "middle" state that's not $S$". Consider, for instance, DQN. In that case, we're approximating states with sequences of observations. So, those approximate states may actually be not really states in $S$. Right? So, you may want to edit this answer to point this out, i.e. in practice, the agent may perceive some state that is not in $S$. It depends on how you define $S$ too, though.


text:
Is it possible to optimize a multi-variable function with a reinforcement learning method?
I want to use RL instead of genetic or any other evolutionary algorithm in order to find the best parameter for a function.
Here is the problem:
Given a function $$f(x,y,z, \text{data}),$$
where $x$, $y$ and $z$ are some integers from 1 to 50.
So I can say I have a 3-dimensional array which is a way to save fitness values:
$$\text{parameters} = [[1..50], [1..50], [1..50]]$$
The $$\text{data}$$ is another input which is the $f$ needed to do some calculation on.
Currently, I am optimizing it using a genetic algorithm with $$\text{cost}(\text{fitness}) = f(x,y,z,data)$$ which is a customized cost function.
Any value for $x$, $y$, and $z$ will result in a cost for example:
$$f(1, 5, 8, X) = 15$$
$$\text{parameters}: [1, 5, 8] = 15$$
or
$$ \text{parameters}: [2, 9, 11] = 30$$
In the provided example 2, 9, and 11 is a better set of parameters.
So I run a genetic algorithm and make some children with a sequence of x,y, and z. Then I calculate the cost(fitness) and then select them and so on.
I want to know is there any alternative or method in reinforcement learning which I can use instead of a genetic algorithm? If yes, please provide the name or any helpful link.
Note that F is completely defined by the user and should be changed in other contexts.

Reinforcement learning is not an optimization method, it's a data driven approach to optimal control. There is no notion of state or action in your problem nor there is sequential decision making involved. It's a wrong tool to use here.

@Brale yeah I know it's not right, but I want to know if there any way to solve this problem in a some customized approach.

if you insist on using RL, maybe adaptive operator selection of GA (or population metaheuristic in general) using RL?

In order to have anything resembling reinforcement learning you must at the very least have a set of states $S$ and a set of actions $A$.
In your formulation I can vaguely identify the set of states $S$ as all possible $(x,y,z)$ triplets. But don't see anything in your description that could be interpreted as a set of actions $A$. You either oversimplified the description of your problem or reinforcement learning is not applicable here by lack of very basic ingredients for it.


text:
What would be the state of the art image captioning deep learning model?
I saw a couple of architectures, like CNN-LSTM, with and without attention model, use of Glove vector, self-critical models, etc. I am overwhelmed looking at different notebooks and architectures, came here for a guidance.  I am looking to build a personal project on image annotations. Also, if I wanted to use this deep learning model together with TFX pipeline, what would be the best type of architecture I can go with?

Here are a couple of Kaggle Kernels, Notebooks and Tutorials for Image Captioning.

Kaggle Kernel | Neural Image Captioning:  ->

Kaggle Kernel | Show Attend and Tell

Kaggle Kernel | Flickr Image Captioning : TPU, TF2 & Glove

Tensorflow Tutorial | Image captioning with visual attention

Show and Tell: A Neural Image Caption Generator by Vinyals et al.


text:
In MCTS, what to do if I do not want to simulate till the end of the game?
I'm trying to implement MCTS with UCT for a board game and I'm kinda stuck. The state space is quite large (3e15), and I'd like to compute a good move in less than 2 seconds. I already have MCTS implemented in Java from here, and I noticed that it takes a long time to actually reach a terminal node in the simulation phase.
So, would it be possible to simulate games up until a specific depth?
Instead of returning the winner of the game after running until the max depth, I could return an evaluation of the board (the board game is simple enough to write an evaluation function), which then back propagates.
The issue I'm having is in handling the backpropagation. I'm not quite sure what to do here. Any help/resources/guidance is appreciated!

Famous example is AlphaZero. It doesn't do unrolls, but consults the value network for leaf node evaluation. The paper has the details on how the update is performed afterwards:

The leaf $s'$ position is expanded and evaluated only once by the network to gene-rate both prior probabilities and evaluation, $(P(s′ , \cdot),V(s ′ )) = f_\theta(s′ )$. Each edge $(s, a)$ traversed in the simulation is updated to increment its visit count $N(s, a)$, and to update its action value to the mean evaluation over these simulations, $Q(s,a) = \frac{1}{N(s,a)}\sum_{s,a\to s'}V(s')$  , where $s, a\to s′$ indicates that a simulation eventually reached s′ after taking move a from position s.


text:
How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?
Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:
$$Q_{target} = (1-\tau) \times Q_{target} + \tau \times Q_{prime}$$
I saved the primary Q network's weights every $n$ episodes (say $n=10$), but, unfortunately, I did not save the target Q network's weights.
Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?

Let's add a step index to your expression
$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$
We can expand it one step further
$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$
And further
$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$
So, I guess, we can write a general formula for $m$ steps behind like:
$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary} $$
For $n-m$ large enough $(1-\tau)^{n-m}$ should be close to 0 and you should be able to approximately reconstruct your $Q_{target}^n$ using only the history of $Q_{primary}$es
Edit: I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.

Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}{primary}, Q^{20}{primary}, ...$)? Nevertheless, your answer has very useful hints.

@Sanyou Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer.

I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation!

Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more.

@NeilSlater I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution.

how bad is it if the approximated target is too far off from the real target? because I have used @Kostya solution with snapshot linear interpolation and the average rewards from the agent plunges and still not catching up with the previous improvement rate after 2 days of running