text:
Will a neural network always predict the correct label if it sees the exact same input during training and testing?
If I'm performing a text classification task using a model built in Keras, and, for example, I am attempting to predict the appropriate tag for a given Stack Overflow question:

How do I subtract 1 from an integer?

And the ground-truth tag for this question is:

objective-c

But my model is predicting:

c#

If I were to retrain my model, but this time add the above question and tag in both the training and testing data, would the model be guaranteed to predict the correct tag for this question in the test data?
I suppose the tl;dr is: Are neural networks deterministic if they encounter identical data during training and testing?
I'm aware it's not a good idea to use the same data in both training and testing, but I'm interested from a hypothetical perspective, and for gaining more insight into how neural networks actually learn. My intuition for this question is "no", but I'd really be interested in being pointed to some relevant literature that expands/explains that intuition.

No, Neural Networks do not have such a guarantee. In fact, I don't believe any kind of classifier in the entire field of Machine Learning has such a guarantee, though some may be slipping my mind...
For an easy counterexample, consider what happens if you have two instances with precisely identical inputs, but different output labels. If your classifier is deterministic (in the sense that there is no stochasticity in the procedure going from input to output after training), which a Neural Network is (unless, for example, you mess up a Dropout implementation and accidentally also apply dropout after training), it cannot possibly generate the correct output for both of those instances, even if they were presented as examples thousands of times during training.
Of course the above is an extreme example, but similar intuition applies to more realistic cases. There can be cases where getting the correct prediction on one instance would reduce the quality of predictions on many other instances if they have somewhat similar inputs. Normally, the training procedure would then prefer getting better predictions on the larger number of instances, and settle for failure on another instance.

I would be tempted to edit this answer to just highlight the 'equal inputs with opposite labels' section, since that is a very clear and elegant verbal proof.

In principle, a rule-based or exact classifier could always be right. For instance, if you store the images that you've seen during training or testing/validation in a database, then, when a new image arrives, compare it with all other images in the database, you can simply return the correct label (if such an image exists in the dataset). So, I suppose that you're excluding this type of "classifier" when you say "ML classifier". In any case, the key idea is: most ML models typically approximate the desired function, so there's a chance that they may be wrong.

After training, all standard models are deterministic (the process each input goes thru is set).
In essence, during training the model attempts to learn the distribution of the training dataset. Whether it is able to depends on the size of the model, if it is big enough, it can simply "memorize" all the training samples and result in perfect accuracy on the training set.
Normally this is considered to be terrible (called overfitting) and many regularization techniques attempt to prevent it. Eventually when training a model, you are giving it the training distribution as an example but you hope that it will be able to estimate the real distribution out of it.


text:
What limitations does the Markov property place on real time learning?
The Markov property is the dependence of a system's future state probability distribution solely on the present state, excluding any dependence on past system history.
The presence of the Markov property saves computing resource requirements in terms of memory and processing in AI implementations, since no indexing, retrieval, or calculations involving past states is required.
However, the Markov property is often an unrealistic and too strong assumption.
Precisely, what limitations does the Markov property place on real-time learning?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why is the actor-critic algorithm limited to using on-policy data?
Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?

It's because, in the actor-critic algorithm, the objective function is an expectation under the $\tau$ of the policy. If we want to use off-policy data, we have to resort to importance sampling relative to the other policy.

What does the tau value mean?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is the meaning of Model(s, a) in the prioritized sweeping algorithm?
I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).
The authors provide the pseudocode of the prioritized sweeping algorithm, but I do not know what is the meaning of Model(s, a). Does it mean that Model(s, a) is the history of rewards gained when we are in state s and the action a is taken?
Does R, S_new = Model(s,a) mean that we should take a random sample from rewards gained in state s and action a is taken?

I think pseudocode was made for tabular case with an assumption of deterministic environment. $Model(s, a)$ would then be a table with information of the next state and reward after taking action $a$ from state $s$. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.

Thanks for your response. Do you mean that if we are in state s and take action a, we will be always in the state S_new in the next state? It seems somehow weird!

Yes, it's weird because it was made with an assumption of deterministic environment which is quite optimistic and unreasonable. I never tried it myself but i believe if you have stochastic environment, for tabular case, you could remember all the different transitions that happened and also have a count of how many times those transitions happened, then you could calculate probabilities of those transitions happening from those counts and do planning with that. Maybe you could do sampling like you suggested but based on those probabilities or maybe some kind of dynamic programming updates.

Thanks. Have you ever seen a paper that used the stochastic version? In the deterministic version, does the model use the last visit of next state and reward?

As for the paper, google search has led me to this . I haven't read the paper but it looks like exactly what you are looking for. As for the deterministic environment, yes, model simply uses last observed next state and reward.

Dear @Brale
I really appreciate you.


text:
Is Monte Carlo Tree Search appropriate for problems with large state and action spaces?
I'm doing a research on a finite-horizon Markov decision process with $t=1, \dots, 40$ periods. In every time step $t$, the (only) agent has to chose an action $a(t) \in A(t)$, while the agent is in state $s(t) \in S(t)$. The chosen action $a(t)$ in state $s(t)$ affects the transition to the following state $s(t+1)$.
In my case, the following holds true: $A(t)=A$ and $S(t)=S$, while the size of $A$ is $6 000 000$ (6 million) and the size of $S$ is $10^8$. Furthermore, the transition function is stochastic.
Would Monte Carlo Tree Search (MCTS) an appropriate method for my problem (in particular due to the large size of $A$ and $S$ and the stochastic transition function?)
I have already read a lot of papers about MCTS (e.g. progressive widening and double progressive widening, which sound quite promising), but maybe someone can tell me about his experiences applying MCTS to similar problems or about appropriate methods for this problem (with large state/action space and a stochastic transition function).

What do you intend to use MCTS for exactly?

I would like to apply MCTS in the context of a stochastic optimization problem, which is why only 1 player exists in my problem.

But which specific part of your problem you would like to solve with MCTS?

Do similar actions yield similar results? I.E. if I choose between two actions that appear similar, will they have very similar value in the long run? If so, it may be appropriate to embed the actions in some continuous space in the agent's policy, then perform an MCTS on the k-nearest neighbors (which are valid actions, of course) to the desired continuous proto-action. This is similar to this paper from DeepMind: https://arxiv.org/abs/1512.07679

i use MCTS for finding max action all the times, at least the action is somehow near the optimal action. Brute-force random action is crazy for problems with large possibility space

I'm not sure if there's a way to make MCTS work for this problem. But I would like to remark that $10^8$ states can probably be exhausted in a reasonable amount of time. And that your branching factor is quite close to the size of the state space, leading to short path on average. So there might be something to exploit there.

MCTS is often said to be a good choice for problems with large branching factors... but the context where that sentiment comes from is that it originally became popular for playing the game of Go, as an alternative to older game-playing approaches such as alpha-beta pruning. The branching factor of Go is more like 250-300 though, which is often viewed as a large branching factor for board games. It's not such an impressive branching factor anymore when compared to your branching factor of $6,000,000$...
I don't see MCTS working well out of the box when you have 6 million choices at every step. Maybe it could do well if you have an extremely efficient implementation of your MDP (e.g. if you can simulate millions of roll-outs per second), and if you have a large amount of "thinking time" (probably in the order of hours or days) available.

To have any chance of doing better with such a massive branching factor, you really need generalization across actions. Are your 6 million actions really all entirely different actions? Or are many of them somehow related to each other? If you gather some experience (a simulation in MCTS, or just a trajectory with Reinforcement Learning approaches), can you generalize the outcome to other actions for which you did not yet collect experience?
If there is some way of treating different actions as being "similar" (in a given state), you can use a single observation to update statistics for multiple different actions at once. The most obvious way would be if you can define meaningful features for actions (or state-action pairs). Standard Reinforcement Learning approaches (with function approximation, maybe linear or maybe Deep Neural Networks) can then relatively "easily" generalize in a meaningful way across lots of actions. They can also be combined with MCTS in various ways (see for example AlphaGo Zero / Alpha Zero).
Even with all that, a branching factor of 6 million still remains massive... but generalization across actions is probably your best bet (which may be done inside MCTS, but really does need a significant number of bells and whistles on top of the standard approach).

The question is whether MCTS is an appropriate method given these conditions.

Action $a_t \in A_t$
State $s_t \in S_t$
$[a_t, s_t] \Rightarrow s_{t+1}$
$t_i \land i \in [1, 40]$
$A(t) \in \text{set} \, A \; \land \; \text{size} (A) = 6 x 10^6$
$S(t) \in \text{set} \, S \; \land \; \text{size} (S) = 1 x 10^8$
Transition function is stochastic

MCTS focuses on the most promising sub-trees and is therefore asymmetric and well suited for some systems with a high branching factor, but the set sizes can be prohibitive unless some symmetries are exploited.
The original MCTS algorithm does not converge quickly.  This is why alpha–beta pruning and other search strategies that focus on the minimization of the search space are often used, but they have limitations with regard to branching breadth too. Pruning heuristics can help with reducing the combinatoric explosion but may lead to the most advantageous action options being missed.
Some variants of MCTS, such as those among the articles below and in the case of AlphaZero used to learn Go from only the game rules, show excellence in search speed, but perhaps not to the degree needed in this case without parallelism in the form of a computing cluster or significant hardware acceleration.
An excellent characteristic of MCTS is that highly promising actions found early may be selected without exhausting all the sub-tree evaluations, but again, that may not be enough.
A possibly constraining consideration is whether the Marcov property will be upheld, how, and whether it should be.  Another consideration is whether the system involves an opposing intelligent participant.  All sampling or pruning search strategies involve the risk of not reliably identifying a single branch in a sub-tree that leads readily to an irreversible (or difficult to reverse) loss.
These are some excellent papers that discuss these considerations and provide a high level of experience in the form of research results. The computational challenge of tree searches with high branching is covered by the articles discussing approximate solvers, an area of high research intensity.

An online POMDP solver for uncertainty planning in dynamic environment, H Kurniawati, V Yadav, 2016
An online and approximate solver for pomdps with continuous action space, KM Seiler, H Kurniawati, 2015
Hindsight is Only 50/50: Unsuitability of MDP based Approximate POMDP Solvers for Multi-resolution Information Gathering, S Arora, S Choudhury, S Scherer, 2018
α-min: A Compact Approximate Solver For Finite-Horizon POMDPs, Y Dujardin, T Dietterich, I Chades, 2015
Reinforcement Learning as a Framework for Ethical Decision Making, D Abel, J MacGlashan, ML Littman, 2016
On the Complexity of Iterative Tropical Computation with Applications to Markov Decision Processes, N Balaji, S Kiefer, P Novotný, GA Pérez, 2018
Optimizing over a Restricted Policy Class in Markov Decision Processes, E Banijamali, Y Abbasi-Yadkori, 2018
Probabilistic inference techniques for scalable multiagent decision making, A Kumar, S Zilberstein, M Toussaint, 2015

Thank you for the detailed answer and the suggestions.
But does a stochastic transition function really imply the need for a POMDP? In my case every state is fully observable (nevertheless the transitions are stochastic) - so I think a MDP is adequate.
Are there any suggestions for coping with stochastic transition functions (in the context of MDP & MCTS)? From what I've read, the stochastic transition is taken into account by the generative model, but is that really all? I doubt that in my case, because every state-action-pair has (due to stochastic) 90 potential subsequent states


text:
Why can the learning rate make the loss increase in stochastic gradient descent?
In Deep Learning by Goodfellow et al., I came across the following line on the chapter on Stochastic Gradient Descent (pg. 287):

The main question is how to set $\epsilon_0$. If it is too large, the
learning curve will show violent oscillations, with the cost function
often increasing significantly.

I'm slightly confused why the loss function would increase at all. My understanding of gradient descent is that given parameters $\theta$ and a loss function $\ell (\vec{\theta})$, the gradient update is performed as follows:
$$\vec{\theta}_{t+1} = \vec{\theta}_{t} - \epsilon \nabla_{\vec{\theta}}\ell (\vec{\theta})$$
The loss function is guaranteed to monotonically decrease because the parameters are updated in the negative direction of the gradient. I would assume the same holds for SGD, but clearly it doesn't. With a high learning rate $\epsilon$, how would the loss function increase in its value? Is my interpretation incorrect, or does SGD have different theoretical guarantees than vanilla gradient descent?

This is the case as the loss doesn't have to monotonically decrease when it's updated in the negative direction.
For example:
Let $L(\theta) = \theta^2 $  and $\theta_0= 3$
Let the subscript n in $\theta_n$ denote the iteration number.
Then $\nabla_{\theta}L(\theta_0) = 2*\theta = 2*3 = 6$
For the loss to decrease in this case $\epsilon < 1$ needs to hold so lets look at when it doesn't hold.
If $\epsilon = 1$ then the loss would be stable at 6 as the update would give $\theta_1 = \theta_0 - 1*\nabla_{\theta}L(\theta_0) = 3-6 = -3 $
$\theta_2 = \theta_1 - 1*\nabla_{\theta}L(\theta_1)  = -3 -- 6 = 3$
$\theta_3 = \theta_2 - 1*\nabla_{\theta}L(\theta_2) = 3 - 6 = -3$
$L(\theta_1) = L(\theta_2) =L(\theta_3) = 6$
As $L(-3) = (-3)^2 = 6$ and $L(3) = 3^2 = 6$
So if $\epsilon = 1$ then $\theta$ will oscillate back and forth between 3 and -3 what gives a loss that is stable at 6.
Now if we look at what happens if $\epsilon >1$ for example $\epsilon =2$
note that $\nabla_{\theta}L(\theta) = 2*\theta$ as $L(\theta) = \theta^2$
$\theta_1 = \theta_0 - 2*\nabla_{\theta}L(\theta_0) =  3 -2*\nabla_{\theta}L(\theta_0)  = 3 - 2 * (2*3) = 3 - 12 = -9$
$\theta_2 = \theta_1 - 2*\nabla_{\theta}L(\theta_1)  = -9 - 2 *\nabla_{\theta}L(\theta_1) = -9-2*(2*-9) = -9 + 36 = 27$
$\theta_3 = \theta_2 - 2*\nabla_{\theta}L(\theta_2)  = 27 - 2 *\nabla_{\theta}L(\theta_2) = 27 -2 * (2*27) = 27 -108 = -81 $
In  turn the loss would  increase after each update .
$L(\theta_1) = (-9)^2 = 81$
$L(\theta_2) = 27^2 = 729$
$(\theta_3) = (-81)^2 = 6561$

The explanations in this answer seem correct to me. However, the learning rate is usually a number in the range $(0, 1)$ (see this answer), so maybe, in that excerpt, the authors were already assuming that the learning rate is not bigger or equal to 1. So, you may want to comment on that case too.

The cited text says when it's too large, so commonly used learning rates don't apply, but I get what you mean. The actual chosen value of epsilon fully depends on the behavior of the loss landscape.


text:
If expert systems are a bunch of if-then-else statements, then how are they termed as AI?
An artificial intelligence (AI) is often defined as something that can learn over time and can imitate human behaviors.
If an Expert system (e.g. MYCIN) that only involves if-then-else statements qualifies to be an AI, then every program we write in our daily lives that involves some condition-based question answering should be an AI. Right? If not, then what should be an exact and universal definition for AI. How can a software qualify to be called AI?

Here is a related question.

TL;DR: What makes AI is not if-then statements, but rather the automated reasoning that went into selecting those particular if-then statements.
You're focusing on the structure of the output rather than how the output was produced. Having if-then control flow statements is not sufficient to make a program "AI". AI aims to enable machines to solve problems which currently people are better at. Machine learning, a subset of AI, extracts useful patterns from data. A commonly cited example of expert systems is diagnostic medicine.
This begs a lot of questions like, "What does 'useful' mean?" If the problem we were addressing was finding cancer in medical images, "useful" might mean, "able to accurately identify cancer in images at or above the rate of a skilled human examiner." There's also questions about the amount of data needed, the quality of the data, etc. These are outside the scope of your question.
There are various AI/ML systems that produce models consisting of if-then states. Decision trees like C4.5 build a hierarchy of if-thens (and random forests combine many decision trees). Learning classifier systems (both Michigan and Pittsburgh varieties) come out of genetic algorithms and form similar collections of logic.

"AI extracts useful patterns from data.", this seems to be more a definition of machine learning. Do not confuse AI with machine learning. They are not the same thing. This answer doesn't really address why expert systems are AI, but you give a general definition of what AI may be or may include as part of the field. I would like to see an answer from someone that actually has worked with (traditional) expert systems (although I didn't actually ask the question). In any case, I think this answer can be improved by explicitly saying why ESs are AI.


text:
How do I create chatbots without machine learning?
I am looking for an non-ML method for two chat bots to communicate to each other about a specific topic. I am looking for an "explainable AI" method, as opposed to a "black-box" one (like a neural network).

The easiest non-ML way would be to use a finite state machine. You could model various states of your conversation topics, and certain utterances of your bots could advance the bot's internal model along different paths. The complexity depends on the complexity of the topic.
You can then enhance the transitions with probabilities, and later move on towards ML by transforming it into an HMM.
However, even simple topics will probably lead to fairly complex state machines. But you should be able to keep track of what is going on in your conversation nevertheless.
Update: just to make it a bit clearer, I was thinking along the lines of having states for particular stages in the conversation. You could either have one model for the whole conversation, or one per participant.
Initially, there would be a state 'greeting'. Possible transitions would be to a further state 'greeting' (the response of the person who has been greeted), or that could be skipped to states such as 'statement', 'question', etc. 'Question' would have transitions to 'answer', 'ignore question', 'counter/clarification question' etc. The level of detail depends on your application.

Thank you! Do you mean similar to dialogflow but without the ML part? Should I code the dialogue with python or do you know any libraries?

It doesn't really matter, any language will do.


text:
What are the strengths of the Hierarchical Temporal Memory model compared to competing models?
What are the strengths of the Hierarchical Temporal Memory model compared to competing models such as 'traditional' Neural Networks as used in deep learning? And for those strengths are there other available models that aren't as bogged down by patents?

IMO, the greatest "strength" of HTM is that it is modeled after the human neocortex, which is the most intelligent thing we know of.
But to understand the importance of this simple idea one must contrast it with the most familiar form of AI - Neural Networks (NNs).
Traditional Neural Network AI has been under development for a long time and has many more people working on it than HTM. NNs are capable of performing a bewildering number of tasks, and the list of its accomplishments grows with every passing day.
However, NNs are not thinking. They perform their magic only after being trained on (typically) massive amounts of training data. Training a NN is essentially an advanced form of curve-fitting. If your training data encompasses closely enough what it encounters in new data then it will likely perform very well. However, if it encounters something new (which is sometimes difficult to know beforehand) then it can fail abysmally, and often in a way that humans would never fail.
One example I heard about was on a NN trained on millions of images that could briefly describe what was in new images it had never seen before. It performed fabulously - something like 95-97% accuracy. However, when it was shown an image of a baby holding a toothbrush, it said, "A boy holding a baseball bat." This is not a human-like error. Humans know the difference between a boy and a baby, and a bat and a toothbrush. This is just an example, but it reveals a fundamental problem of NNs - they are not thinking. Useful? Yes. Thinking? No.
Back to HTM. HTM is new and currently has only a handful of researchers working on it. It is "better" than NNs in only a small number of cases - it has a long way to go.
So if by "strengths" you're thinking about what tasks can currently be done better with HTM than with NNs, then most people should still chose NNs.
However, if by "strengths" you're thinking about what has the best chance of achieving general intelligence someday, then I would say hands-down it is HTM.

HTM is a credible theory about how the brain works, and how brain-like systems could be constructed in software. It includes:

SDR (Sparse Distributed Representation), a means for representing just about any kind of sensory, intermediate or motor data, innately noise resistant and suited to recognising patterns
TM (Temporal Memory), which can recognise SDRs in the context of other preceding SDRs, to learn new patterns "on the job" with no separate training phase
SM (Sequence Memory), which can learn, remember and replay arbitrarily long sequences of SDRs.

ANNs are mature, commercially valuable pattern recognisers made possible by the confluence of vast amounts of computing power, data and commercial opportunity.
HTM is immature and just a fascinating toy, for now. But HTM might just put us on the path to the Holy Grail, true artificial general intelligence, and ANNs will never do that.

Hi. I've never heard of "sequence memory" in the context of TM. Are you sure you're using the appropriate term here? Can you cite a paper that describes the sequence memory?

@nbro: I'm glad you asked: https://numenta.com/neuroscience-research/research-publications/papers/continuous-online-sequence-learning-with-an-unsupervised-neural-network-model/. ;-)

But isn't sequence memory the same thing as temporal memory? Maybe you meant spatial pooler as opposed to one of these two?

@nbro: Related but distinct is my take. But it is hard to find reliable, consistent, authoritative definitions for these terms.


text:
Why does PCA work well while the total variance retained is small?
I'm learning machine learning by looking through other people's kernel on Kaggle, specifically this Mushroom Classification kernel.
The author first applied PCA to the transformed indicator matrix. He only used 2 principal components for visualization later. Then I checked how much variance it has maintained, and found out that only 16% variance is maintained.
in [18]: pca.explained_variance_ratio_.cumsum()
out[18]: array([0.09412961, 0.16600686])

But the test result with 90% accuracy suggests it works well.
If variance stands for information, then how can the ML model work well when so much information is lost?

Because it selects both Xtrain and Xtest from the space of two selected principal components. Hence, the 90% accuracy is in that 2-D selected space.
This fact that the ratio in PCA stands the information, depends on the distribution of the data and it's not true at all.


text:
How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?
Consider the following loss function
$$
L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2
$$
where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are represented as neural networks, where $w^-$ and $w$ are the corresponding weights.
But how do you calculate $max_{a'} Q(s', a', \mathbf{w^-})$? Do you really need to hold always an older version of the network? If yes, why and how old should it be?

You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.
The details depend a little on your neural network architecture:

If you have a network that takes the state vector as input and outputs all possible action values $(\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2) ...)$, then you can run it once forward with the next_state as input to getthe array $(\hat{q}(s',a_0), \hat{q}(s',a_1), \hat{q}(s',a_2) ...)$, and take the maximum element (you don't need to care which action $a'$ caused it). You will then have the problem that you now have a loss for $Q(s, a, \mathbf{w})$ for the single action $a$ just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate $Q(s, a, \mathbf{w})$ then substitute in this new estimated value against the action $a$ and train using this modified vector.
If you have a network that takes the state vector and action combined as input and outputs a single estimate $\hat{q}(s,a)$, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination.

Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.

do you really need to hold always an older version of the network. If yes why and how old should it be?

You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.
How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.
One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. $\mathbf{w}^{\_} = (1 - \beta)\mathbf{w}^{\_} + \beta \mathbf{w}$ where $\beta$ might be $0.001$
In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.

I'd describe the need for target networks being due to the "moving target" problem that you otherwise get, rather than numeric stability. Numeric stability can easily be confused with problems/inaccuracies related to floating point arithmetic, which is not really the problem addressed by target networks (although I suppose in some cases the moving target instability may get so severe that reaching the realms of NaN becomes a symptom... but that's not the only possible symptom I think?)

So what wrong with the Brain.class (I did exactly this except experience replay)
https://github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/rlgame/Brain.java

@TVSuchty: Sorry I am not going to try and debug your project. I don't work in Java so it would take me too long and I may miss any obvious problem.

Could you help me if i translate it into pseudocode? or python?

@TVSuchty: Sorry no. If you can write a question about some part of the code that you are having trouble with, then I or someone else here might be able to help you with an answer. One thing you could do is write out a brief high-level pseudo-code for what your Brain class is doing, and ask if that is correct in a question, or if you have implemented it and it is not working, ask about the problems you are having. There are already many examples of working DQN in Python available though if you search . . .


text:
How do I combine two electromagnetic readings to predict the position of a sensor?
I have an electromagnetic sensor and electromagnetic field emitter.
The sensor will read power from the emitter. I want to predict the position of the sensor using the reading.
Let me simplify the problem, suppose the sensor and the emitter are in 1 dimension world where there are only position X (not X,Y,Z) and the emitter emits power as a function of distance squared.
From the painted image below, you will see that the emitter is drawn as a circle and the sensor is drawn as a cross.

E.g. if the sensor is 5 meter away from the emitter, the reading you get on the sensor will be 5^2 = 25. So the correct position will be either 0 or 10, because the emitter is at position 5.
So, with one emitter, I cannot know the exact position of the sensor. I only know that there are 50% chance it's at 0, and 50% chance it's at 10.
So if I have two emitters like the following image:

I will get two readings. And I can know exactly where the sensor is. If the reading is 25 and 16, I know the sensor is at 10.
So from this fact, I want to use 2 emitters to locate the sensor.
Now that I've explained you the situation, my problems are like this:

The emitter has a more complicated function of the distance. It's
not just distance squared. And it also have noise. so I'm trying to
model it using machine learning.
Some of the areas, the emitter don't work so well. E.g. if you are
between 3 to 4 meters away, the emitter will always give you a fixed
reading of 9 instead of going from 9 to 16.
When I train the machine learning model with 2 inputs, the
prediction is very accurate. E.g. if the input is 25,36 and the
output will be position 0. But it means that after training, I
cannot move the emitters at all. If I move one of the emitters to be
further apart, the prediction will be broken immediately because the
reading will be something like 25,49 when the right emitter moves to
the right 1 meter. And the prediction can be anything because the
model has not seen this input pair before. And I cannot afford to
train the model on all possible distance of the 2 emitters.
The emitters can be slightly not identical. The difference will
be on the scale. E.g. one of the emitters can be giving 10% bigger
reading. But you can ignore this problem for now.

My question is How do I make the model work when the emitters are allowed to move? Give me some ideas.
Some of my ideas:

I think that I have to figure out the position of both
emitters relative to each other dynamically. But after knowing the
position of both emitters, how do I tell that to the model?
I have tried training each emitter separately instead of pairing
them as input. But that means there are many positions that cause
conflict like when you get reading=25, the model will predict the
average of 0 and 10 because both are valid position of reading=25.
You might suggest training to predict distance instead of position,
that's possible if there is no problem number 2. But because
there is problem number 2, the prediction between 3 to 4 meters away
will be wrong. The model will get input as 9, and the output will be
the average distance 3.5 meters or somewhere between 3 to 4 meters.
Use the model to predict position
probability density function instead of predicting the position.
E.g. when the reading is 9, the model should predict a uniform
density function from 3 to 4 meters. And then you can combine the 2
density functions from the 2 readings somehow. But I think it's not
going to be that accurate compared to modeling 2 emitters together
because the density function can be quite complicated. We cannot
assume normal distribution or even uniform distribution.
Use some kind of optimizer to predict the position separately for each
emitter based on the assumption that both predictions must be the same. If
the predictions are not the same, the optimizer must try to move the
predictions so that they are exactly at the same point. Maybe reinforcement
learning where the actions are "move left", "move right", etc.

I told you my ideas so that it might evoke some ideas in you. Because this is already my best but it's not solving the issue elegantly yet.
So ideally, I would want the end-to-end model that are fed 2 readings, and give me position even when the emitters are moved. How would I go about that?
PS. The emitters are only allowed to move before usage. During usage or prediction, the model can assume that the emitter will not be moved anymore. This allows you to have time to run emitters position calibration algorithm before usage. Maybe this will be a helpful thing for you to know.

This is exactly the sort of thing that Kalman filters are used for.

Position Detection
In a traditional data acquisition and control scenario, with some assumptions, the relation between sensors signals $s_i$, emitters drive $\epsilon_j$, distances $x_{ij}$, and calibration factors is modelled as follows.
$$ \forall \, (i, j) \text{,} \quad \frac {s_i} {v_i} = \frac {\epsilon_j} {v_j \, x_{ij}^2} $$
The assumptions include these.

Linear acquisition of magnetic flux signal strength
Linear control of magnetic flux signal strength
Independent readings either by sequential reading or by use of two distinct emitter frequencies
Dismissal of relativistic phenomena
Single point emission
Single point detection

It is correct that, with only a single emitter, position of the sensor cannot be accurately determined because the direction from which the signal originates cannot be disambiguated. Two emitters are necessary for reliability. In two dimensional space, three are necessary, thus the term triangulation. In three dimensional space, four are necessary.
Less Known Function with Motion and Noise
The more complicated function of distance was not specified, whether the sample rate is tightly controlled is not indicated, and the nature and magnitude of the noise relative to the signal was not provided. It also appears there is a low digital accuracy in the readings.
To model these contingencies, for motion, $j$ shall be the sample index, and $i$ remains the detector number. The data acquisition vector is now a tuple of the reading $r_{ij}$ and time $t_j$. The function $f$ may differ from the inverse squared function due to flux conduction curvature, non-point emission and detection, and other secondary effects. The combination of this function and noise $n$, a function of sample time, $s_j$, is made discrete, according to the question, rounding or truncating to the nearest integer (indicated by $\mathcal{I}$).
$$ \forall \, (i, j) \text{,} \quad (r_{ij}, \; t_j) = \bigg(\mathcal{I}\big(f(\epsilon_j, x_{ij}) + n(s_j)\big), \; s_j\bigg) $$
There are other benefits to the additional emitter than disambiguation of direction. The impact of noise is reduced as redundancy is added, and there is the calibration issue.
Calibration
High volume, low cost, electronic parts are not usually calibrated in the factory. Even when they are, the calibration cannot be trusted. Even if calibrated in the factory and then in the lab, the phenomenon of temperature and pressure drift complicates acquisition for passive emitters and transducers. Carefully designed instrumentation and measurement strategy can compensate for component behavioral variance, and redundant emitters and detectors can be used in such designs.
Assuming accuracy above that of a mass produced part is required, the calibration voltages $v_i$ and $v_j$ must be determined simultaneously and be consistently either relative to magnetic flux levels at some point or to each other. If the environment cannot be controlled, re-calibration may be periodically necessary so that the calibration will remain representative.

The emitters can be slightly not identical. The difference will be on the scale. E.g. one of the emitters can be giving 10% bigger reading. But you can ignore this problem for now.

Calibration issues should not be dismissed until later. They must be built into the model tuned by the parameters converged to an optimal during training. Fortunately, since $f$ is unknown and encapsulates calibration factors, the addressing up front of calibration will not likely frustrate proper analysis.
Drawing of Samples and Aligning Training and Use Distributions
It is important, however, to understand that, When training, the distribution of the training samples must match the distribution of the samples encountered when the training is expected to work. This applies directly to the calibration issue and determines the frequency of re-calibration. In essence, training is calibration. This is not new to the recent machine learning craze. Such was the case with self-adjusting PID controllers in the 1990s.
Addressing Questions in the Ideas Section of the Question

When I train the machine learning model with 2 inputs, the prediction is very accurate ... but it means that after training, I cannot move the emitters at all. ... I cannot afford to train the model on all possible distance of the 2 emitters.

That is the case if the training samples are not representative or insufficient in number or the model $f$ is entirely unknown or not used in the convergence strategy.

I have to figure out the position of both emitters relative to each other dynamically. But after knowing the position of both emitters, how do I tell that to the model?

A model does not know the position of emitters or detectors. A model generalizes these. What you tell the model is what is known for sure about $f$ and $\mathcal{I}$.

I have tried training each emitter separately instead of pairing them as input.

That defies the rule that the training distribution must match the usage distribution. Reliability, accuracy, and speed of convergence will all be damaged by doing that.

Use the model to predict position probability density function instead of predicting the position. ... We cannot assume normal distribution or even uniform distribution.

Because of the noise function $n$, the function to be learned is necessarily stochastic, but that is not unusual, and that does not mean that convergence during learning will not occur. It merely means that the loss function cannot be expected to reach zero. It can nonetheless be minimized, even with motion.
Because the objects attached to detectors and sensors are physical and have mass and forces involved are not nuclear or supernatural, acceleration cannot be either $\infty$ or $- \infty$, thus the vectors do not have the Markov property.
If the preparation of training data allows the labeling of the readings and time stamps with reference positions derived from a test fixture using digital encoders with high accuracy, then this project is much more feasible. In such a case, it is the patterns in the time series and their relationship to actual position that is being learned. Then a B-LSTM or GNU type cell for network layers may be the best choice.

Maybe reinforcement learning where the actions are "move left", "move right", ...

Unless the system being designed is required to produce motion control, reinforcement learning or other adaptive control strategies are not indicated. In either case, that the Markov property is not present in a system that involves physical momentum, a form of learning that requires that property may not be the best control strategy.

The emitters are only allowed to move before usage. During usage or prediction, the model can assume that the emitter will not be [stationary]. This allows you to have time to run emitters position calibration algorithm before usage.

It is recommended to design the math and fixture used for training as flexibly as possible and then binding variables only after there is no doubt the system is working and various degrees of freedom are superfluous.

Model input:

1 mean scaled input for each emitter
1 distance value for each
distance

Multiple input
You mentioned there is noise. If the noise is constant, ie you test it in place A and the values returned are always the same, then it means training in different places. If you place it in a place and the first reading is different from the second reading. Then you need to take a lot of readings and select the mean or median of the readings. The central limit theorem says at least 30 readings. This would be the easiest. You could use each sample as an input which allows the NN to learn to filter out the noise. This makes training longer but is probably better than just taking an average.
Scaled input
I know you said it is not important for now, and that the emitters come in different scales. I would scale the output of the emitter proportional to it's scale so that two emitters of different size will produce the same output relative to the sensor no matter how far the sensor is from the emitter. This function might be a simple linear function or more complex, depending if output of the emitter drops faster for a smaller emitter than a larger one.
Distance value
You mentioned dynamically calculating the position of an emitter, but it was placed there so you must know where it is.
This means you could use two solutions. One a co-ordinate system which is a little more complex, or a simpler solution is a distance vector.
There must be a maximum distance that one could place the emitters. Lets assume this distance is 25. You could normalise the data as any distance / maximum distance. This should be repeated for all unique combinations of emitters. i.e if you have 2 (A, B) then only one distance value if you have 3 emitters (A,B,C), then 3 distance values ie from A-B, and A-C, and B-C.
A co-ordinate system is more complex because using a number to denote a position will apply importance to it, for example a grid from 1 to 10 across the x, and y. Emitter at position 10,10 will have a greater importance than an emitter at 1,1. And if it is 0,0 without a bias input your result will be 0.
Structure of the NN
Of course the structure of the NN, the data samples, what you use for validation etc will all play a role.
Perhaps some research on previous work.
See the following:

Significant Location Detection & Prediction in Cellular Networks
Using Artificial Neural Networks
Mobile Localization Based on Received Signal Strength and Pearson's
Correlation Coefficient
Discrete Indoor Three-Dimensional Localization System Based on
Neural Networks Using Visible Light Communication


text:
Suitable reward function for trading buy and sell orders
I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are {"Buy": 0 , "Do Nothing": 1, "Sell": 2}.
Suppose that all the features are well suited for this task. I wanted to use just the standard "Profit & Loss" as a reward, but I hardly thought to get something similar to the above image. The standard P&L will simply place the pair (limit buy order, limit sell order) on every up movement. I don't want that because very often it won't cover the commission and it is not a good indicator to trade manually. I would be interested that the agent can maximize the profit and give me a minimum profit of $100 on every pair (limit buy order, limit sell order).
I would be interested in something similar to the picture below.

Is there a reward function that could allow me to get such a result? If so, what is it?
UPDATE
Is the following utility function can work with the purpose of that question?
$$
U(x) = \max(\\\$100, x)
$$
That seems correct, but I don't know how the agent will be penalized if it covers a wrong transaction, i.e. the pair (limit buy order, limit sell order) creates a loss of money.

Today we have a relevant xkcd: https://xkcd.com/2101/ - I have re-read your question and I think that the big issue for you is "Suppose that all the features are well suited for this task." I am certain that no amount of RL will help you unless you solve the prediction problem first. Looking at reward functions is not going to help you. RL does not magically make the prediction problem go away, it just obscures it. However, if you could create a predictive model that does better than random luck, then you could use RL to discover an investment strategy.

Use the prediction, for multiple time steps ahead, as state features that help the RL to learn a strategy. For this to work, you need your prediction model to have some accuracy better than random guessing at the time scale and profit amounts that you care about.

Please take care on this project. Applying ML to market data is not a new thing, and professional traders with strong math and computing knowledge have already taken  the low-hanging fruit years ago. Your best outcome is likely learning enough to appreciate how hard the problem is. Your worst outcome is thinking you've solved it then losing all your savings when it turns out that was just a random test result.

@NeilSlater I need someone to adjust the code so that we can have predict more than one step ahead. I can surely pay you for your job

Not interested, sorry, and I think you will find that the further you predict ahead, the worse the predictions will become (like weather forecasting), quickly becoming useless for your purpose. However, that does seem like the kind of question you could ask on DataScience stack exchange - explain the architecture of your predictive model, including the rough nature of inputs, and ask how it could be adapted from single-step to multi-step predictions.

While the ML approach to this kind of problems could be interesting, I feel like this is a quantitative finance question, more precisely a question of taking into account market friction in portfolio optimisation. I suggest you read the latest reasearch on the subject to undestand the formalism and the limitations of the standard approaches (combinatorial explosion, lack of predictive value). The bottom line is that this problem won't probably be magically solved by ML. If the features were well suited for this task the problem would have been solved a long time ago.

@fgauth there are better alternatives to a neural network for trading rule generation. Have you looked at Genetic Programs?

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function.
$ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$
where $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals a buy and sell, respectively, and zero otherwise; $sc$ represents the selling cost and $bc$ the buying cost. ROI is the difference between final bank balance and starting bank balance after trading.
You are correct, that the machine learning algorithm will then be influenced by spikes just before a sell.
Nicholls et al showed that using the average profit or area under the trade resulted in better performing trading rules. This approach was used by Schoreels et al. This approach focuses on being in the market to capitalize on profit. It does not penalize the trading rule when it is in the market and the market is going down. The accumulated asset value (AAV) is defined as:
$AAV = \frac{\sum_{i=1}^N [(Price_s - sc) - (Price_b + bc)]}{N}$
where $i$ is a buy and sell trading event, $N$ is the number of buy and sell events, $s$ the day the sale took place, and $b$ is the day the purchase took place.
Nicholls MSc thesis [available April 2019] showed that the fitness function used by Allen and Karjalainen is the preferred fitness function when evolving trading rules for the JSE using evolutionary programs.
Allen and Karjalainen used a fitness function based on the compounded excess returns over the buy-and-hold (buy the first day, sell the last day) strategy. The excess return is given by:
$\Delta r = r - r_{bh}$
where the continuously compounded return of the trading rule is computed as
$r = \sum_{t=1}^T r_i I_b(t) + \sum_{t=1}^T r_f I_s(t) + n\log\left(\frac{1-c}{1+c'}\right)$
and the return for the buy-and-hold strategy is calculated as
$r_{bh} = \sum_{t=1}^T r_t + \log\left(\frac{1-c}{1+c'}\right)$
In the above,
$r_i = \log P_t - \log P_{t-1}$
and $P$ is the daily close price for a given day $t$, $c$ denotes the one-way transaction cost; $r_f$ is the risk free cost when the trader is not trading, $I_b(t)$ and $I_s(t)$ are equal to one if a rule signals buy and sell, respectively, and zero otherwise; $n$ denotes the number of trades and $r_{bh}$ represents the returns of a buy-and-hold, while $r$ represents the returns of the trader.
A fixed trading cost of $c = 0.25\%$ of the transaction was defined but this could be anything like a STATE fee + Brocker fee + Tax, and might even be 2 different values, one for buying and one for selling. Which was the approach used by Nicholls. The continuously compounded return function rewards an individual when the share value is dropping and the individual is out of the market. The continuously compounded return function penalises the individual when the market is rising and the individual is out of the market.
I would recommend that you use the compounded excess return over the buy and hold strategy as your reward function.

Thanks for sharing the detailed answer. Would you mind explaining what $c'$ is in the denominator of the $\log$, as well as what $r_f$, the "risk free cost when the trader is not trading" is?

@PyRsquared $c′$ is the trading cost when selling, while $c$ is the trading cost when buying.  Allen and Karjalainen used a fixed value for both $c$ and $c'$ of $0.25%$ of the trading price, while Nicholls et al used two different values one for buying and one for selling. $r_{f}$ is the risk free rate of return, this is the interest one would get if the money stayed in the bank eg $10%$ would be $1.01$. Nicholls et al used a value of $1$ which results in the day's price

@PyRsquared  rf is the risk free rate of return per day so the 10% in my previous comment is crazy high, it should be divided by 365, and of course that is then compounded, to keep it simple, I kept it at 1.

in any case, don't the log terms cancel out once you subtract $r_{bh}$ from $r$?


text:
How do I compute the variance of the return of an evaluation policy using two behaviour policies?
Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through behavior policies via importance sampling, which is unbiased. But I do not know about the variance of return estimated through two behavior policies $\pi_{b1}$ and $\pi_{b2}$. Does anybody know about the variance or any bound on the variance of estimated return?
Let $G_{0}^{b1}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b1}$ represent the total return for an episode through behavior policy $\pi_{b1}$ and $G_{0}^{b2}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b2}$ represent the total return for an episode through behavior policy $\pi_{b2}$.
It is possible to estimate the return of policy $\pi_{e}$ as follows:
$$G_{0}^{(e,b1)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b1}(a_{t}|s_{t})}*G_{0}^{b1}$$
$$G_{0}^{(e,b2)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b2}(a_{t}|s_{t})}*G_{0}^{b2}$$
I want to compare the variance of $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$. Is there any formulation to compute the variance $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$?

Perhaps give a formula or notes on a variance that you do know how to obtain in the question, so that someone answering can use the same notation, and understands where to start from so that you will understand

If you simulate many trajectories and receive many estimates of the two returns you're interested in, you could empirically compare their sample variances.
However, the variance of ordinary importance sampling is in general unbounded.
If you're wanting some theoretical bounds on the variance of importance sampling estimates, I'd start with weighted importance sampling, whose variance converges to zero (Sutton and Barto, section 5.5).


text:
Why would we use attention in convolutional neural networks, and how would we apply it?
Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures sequence information. The hidden state can be fed into a small MLP that produces a context vector summarizing the salient features of the hidden state.
In the context of NLP, convolutional networks are not as straightforward. They have the notion of channels that are different feature representations of the input, but are channels the equivalent to hidden states? Particularly, this raises two questions for me:

Why use attention in convolutional networks at all? Convolutions have shown to be adept feature detectors––for example, it is known that higher layers learn small features such as edges while lower layers learn more abstract representations. Would attention be used to sort through and weigh these features?

In practice, how would attention be applied to convolutional networks? The output of these networks is usually (batch, channels, input_size) (at least in PyTorch), so how would the attention operations in recurrent networks be applied to the output of convolutional networks?

References
Convolutional Sequence to Sequence Learning, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017

It seems that in the reference you provided they use attention to compute weights for the encoder representations based on the context provided by the decoder (Fig. 1). As far as I can tell, attention is applied after convolution (in fact, after the GLU step), so it does not affect the feature maps directly. Rather, attention is used to select the target words in the decoder. In other papers (e.g., this one), attention is applied directly to the feature maps in a way that is more similar to what you described.
Regarding your second question, the paper you reference actually provides a link to the source code they used. It is written in Lua (using Torch) instead of PyTorch, probably because PyTorch development was just starting when the paper was published. At any rate, you should be able to follow the Lua code and translate it into PyTorch.


text:
Is it possible to use AI to reverse engineer software?
I was thinking of something of the sort:

Build a program (call this one fake user) that generates lots and lots and lots of data based on the usage of another program (call this one target) using stimuli and response. For example, if the target is a minesweeper, the fake user would play the game a carl sagan number of times, as well as try to click all buttons on all sorts of different situations, etc...
run a machine learning program (call this one the copier) designed to evolve a code that works as similar as possible to the target.
kablam, you have a "sufficiently nice" open source copy of the target.

Is this possible?
Is something else possible to achieve the same result, namely, to obtain a "sufficiently nice" open source copy of the original target program?

Perhaps I should add a disclaimer: I am not a computer scientist, but I'm learning to code (python). I'm a m.sc. in math.

Remarkably, more or less the scenario you describe is not only feasible and has already been demonstrated (detailed explanation and fascinating videos at link).
However, the fidelity of the copy is currently quite limited:

So for now, your copy will be quite low quality.  However, there is a big exception to this rule: if the software you are copying is itself based on machine learning, then you can probably make a high-quality copy quite cheaply and easy, as I and my co-authors explain in this short article.
Interesting question and I'm quite sure that the correct answer will change rapidly over the next few years.

you have no idea how much this is EXACTLY what i wanted to hear (or read)

are there evil capitalist responses to this? As in, trying to make this illegal or something...

Also, how much power (computing, physical, intellectual) does this require at the moment?

@IpsumPanEst Actually, reverse-engineering is a perfectly respectable free market idea that helps to keep our market nice and dynamic.  Definitely something that terms of service may try to restrict for obvious reasons, but reverse engineering is an important part of keeping markets competitive.

@IpsumPanEst Power/compute requirements are really modest for copying machine learning services, but very immodest (probably not practical) for copying conventional software (and, the fidelity will be low), but this is a very active area of research & you should expect non-linear progress.

Dear @Edward I recommend that you read Max Tegmarks book LIFE 3.0 because in it he has some excellent thought-examples of how A.I.'s could create and market their "home made" videos, tv shows and series and even enter subliminal messages that could in theory affect peoples judgment when it comes to elections or choices regarding limiting A.I.'s influence etc. Read the book! https://en.wikipedia.org/wiki/Life_3.0

Dear @EdwardDixon. If this praxis (automating reverse-engineering) actually helps big capital and not the people, i'm not gonna be a part of it.

@IpsumPanEst if you think about the second-order effects of reverse engineering, it increases competition (think scrappy startup learning from strengths, weaknesses of incumbent's products) => downward pressure on big company margins => increased consumer surplus for the litle guy.

i was thinking more of something closer to: make all software free and accessible to all people, not just so small startups can be competitive. To make charging for software an obsolete anti-praxis.

politics apart, the implications for imago-mundi creation in sentient beings' minds is absolutely awesome for philosophy of mind. We might learn a lot about consciousness through ML and your program

This is the proposed way to reverse engineer software using AI.

Program fake_user operates program target_prog in diverse ways to generate a huge and comprehensive data set.
The parameters of an artificial network are trained to produce within specified accuracy and reliability criteria a behavioral equivalent of target_prog.

Not only is this possible, but it is becoming standard practice for AI projects other than reverse engineering games.
There are caveats.

Program target_prog may be of sufficient complexity to exceed the capabilities of existing network designs and convergence techniques.
The project may lack access to funds and computing resources to complete the generation and training required to achieve reasonable accuracy, with sufficient reliability, in the time allotted.
The expertise of those involved may not be sufficient to produce satisfactory results.
Although the source code is not copied and the parameter state achieved through learning contains equivalent functionality, there is no guarantee that a civil liability may not result. Copyright law one or more jurisdictions may be interpreted as a protection against this kind of copying even though the text of the source code was not copied verbatim.

A recent paper from January 2023 does this too: "an algorithm that synthesizes the source code of simple 2D video games from a small amount of observed video data"
https://www.basis.ai/blog/autumn/
Original research article: https://dl.acm.org/doi/10.1145/3571249

Why do I suddenly have ants in my pants? :)


text:
How do I determine which relevant features have been learned during training in a CNN?
Is there any way to control the extraction of features? How do I determine which features are been learned during training, i.e relevant information is been learned or not?

There are methods called "scoring systems" where you give a image scores such as "0.9 stripes, 0.0 red, 0.8 hair, ..." and use those scores to classify objects. It's an older idea, not used to determine if the network is learning. It's not in a standard CNN.
To determine if relevant information is being learned or not, it's standard to use the testing accuracy, training accuracy, confusion matrix, or AUC.
Determining what exactly a CNN is learning is a complicated research problem that's ongoing. In short - you can't really know. For a basic network, you can tell that it is learning something but not what it's actually using to make determinations.

Yes there is! If a model generalizes well to the test set, we already know that it has found some useful features. However, the latent representation of the data may still be "entangled" - a single element of the latent vector may actually encode information about multiple attributes of the input, or a single attribute may be spread across multiple elements.  We usually prefer a representation in which the features are represented by the axes of the latent space - a "disentangled" representation.  For example, if we were encoding faces, it would be nice to have an axis for smiling/not, another for masculine/feminine, and ao on.
Pushing models to learn "clean" (disentangled) representations is an active sub-field of machine learning research with practical applications (like interpretability, but also because it makes it easier for "downstream" models to learn their tasks, e.g. a control policy in reinforcement learning system taking as input a learned representation from a world model).
Where to begin?  Start with L2 regularisation to push your network to "spend" those weights wisely (more weights close to zero => sparser latent vector) and work your way up from there.

So there's pictures of low level activation maps, and some gradient based information where yoy take the deriviative of the output with respect to the input and generate a heatmap.
I kind of have my doubts on how usefull this is in general, imo it kind of is creating a fallacious illusion of understanding.
There's some additional research using blurring to figure out the relevant features also but again I have my doubts.
Probably the most usefull is generating images by optimizing your class score. You can learn how badly your CNN actually labels things(doing this makes you realize quickly that CNN are garbage at actually understanding and incredibly easy to trick).


text:
What is a continuous-attractor neural network?
I am reading about CANN. However, I do not seem to grasp what it is. Maybe someone who has worked with it can explain it? I found out about it while reading about RatSLAM. I understand that it helps to keep long/short term memory.


text:
Mismatch between the definition of the GAN loss function in two papers
I was trying to understand the loss function of GANs, but I found a little mismatch between different papers.
This is taken from the original GAN paper:

The adversarial modeling framework is most straightforward to apply when the models are both multilayer perceptrons. To learn the generator's distribution $p_{g}$ over data $\boldsymbol{x}$, we define a prior on input noise variables $p_{\boldsymbol{z}}(\boldsymbol{z})$, then represent a mapping to data space as $G\left(\boldsymbol{z} ; \theta_{g}\right)$, where $G$ is a differentiable function represented by a multilayer perceptron with parameters $\theta_{g} .$ We also define a second multilayer perceptron $D\left(\boldsymbol{x} ; \theta_{d}\right)$ that outputs a single scalar. $D(\boldsymbol{x})$ represents the probability that $\boldsymbol{x}$ came from the data rather than $p_{g}$. We train $D$ to maximize the probability of assigning the correct label to both training examples and samples from $G$. We simultaneously train $G$ to minimize $\log (1-D(G(\boldsymbol{z})))$ :
In other words, $D$ and $G$ play the following two-player minimax game with value function $V(G, D)$ :

$$
\min _{G} \max _{D} V(D, G)=\mathbb{E}_{\boldsymbol{x} \sim p_{\text {data }}(\boldsymbol{x})}[\log D(\boldsymbol{x})]+\mathbb{E}_{\boldsymbol{z} \sim p_{\boldsymbol{z}}(\boldsymbol{z})}[\log (1-D(G(\boldsymbol{z})))]
$$
Equation (1) in this version of the pix2pix paper

The objective of a conditional GAN can be expressed as
$$
\begin{aligned}
\mathcal{L}_{c G A N}(G, D)=& \mathbb{E}_{x, y}[\log D(x, y)]+\\
& \mathbb{E}_{x, z}[\log (1-D(x, G(x, z))],
\end{aligned}
$$
where $G$ tries to minimize this objective against an adversarial $D$ that tries to maximize it, i.e. $G^{*}=$ $\arg \min _{G} \max _{D} \mathcal{L}_{c G A N}(G, D)$.
To test the importance of conditioning the discriminator, we also compare to an unconditional variant in which the discriminator does not observe $x$ :
$$
\begin{aligned}
\mathcal{L}_{G A N}(G, D)=& \mathbb{E}_{y}[\log D(y)]+\\
& \mathbb{E}_{x, z}[\log (1-D(G(x, z))] .
\end{aligned}
$$

Putting aside the fact that pix2pix is using conditional GAN, which introduces a second term $y$, the 2 formulas are quite resemble, except that in the pix2pix paper, they try to get minimax of ${\cal{L}}_{cGAN}(G, D)$, which is defined to be $E_{x,y}[...] + E_{x,z}[...]$, whereas in the original paper, they define $\min\max V(G, D) = E[...] + E[...]$.
I am not coming from a good math background, so I am quite confused. I'm not sure where the mistake is, but assuming that $E$ is expectation (correct me if I'm wrong), the version in pix2pix makes more sense to me, although I think it's quite less likely that Goodfellow could make this mistake in his amazing paper. Maybe there's no mistake at all and it's me who do not understand them correctly.

What mistake were you talking about? I don't understand which mistake you're referring to from you description.

The question is about a mismatch between the loss function in two papers on GANs. The first paper is Generative Adversarial Nets Ian J. Goodfellow et. al., 2014, and the excerpt image in the question is this.

The adversarial modeling framework is most straightforward to apply when the models are both multilayer perceptrons. To learn the generator’s distribution $p_g$ over data $x$, we define a prior on input noise variables $p_z (z)$, then represent a mapping to data space as $G (z; \theta_g)$, where $G$ is a differentiable function represented by a multilayer perceptron with parameters $\theta_g$. We also define a second multilayer perceptron $D (x; \theta_d)$ that outputs a single scalar. $D (x)$ represents the probability that $x$ came from the data rather than pg. We train $D$ to maximize the probability of assigning the correct label to both training examples and samples from $G$. We simultaneously train $G$ to minimize $\log (1 − D(G(z)))$:
In other words, $D$ and $G$ play the following two-player minimax game with value function $V (G, D)$:
$$ \min_G \, \max_D V (D, G) = \mathbb{E}_{x∼p_{data}(x)} \, [\log \, D(x)] \\
\quad\quad\quad\quad\quad\quad\quad  + \, \mathbb{E}_{z∼p_z(z)} \, [\log \, (1 − D(G(z)))] \, \text{.} \quad \text{(1)} $$

The second paper is Image-to-Image Translation with Conditional Adversarial Networks, Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A. Efros, 2018, and the excerpt image in the question is this.

The objective of a conditional GAN can be expressed as
$$ \mathcal{L}_{cGAN} (G, D) = \mathbb{E}_{x, y} \, [\log D(x, y)] \\ \quad\quad\quad\quad\quad\quad\quad + \mathbb{E}_{x, z} \, [\log \, (1 − D(x, G(x, z))], \quad \text{(1)} $$
where $G$ tries to minimize this objective against an adversarial $D$ that tries to maximize it, i.e.
$$ G^{∗} = \arg \, \min_G \, \max_D \mathcal{L}_{cGAN} (G, D) \, \text{.} $$
To test the importance of conditioning the discriminator, we also compare to an unconditional variant in which the discriminator does not observe $x$:
$$ \mathcal{L}_{GAN} (G, D) = \mathbb{E}_y \, [\log \, D(y)] \\
\quad\quad\quad\quad\quad\quad\quad + \mathbb{E}_{x, z} \, [\log \, (1 − D(G(x, z))] \, \text{.} \quad \text{(2)} $$

In the above $G$ refers to the generative network, $D$ refers to the discriminative network, and $G^{*}$ refers to the minimum with respect to $G$ of the maximum with respect to $D$. As the question author tentatively put forward, $\mathbb{E}$ is the expectation with respect to its subscripts.
The question of discrepancy is that the right hand sides do not match between the first paper's equation (1) and the second paper's equation (2) which is absent of the condition involving $y$.
First paper:
$$ \mathbb{E}_{x∼p_{data}(x)} \, [\log \, D(x)] \\
\quad\quad\quad\quad\quad\quad\quad  + \, \mathbb{E}_{z∼p_z(z)} \, [\log \, (1 − D(G(z)))] \, \text{.} \quad \text{(1)} $$
Second paper:
$$ \mathbb{E}_y \, [\log \, D(y)] \\
\quad\quad\quad\quad\quad\quad\quad + \mathbb{E}_{x, z} \, [\log \, (1 − D(G(x, z))] \, \text{.} \quad \text{(2)} $$
The second later paper further states this.

GANs are generative models that learn a mapping from random noise vector $z$ to output image $y, G : z \rightarrow y$. In contrast, conditional GANs learn a mapping from observed image $x$ and random noise vector $z$, to $y, G : {x, z} \rightarrow y$.

Notice that there is no $y$ in the first paper and the removal of the condition in the second paper corresponds to the removal of $x$ as the first parameter of $D$. This is one of the causes of confusion when comparing the right hand sides. The others are use of variables and degree of explicitness in notation.
The tilda $~$ means drawn according to. The right hand side in the first paper indicates that the expectation involving $x$ is based on a drawing according to the probability distribution of the data with respect to $x$ and the expectation involving $z$ is based on a drawing according to the probability distribution of $z$ with respect to $z$.
The removal of the observation of $x$ from the second right hand term of the second paper's equation (2), which is the first parameter of $G$, the replacement of that equation's $y$ variable with the now freed up $x$ variable, and the acceptance of the abbreviation of the tilda notation used in the first paper then brings both papers into exact agreement.

Thank you for your reply and sorry for the late response. Regarding this paragraph: "The question of discrepancy is that the right hand sides do not match between the first paper's equation (1) and the second paper's equation (2) which is absent of the condition involving ." No, it is not what confuses me. The question of discrepancy is the mismatch of the left hand sides of the 2 papers: the first paper defines the Expectations to be the results of the min-max operation, where as the second paper suggests to take the min-max of the Expections.

What is meant by both papers is that we have two agents (generator and discriminator) playing a game with the value function V defined as a sum of the expectations (i.e. an expectation of the outcome value defined as a sum of two terms, or actually a logarithm of a product...). The generator uses a strategy G encoded in the parameters of its neural network (θg), the discriminator uses a strategy D encoded in the parameters of its neural network (θd). Our goal is to (hopefully) find such a pair of strategies (a pair of parameter sets θgmin and θdmax) that produce the minimax value.
While trying to find the (θgmin, θdmax) pair using gradient descent, we actually have two loss functions: one is the loss function for G, parameterized by θg, another is the loss function for D, parameterized by θd, and we train them alternatively on minibatches together.
If you look at the Algorithm 1 in the original paper, the loss function for the discriminator is -log(D(x; θd)) - log(1 - D(G(z); θd), and the loss function for the generator is log(1 - D(G(z; θg)) (in both cases, in the original paper, x is sampled from the reference data distribution and z is sampled from noise):
The ideal value for the loss function of the discriminator is 0, otherwise it's greater than 0. The "loss" function of the generator is actually negative, but, for better gradient descent behavior, can be replaced with -log(D(G(z; θg)), which also has the ideal value for the generator at 0. It is impossible to reach zero loss for both generator and discriminator in the same GAN at the same time. However, the idea of the GAN is not to reach zero loss for any of the game agents (this is actually counterproductive), but to use that "double gradient descent" to "converge" the distribution of G(z) to the distribution of x.

Thanks for your reply. Yet, I am more concerned of the whole equations themselves. The pix2pix paper wants to minimize the expectation while the GAN paper shows that the expectation is the result of the minimization.

Yes, and they are different that one is defined to be the min of max of expectation (pix2pix) while the other is defined as the min of max of value function (GAN paper). Does that make sense?

So let me rephrase the question like this: what is the target function to be optimize for a general GAN? In the GAN paper, Goodfellow uses $V(D, G)$, but what is $V(D, G)$ exactly? Is it $V(D,G) = E(\log(D(y)) + E(1-\log(D(G(x, z))$

@AugLe Edited again. Hopefully now it's clearer what is actually optimized in GANs.

I'm not sure I understand your question. However responding to your question in the comments. The difference between the two objectives is that:
In an ordinary GAN, we want to push $p(G)$ to be as close as possible to $p(data)$
In a conditional GAN, we have a context $c$. If we imagine for ease of understanding that $c=[1,2,3]$ is a discrete variable where all the data can be categorised under one of these c values , then we want to:
 push $p(G|c=1)$ as close as possible to $p(data|c=1)$
push $p(G|c=2)$ as close as possible to $p(data|c=2)$
push $p(G|c=3)$ as close as possible to $p(data|c=3)$


text:
What is the impact of scaling the features on the performance of the model?
I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $Y$. I would like to use PLS Regression, and I understand that standardization is very important here. I am programming in Python, using scikit-learn.
The type and range for the features varies. Some are int64 while others are floating point numbers. Some features generally have small (positive or negative) values, while others have a very large value. I have tried using various scalers (e.g. standard scaler, normalize, min-max scaler, etc.). Yet, the R2/Q2 are still low.
I have a few questions:

Is it possible that by scaling, some of the very important features lose their significance, and thus contribute less to explaining the variance of the response variable?
If yes, if I identify some important features (by expert knowledge), is it OK to scale other features but those? Or scale the important features only?
Some of the features, although not always correlated, have values that are in a similar range (e.g. 100-400), compared to others (e.g. -1 to 10). Is it possible to scale only a specific group of features that are within the same range?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. We do feature scaling to make our model robust to outliers and make an initial impact of every feature on the model will be roughly similar
Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling, but, still, it might be a good idea to rescale/standardize your data.

You should explore your data more carefully, find the outliers, apply transformation if needed.
Not sure if it is a good idea
You can apply different preprocessing techniques like MinMaxScaller, Rank,  Log transform, Extracting square root, StandartScaller and etc.


text:
What is the use of the $\epsilon$ term in this back-propagation equation?
I am currently looking at different documents to understand back-propagation, mainly at this document. Now, on page 3, there is the $\epsilon$ symbol involved:
$$
\Delta w_{k j}=\varepsilon \overbrace{\left(t_{k}-a_{k}\right) a_{k}\left(1-a_{k}\right)}^{\delta_{k}} a_{j}
$$
While I understand the main part of the equation, I don't understand the $\epsilon$ factor. Searching for the meaning of the $\epsilon$ in math, it means (for example) an error value to be minimized, but why should I multiply with the error (it is denoted as E anyways).
Shouldn't the $\epsilon$  be the learning rate in this equation? I think that would be what makes sense, because we want to calculate by how much we want to adjust the weight, and since we calculate the gradient, I think the only thing that's missing is the multiplication with the learning rate. The thing is, isn't the learning rate usually denoted with the $\alpha$?


text:
Can Machine Learning be applied to decipher the script of lost ancient languages?
Can Machine Learning be applied to decipher the script of lost ancient languages (namely, languages that were being used many years ago, but currently are not used in human societies and have been forgotten, e.g. Avestan language)?
If yes, is there already any successful experiment to decipher the script of unknown ancient languages using Machine Learning?

Avestan is still survived as a liturgical language, though its modern form is sometimes synthesised with Sanskrit (text-books on Avestan have references to Sanskrit). I'm sure you could find someone speaking Avestan on YouTube, if you were so inclined.

@TautologicalRevelations , (1) know that currently Avestan language is known. (2) The question is about possibility of deciphering forgotten language using Machine Learning and not the Avestan language. You can consider another example rather than Avestan if you believe Avestan is still alive. But consider the main question: Using Machine Learning for deciphering forgotten languages.

There is a bit of controversy as to what constitutes a dead language. I've chosen Ugaritic and Linear B for my answer. Best wishes to you.

Avestan is either dead or near-dead, but it is not a forgotten/lost language.

I say yes it definitely could be. But i agree with Skim you need some information as a starting point. Egyptian hieroglyphs were only (recently) understood following the discovery of the Rosetta Stone (https://en.m.wikipedia.org/wiki/Rosetta_Stone). With the same message in both known and unknown language the program could find the/a correlation.
Without that info you would have to guess the potential content of the message. The results would then be confirmation-biased: how could you know it had worked properly? Say the program managed to ‘translate’, outputting a coherent phrase about bananas, great. Even if it was very consistent across multiple samples and 99% confidence, without any “control set data” it could just as easily be about, say, fish or even a prophesy on the super-bowl winner.
Still a fun project tho.

This is very similar issue to the Symbol Grounding Problem - which is a general problem with important consequences for AI

There is an academic paper that studies a neural approach to deciphering ancient languages:
Source: https://arxiv.org/pdf/1906.06718.pdf

"In this paper we propose a novel neural approach for automatic decipherment of lost languages. To compensate for the lack of strong supervision signal, our model design is informed by patterns in language change documented in historical linguistics. The model utilizes  an  expressive  sequence-to-sequence model to capture character-level  correspondences between cognates. To effectively train the model in an unsupervised manner, we innovate the training procedure by formalizing it as a minimum-cost flow  problem. When applied to the decipherment of Ugaritic, we achieve a 5.5% absolute improvement overstate-of-the-art  results. We also  report  the first automatic results in deciphering Linear B, a syllabic language related to ancient Greek, where our model correctly translates 67.3% of cognates." ― Luo, Jiaming, Yuan Cao, and Regina Barzilay. "Neural Decipherment via Minimum-Cost Flow: from Ugaritic to Linear B." arXiv preprint arXiv:1906.06718 (2019).

Further Reading and Articles for the Layperson:--

https://themindguild.com/artificial-intelligence-is-now-translating-long-lost-ancient-languages/
https://bstrategyhub.com/sanskrit-is-the-best-language-for-artificial-intelligence-says-nasa/
https://www.openculture.com/2019/07/can-artificial-intelligence-decipher-lost-languages.html

I would guess no, because if the language is unknown (no data available on it), then we would not have training data with which the machine learning algorithm could learn from.
If it is related to some known language, then some statistical analysis can lead to a guess at decipherment (assuming certain similarities among the two languages).
If interested on general language decipherment, see the following where they decipher scripts using available information on the language of interest: http://www.aclweb.org/anthology/W99-0906. They utilize the Expectation Maximization algorithm.
I'm sure more google searching can lead to other examples that use machine learning algorithms, but they would most likely have some known information or a body of assumptions to make our problem scope easier.

I would say that it depends on whether that language would have wanted to be decyphered.
The origin of Cryptography dates back to around 700-800 AD.
We may not know the methods by which these texts are obscurred. One such example is the Lesser Banishing Ritual of the Pentagram which, revealed only by happenstance, formed the basis of a whole new esoteric renaissance in the west, the lexicon included forming a basis for the lexicon in use by these orders today.
This Rite centers around the letters INRI, which tradition says were written upon the cross of Jesus Christ as an abbreviation for Jesus of Nazareth, King of the Jews. There are, however, numerous other levels of occult meaning regarding these four letters in the Rosicrucian Magical Tradition. One of these is a Hermetic secret alluded to by the Latin phrase "Igne Natura Renovatur Integra" which means "By fire, nature is perfectly renewed." These four letters additionally adorn the rays of the angles of the Rose Cross Lamen worn by Adepts of the Ordo Rosae Rubeae et Aureae Crucis.
A deeper interpretation lies occulted behind the attributions of the Hebrew letters and the Magical Forces to the Paths on the Qabalistic Tree of Life. The Path attributed to the Hebrew letter y is attributed to the Zodiacal Sign Virgo as well, that of n to Scorpio, and r to the Sun. There exist further magical associations between the Sign Virgo with the Egyptian Goddess Isis, Scorpio with Apophis, and the Sun with Osiris. When the first letter is taken from the Names of each of these Gods, the name "IAO" is formed. Additionally, due to the Signs associated with Isis, Apophis, and Osiris, they form the letters "LVX."
Thus within the letters IRNI lie concealed the letters IAO and LVX, which may also be found upon the rays of the angles of the Rose Cross Lamen worn by Adepts of the R. R. et A. C. The name IAO was considered by the Gnostics to be the Supreme Name of God. Its letters further allude to Salt, Sulfur, and Mercury in Alchemy and to an even more recondite secret symbolized by the relationship between Isis, Apophis, and Osiris.
So I would say for machine learning to be able to decipher real languages, it would almost have to be intuitive, able to draw parallels and connect various points of reference together. It would have to determine what things mean, when a lot of the times, this is virtually impossible.
Your variables are infinite. You have no real method of finding out what they are. Add to that context, lexicons, deliberate obfuscation, culture and the passage of time and I think it would not be realistic to expect machine-learning to provide much help.


text:
What kind of search method is A*?
What kind of search method is A*? Explain to me with an example.

A* is an informed search algorithm. A* informed because it is based on the use of a heuristic function, which estimates the distance of each node to the goal, that is, the heuristic function provides information about the distance from any node to the goal node. The heuristic function can e.g. be the Euclidean distance (in the case this can be defined).
More precisely, at each step, A* needs to select the next node to explore. It chooses the node $n$ with the smallest value of $$f(n) = g(n) + h(n),$$ where $g(n)$ is the actual distance from the starting node to node $n$ and $h(n)$ is the heuristic function that estimates the distance from node $n$ to the goal node.
For example, suppose that we want to find the shortest path from Paris to Madrid. If you are already in Madrid, then you can estimate that the distance from Madrid to Madrid is zero, so $h(\text{Madrid}) = 0$. However, if you're still in Paris, what is the estimate of the distance from Paris to Madrid? We can e.g. pick up a map of the world and choose $h(\text{Paris})$ to be the length of the straight segment that goes from Paris to Madrid. Similarly, we can do this for other intermediate cities. There are other ways of estimating this distance, but this is an understandable one, given that people are usually familiar with the Euclidean distance. (This example is a little bit misleading because the Earth is not flat and so the Euclidean geometry does not really apply, but, for simplicity, you can ignore this).
For info regarding the completeness and optimality of A*, have a look at https://ai.stackexchange.com/a/8907/2444.


text:
How do we prove the n-step return error reduction property?
In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-step return error reduction property":

But they don't prove it. I was thinking it should not be too hard but how can we show this? I was thinking of using the definition of n-step return (eq. 7.1 on previous page):
$$G_{t:t+n} = R_{t+1} + \gamma*R_{t+2} + ... + \gamma^{n-1}*R_{t+n} + \gamma^{n}*V_{t+n-1}(S_{t+n})$$
Because then this has the $V_{t+n-1}$ in it already. But in the definition above of the n-step return it uses $V_{t+n-1}(S_{t+n})$ but on the right side of the inequality (7.3) that we want to prove it is just little s $V_{t+n-1}(s)$ ? So kind of confused here which state s it is using? And then I guess after this probably pull out a $\gamma^{n}$ term or something, how should we go from here?
This is the newest Sutton Barto book (book page 144, equation 7.3):
https://drive.google.com/file/d/1opPSz5AZ_kVa1uWOdOiveNiBFiEOHjkG/view

Let's start by looking at:
$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$
We can rewrite this by plugging in the definition of $G_{t:t+n}$:
\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert,
\end{aligned}
where $R_{t:t+n} \doteq R_{t + 1} + \gamma R_{t + 2} + \dots + \gamma^{n - 1} R_{t + n}$.
If you go all the way back to page 58 of the book, you can see the definition of $v_{\pi}(s)$:
\begin{aligned}
v_{\pi}(s) &\doteq \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \\
%
&= \mathbb{E}_{\pi} \left[ R_{t:t+n} \mid S_t = s \right] + \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right]
\end{aligned}
Using this, we can continue rewriting where we left off above:
\begin{aligned}
& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ R_{t:t+n} + \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert \\
%
=& \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ \gamma^n V_{t + n - 1}(S_{t + n}) \mid S_t = s \right] - \gamma^n \mathbb{E}_{\pi} \left[ \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert \\
%
=& \gamma^n \max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \mid S_t = s \right] \Bigr\rvert
\end{aligned}
Because the absolute value function is convex, we can use Jensen's inequality to show that the absolute value of an expectation is less than or equal to the expectation of the corresponding absolute value:
$$\left| \mathbb{E} \left[ X \right] \right| \leq \mathbb{E} \left[ \left| X \right| \right].$$
This means that:
\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
\end{aligned}
Now, the important trick here is to see that:
$$\max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t + n}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + n + k + 1} \Bigr\rvert \mid S_t = s \right] \leq \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right]$$
I'm skipping the formal steps to show that this is the case to save space, but the intuition is that:

The left-hand side of this inequality involves finding an $S_t = s$ such that some function of $S_{t + n}$ is maximized, whereas the right-hand side involves finding an $S_t = s$ such that exactly the same function of $S_{t}$ is maximized.
In the left-hand side, selecting an $S_t = s$ implicitly induces a probability distribution over multiple possible states $S_{t + n}$, given by $S_t$, the environment's transition dynamics, and the policy $\pi$. Intuitively, this is more "restrictive" for the $\max$ operator, it does not have the "freedom" to directly select a single state $S_{t + n}$ such that the function of $S_{t + n}$ is maximized. The right-hand side is free to choose any single state $S_t = s$ such that $S_t$ in the right-hand side were equal to an "optimal" $S_{t+n}$ on the left-hand side, but it is also free to make even better choices which might never be uniquely reachable after $n$ steps on the left-hand side.

We can use this to rewrite the previous inequality we had (where we might be making the right-hand side a bit bigger than it was, but that's fine, it already was an upper bound anyway so that inequality will still hold):
\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - \sum_{k = 0}^{\infty} \gamma^k R_{t + k + 1} \Bigr\rvert \mid S_t = s \right] \\
%
&= \gamma^n \max_s \mathbb{E}_{\pi} \left[ \Bigl\lvert V_{t + n - 1}(S_{t}) - v_{\pi}(s) \Bigr\rvert \mid S_t = s \right].
\end{aligned}
After this rewriting we've got a hidden $\mathbb{E}_{\pi}$ "inside" another $\mathbb{E}_{\pi}$ (because the definition of $v_{\pi}(s)$ contains an $\mathbb{E}_{\pi}$), which I suppose is kind of ugly... but mathematically meaningless.
The maximum of a random variable is an upper bound on the expectation of that random variable, so we can get rid of the expectation in the right-hand side (again potentially increasing the right-hand side, which again is still fine since it's already an upper bound anyway):
\begin{aligned}
\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} - v_{\pi}(s) \mid S_t = s \right] \Bigr\rvert &\leq \gamma^n \max_s \Bigl\lvert V_{t + n - 1}(s) - v_{\pi}(s) \Bigr\rvert,
\end{aligned}
which we can finally rewrite to Equation (7.3) in the book by moving the subtraction of $v_{\pi}(s)$ outside of the expectation on the left-hand side of the inequality (which is fine because, as I already mentioned above, the definition of $v_{\pi}(s)$ itself contains another $\mathbb{E}_{\pi}$ anyway).

Your bullet points explaining the "trick" make intuitive sense. Could you provide a link to someone showing the formal steps?

@PhilipRaeisghasem I don't know of any such links by heart, so can't provide one, sorry :(

I'd argue that, if it's that hard to find this information, then including it in the post would greatly increase the post's value. Or maybe as a separate, linked question?

@PhilipRaeisghasem Absolutely, either would be fine... maybe separate question better, since this one's already fairly long. I'm currently working towards a paper submission deadline though, so I really don't personally feel like working out the math and writing it all down any time soon.

Can we just say that s_t+1 is a subset of s therefore its max over s_t+1 must <= max over s?

@Phizaz Hmmm not exactly. Close to that, I'd say "given a completely free, unrestricted choice for $S_t$, the set of possible states $S_{t+n}$ is a subset of the set of possible states $S_t$". Very often it will not be a proper subset though, very often they'll be equal. But even then, a very important other point is that the probability distributions over those sets will be different. When the choice of $S_t$ is unrestricted, you can assign full probability mass to whichever is "best". Even if that best state may still be possible for $S_{t+n}$, it may no longer have full probability mass

@DennisSoemers I agree. However, will we not be able to prove without the probability mass argument?

@Phizaz Ah yes I think you're completely right. Intuitively it didn't "feel" right to me to omit that... but I guess math doesn't care about my feelings :)

how to intuitively understand the error reduction property? how does this guarantee the convergence?