text:
What is meant by gene, chromosome, population in genetic algorithm in terms of feature selection?
I am trying to understand the genetic algorithm in terms of feature selection and these features are extracted using a machine learning algorithm.
Let's suppose I have data of heart rate for 3 minutes collected from $50$ subjects. From these 3-minute heart rate, I extracted $5$ features, like the mean, standard deviation, variance, skewness and kurtosis. Now the shape of my feature set is (50, 5).
I want to know what are gene, chromosome and population in genetic algorithm related to the above scenario.
What I understand is each feature is a gene, and a set of all features for one subject (1, 5) is the chromosome, and the whole feature set (50, 5) is a population. But I think this concept is not correct. Because in the genetic algorithm, we take a random population, but according to my concept complete data is population, so how random data is selected.
Can anyone help me to understand it?

What is a goal of the model? Regression, classification or something else?
I think your 50 subjects are nor individuals of a population, but a part of a way to define a fitness function. Individuals of a population will be binary 5-element vectors, that represent if features are important or not.

its classification, the fitness function gives accuracy

A chromosome in this case could be a set of filters, each extracting a different feature (analogous to Convolutional Neural Network). Your question doesn't say what you want to do with these features, so this solution is made under the assumption that there is a fitness function which would take these features as an input and output a score. Then, each gene is a parameter for a filter, each chromosome defines a set of such filters, which makes up an individual. A population is a set of such individuals.

Genetic algorithms, also known as evolutionary search, provide a general technique to optimize an objective function. We also say that we are trying to maximize fitness. This means that we are trying to find an individual with the highest possible fitness. We start with a population, say 100 individuals, and using mutation and crossover  we generate offspring among whom we hope to find fitter individuals and as the generations progress, we get better and better.
One way to start this all is to think about the "fitness" or objective function. What is it that we want in the best individuals? Can we model that? How do we model that?
In your case, does a specific measurement (those 5 numbers you mention) say how fit an individual is? And that fitness can be one number, say from 1 to 100, or it could be unbounded (as in real life where things get better and better with temporary regressions).
So the challenge is how to map the features to a number. That's a math function to design.
Genes are what change from individual to individual and they mutate from parent to offspring and they are shared in crossover. Given a set of genes, what is the fitness? If you can answer that question (meaning a mathematical function to map the genes of each individual to a number), then you have a genetic algorithm to run and it will find the fittest individuals according to your (math) function.


text:
How can I have the same input and output shape in an auto-encoder?
I'm building a denoising autoencoder. I want to have the same input and output shape image.
This is my architecture:
input_img = Input(shape=(IMG_HEIGHT, IMG_WIDTH, 1))

x = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(64, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

x = Conv2D(32, (3, 3), activation='relu', padding='valid')(encoded)
x = UpSampling2D((2, 2))(x)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

# decodedSize = K.int_shape(decoded)[1:]

# x_size = K.int_shape(input_img)
# decoded = Reshape(decodedSize, input_shape=decodedSize)(decoded)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

My input shape is: 1169x827
This is Keras output:
Model: "model_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_7 (InputLayer)         [(None, 1169, 827, 1)]    0
_________________________________________________________________
conv2d_30 (Conv2D)           (None, 1169, 827, 32)     320
_________________________________________________________________
max_pooling2d_12 (MaxPooling (None, 585, 414, 32)      0
_________________________________________________________________
conv2d_31 (Conv2D)           (None, 585, 414, 64)      18496
_________________________________________________________________
max_pooling2d_13 (MaxPooling (None, 293, 207, 64)      0
_________________________________________________________________
conv2d_32 (Conv2D)           (None, 291, 205, 32)      18464
_________________________________________________________________
up_sampling2d_12 (UpSampling (None, 582, 410, 32)      0
_________________________________________________________________
conv2d_33 (Conv2D)           (None, 582, 410, 32)      9248
_________________________________________________________________
up_sampling2d_13 (UpSampling (None, 1164, 820, 32)     0
_________________________________________________________________
conv2d_34 (Conv2D)           (None, 1162, 818, 1)      289
===============================================================

How can I have the same input and output shape?

If you look at Keras' output, there are various steps which lose pixels:
Max pooling on odd sizes will always lose one pixel. Conv2D using 3x3 kernels will also lose 2pixels, although I'm puzzled that it doesn't seem to happen in the downsampling steps.
Intuitively, padding the original images with enough border pixels to compensate for the pixel loss due to the various layers would be the simplest solution. At the moment I can't calculate how much it should be, but I suspect rounding up to a multiple of 4 should take care of the max pooling layers. For denoising, borders could be just copied from the outermost pixels, probably with some sort of low pass filtering to avoid artefacts.

See if the solution above makes sense to you.

Yes this makes sense. It will probably introduce a very slight scaling but that should barely be noticeable if at all. Have you tried it on sample pictures with satisfying results?

I don't know if this is the right way of doing it but I solved the problem.
Following the code from above I've added:
img_size = K.int_shape(input_img)[1:]

resized_image_tensor = tf.image.resize(decoded, list(img_size[:2]))****

autoencoder = Model(input_img, resized_image_tensor)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

I used tf.image.resize to synchronize the shape of reconstructed image and input image.
Hope it helps.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is there an algorithm to calculate the weights of an ontology tree's inner nodes?
I have a tree that represents a hierarchical ontology of computer science topics (such as AI, data mining, IR, etc). Each node is a topic, and its child nodes are its sub-topics. Leaf nodes are weighted based on their occurrence in a given document.
Is there a well-known algorithm or function to calculate the weight of inner nodes based on the weights of leaf nodes? Or is it totally based on the application to decide mathematical calculation of the weights?
In my application, the node's weights should be some sort of accumulation of its child nodes weights. Is there a better mathematical formula or function to do that than just summing up weights of child nodes?
I am not asking about traversal, but rather about weighting function.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is some kind of dropout used in the human brain?
I've read that ANNs are based on how the human brain works. Now, I am reading about dropout.
Is some kind of dropout used in the human brain? Can we say that the ability to forget is some kind of dropout?

Interesting question !

The human brain works by having neurons constantly fire at different rates. So, if the firing rate increases, the neuron is transmitting overly exciting or calming information to further neurons connected to it. How other neurons connected to the former neuron respond on the messages sent by it, depends on the strength of the connection between the connected neurons.
However, having true dropout in the brain would be like having a neuron stop firing entirely for some time. As far as I know, this is not gonna happen unless a neuron dies. So, no, there doesn't seem to be any real biological equivalent to dropout in the brain. If you forget something, that is just caused by a weakening of some connections in the brain. But such a weakening (or strengthening alternatively) is a gradual process and is a function of the degree of firing synchrony between any two connected nodes.
It is true that the introductory texts to Neural Nets tell you about those nets' neuro-scientific inspiration, but this similarity holds only in the most abstract way. Yes, you have neurons in both biological and artificial neural networks. But communication in the brain works in an asynchronous, continuous bio-chemical way, while communication in an artificial neural net works by taking the numeric state of one node, scaling it by the weight of the connection between two nodes, and then adding the weighted contribution of the previous node to the activation of the receiving node. So, already the fundamentals are very different between the two approaches (with the biological variant being much more complex and elaborate).
If you want to find something that possesses more biologically inspired properties, look at how Convolutional Neural Networks are theoretically inspired by the working of the (human) visual system. Besides that, dropout is a purely mechanical strategy to regularize a model well, in order to enhance the model's generalization ability, i.e. making the model work well also on unseen data. It's just part of an algorithmic optimization procedure.

so when a physical neuron fires at minimal rate, like zero output, it's dropout?

No, you can think of a physical neuron like a switch or feature detector. If enough activation has accumulated in the neuron (as received from previous neurons), some feature has been detected and it changes into the 'active'- or 'on'-state (increasing its firing rate). But if it has received too little activation to overcome its firing threshold, that merely indicates the absence of the feature the neuron is supposed to detect. Dropout is like forcefully preventing a neuron from firing in the presence of its feature. That's just not happening in the brain.


text:
How is the performance of a CNN trained with monochrome images on image recognition tasks degraded?
For CNN image recognition tasks, like object recognition/face recognition/object segmentation/posture recognition, are there experiment results about how much will the performance be degraded with monochrome images?
The imaginary experiment is like:

Take the existing frameworks, reduce the channel number in the framework to fit monochrome images

Transform the existing training data and testing data to monochrome images

Train the model with monochrome training data.

Use the model to test the monochrome testing data.

Compare the result with the original result.

By "Compare the result with the original result", you assume that another CNN has been trained with the original images, so that we can compare it with the CNN trained with monochrome images, right?

@nbro Yes, everything begin with an existing successful CNN model for color image.


text:
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Spinning Up by Open Ai.
Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html

To know the difference, I would write out the algorithms or go through implementations in detail.


text:
Does the off-policy evaluation work for non-stationary policies?
As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?
For example, IS (importance sampling)-based estimators, such as weighted IS or doubly robust, are still unbiased when they are used to evaluate UCB1, which is a non-stationary policy, as it chooses an action based on the history of rewards?

Weighted IS is not unbiased - it is initially biased towards evaluating the behaviour policy, and that bias is reduced as more samples are taken. So it cannot be "still unbiased" in other circumstances. Also, UCB1 as an example behaviour policy has other problems (such as how to define the policy in stochastic terms). In addition, what does it even mean to "evaluate a non-stationary policy"? By definition, the value changes. Please clarify: Is your question about evaluation or control scenarios (you would generally only use UCB1 in control), and is it specifically about UCB1?


text:
Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning
Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an integer number) shows the time between two arrivals, should I update Q-functions as follows:
$Q(s,a) = Q(s,a)+\alpha.(R+\gamma^{\tau} \max_{b \in A}Q(s^{\prime},b)-Q(s,a))$
And, to compare different algorithms, total rewards ($TR=R_{1}+ R_2+R_{3}+...+R_{T}$) is used.
What measure should be used in the SMDP setting? I would be thankful if someone can explain the Q-Learning algorithm for the SMDP problem with this setting.
Moreover, I am wondering when Q-functions are updated. For example, if a customer enters our website and purchases a product, we want to update the Q-functions. Suppose that the planning horizon (state $S_{0}$) starts at 10:00 am, and the first customer enters at 10:02 am, and we sell a product and gain $R_1$ and the state will be $S_1$. The next customer enters at 10:04 am, and buy a product, and gain reward $R_2$ (state $S_{2}$). In this situation, should we wait until 10:02 to update the Q-function for state $S_0$?
Is the following formula correct?
$$V(S_0)= R_1 \gamma^2+  \gamma^2V(S_1)$$
In this case, if I discretize the time horizon to 1-minute intervals, the problem will be a regular MDP problem. Should I update Q-functions when no customer enters in a time interval (reward =0)?

Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it as a normal MDP again, but with some extra domain knowledge that you can exploit primarily for computational efficiency:

Only at time steps that really correspond to "events" in your SMDP can you observe non-zero rewards; at all other time steps you just get rewards equal to $0$.
Only at time steps that really correspond to "events" in your SMDP do you have an action space greater than $1$; in all the "fake" time steps, you have no agency, you just have a single action available (say, a "dummy" or "null" action). So all of these "fake" time steps do not contribute in any way to the "credit assignment" problem in RL, and you can kind of ignore them in your learning steps; only the time spent in them can still be important for discount factors $\gamma < 1$.

If $\tau$ (real number not an integer number) shows the time between two arrivals, should I update Q-functions as follows:

Yes, an update rule like that looks correct to me. Let's take an example situation, where $\tau = 2.0$, and instead of using the update rule you suggest, we take the "proper" approach of discretising into smaller time steps, and treating it as a regular MDP. In this simple example case, it is sufficient to discretise by taking time steps that correspond to durations of $1.0$.
In the SMDP, we'll have only a single transition $s_0 \rightarrow s_2$ (it will become clear why I use slightly strange time-indexing here soon), after which we observe a reward, and this transition takes time $\tau = 2.0$. In the corresponding MDP, we'll have two state transitions; $s_0 \rightarrow s_1$, and $s_1 \rightarrow s_2$, with two reward observations $R_1$ and $R_2$, where we know for sure that:

$R_2 = 0$ (because it does not actually correspond to any event in the SMDP)
We have a meaningful choice between multiple actions at $s_0$, each of which can have different transition probabilities for taking us into different "dummy" states $s_1$, and yield possibly-non-zero rewards $R_1$. In the dummy state $s_1$, we'll always only have the choice for a single dummy/null action (because this state does not correspond to any event in the SMDP), which always yields $R_2 = 0$ as mentioned above.

So, the correct update rule for $s_1$ where we picked a forced dummy action $\varnothing$ and are doomed to receive a reward $R_2 = 0$, would be:
$$Q(s_1, \varnothing) \gets Q(s_1, \varnothing) + \alpha \left( 0 + \gamma \max_{a'} Q(s_2, a') - Q(s_1, \varnothing) \right)$$
and the correct update rule for $s_0$, where we picked a meaningful action $a_0$ and may get a non-zero reward $R_1$, would be:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \max_{a'} Q(s_1, a') - Q(s_0, a_0) \right)$$
In this last update rule, we know that $s_1$ is a dummy state in which the dummy action $\varnothing$ is the only legal action. So, we can get rid of the $\max$ operator there and simplify it to:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma Q(s_1, \varnothing) - Q(s_0, a_0) \right)$$
Since we know that $s_1$ is a dummy state in which we are never able to make meaningful choices anyway, it seems a bit wasteful to actually keep track of $Q(s_1, \varnothing)$ values for it. Luckily, we can easily express $Q(s_1, \varnothing)$ directly in terms of $Q(s2, \cdot)$ -- which is exactly the next set of $Q$-values that we would be interested in keeping track of again:
$$Q(s_1, \varnothing) = \mathbb{E} \left[ 0 + \gamma \max_{a'} Q(s_2, a') \right]$$
So if we want to skip learning $Q$-values for $s_1$ (because it's kind of a waste of effort), we can just use this definition and plug it straight into the update rule for $Q(s_0, a_0)$. $Q$-learning is inherently an algorithm that just concrete samples of experience to estimate expectations (and this is a major reason why it typically uses learning rates $\alpha < 1.0$, so we can simply get rid of the expectation operator when doing this:
$$Q(s_0, a_0) \gets Q(s_0, a_0) + \alpha \left( R_1 + \gamma \left[ \gamma \max_{a'} Q(s_2, a') \right] - Q(s_0, a_0) \right)$$
and this is basically the update rule that you suggested. Note; here I assumed that you receive your rewards directly when you do take actions in your SMDP, which is why I had $R_1$ as a possibly-non-zero reward, and always $R_2 = 0$. I suppose you could also in some cases envision SMDPs where the reward only arrives on the next SMDP-time-step, and that the amount of time that ends up being elpased in between the two events is important to take into account via the discount factor $\gamma$. So you could also choose to model a problem where $R_1 = 0$ and $R_2$ may be non-zero, and this would yield a different update rule (I think one where the reward gets multiplied by $\gamma^{\tau - 1}$? not sure, would have to go through the steps again).

What measure should be used in the SMDP setting? I would be thankful if someone can explain the Q-Learning algorithm for the SMDP problem with this setting.

I think it would be important to involve the amount of time that you take somehow in your evaluation criterion. You could run episodes for a fixed amount of time, and then just evaluate agents based on the sum of rewards. If you don't run for a fixed amount of time (but instead for a fixed number of steps, each of which may take variable amounts of time, for example), you would probably instead want to evaluate agents based on the average rewards per unit of time. You could also include discount factors in your evaluation if you want, but probably don't have to.

Moreover, I am wondering when Q-functions are updated. For example, if a customer enters our website and purchases a product, we want to update the Q-functions. Suppose that the planning horizon (state $S_0$) starts at 10:00 am, and the first customer enters at 10:02 am, and we sell a product and gain $R_1$ and the state will be $S_1$. The next customer enters at 10:04 am, and buy a product, and gain reward $R_2$ (state $S_2$). In this situation, should we wait until 10:02 to update the Q-function for state $S_0$?

This depends on you state representation, and how you model a "state", and to what extent previous actions have influence the state you end up in. Keep in mind that the update rule for $Q(S_0)$ also requires for $S_1$ (or even $S_2$ if $S_1$ is a "dummy state" that you skip) to have been observed. So, if your state representation includes some features describing the "current customer" for which you want to pick an action (do you offer them a discount or not, for example?), then you can only update the $Q$-value for the previous customer when the next customer has arrived. This model does assume that your previous actions have some level of influence over the future states that you may end up in though. For example, you might assume that if your actions make the first customer very happy, you get a better reputation and are therefore more likely to end up in future states where other customers visit more frequently.


text:
In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be.

Is it a scalar that I have to use to update all the weights (so all weights are updated with the same value)? Or is it a list with a different values to use for updating for each weight? I'm used to working with loss functions and an $y$ target, but here I don't have them so I'm quite confused.

Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.

So to have those entries I have to compute the jacobian?

Yeah, basically.


text:
Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case?
I am a bit confused about the off-policy case (right figure). Why does the right figure represent the off-policy actor-critic methods?


text:
What introductory books to reinforcement learning do you know, and how do they approach this topic?
Currently, I'm only going through these two books

Reinforcement Learning: An Introduction, by Sutton and Barto: RL explained on an engineering level (mathematical, but readable for a non-mathematician). Elementary notions from probability and statistics are required (conditional probability, total probability theorem, total expectation theorem, and similar. The MIT RES.6-012 "Introduction to Probability" course is a great source of information for these topics.).

Deep Reinforcement Learning, by Miguel Morales: this book introduces the main elements of reinforcement learning in a less formal way than Sutton and Barto (derivations for some equations are not given), using examples to describe the math.

What other introductory books to reinforcement learning do you know, and how do they approach this topic?

I expect a good answer to provide links to other books but also a brief description of the book and how they treat the subject.

In addition to the ones you mentioned, I would add Algorithms of Reinforcement Learning by Csaba Szepesvári. There is a number of professors who use it as a reference in their RL teaching materials (for example this one).
It generally follows the same outline as Sutton & Barto's book (except the part on bandits, it is included in the Chapter on Control). In fact, it may be considered as a condensed version of Sutton & Barto (about 100 pages). In addition, it's freely available online.
I like the author's justification as to why he wrote this book, so I'm just going to quote it:

Why did I write this book? Good question! There exist a good number of
really great books on Reinforcement Learning. So why a new book? I had
selfish reasons: I wanted a short book, which nevertheless contained
the major ideas underlying state-of-the-art RL algorithms (back in
2010), a discussion of their relative strengths and weaknesses, with
hints on what is known (and not known, but would be good to know)
about these algorithms.

The (draft) book Reinforcement Learning: Theory and Algorithms, by Sham M. Kakade (who published a natural policy gradient algorithm and other important research) and others, introduces RL in a mathematical/formal way. It seems to me that this is a reliable book, but a bit advanced for "regular people". Yes, I know the question was about introductory books on RL, but this may be suitable for people that have a solid knowledge of math and would like a hardcore intro to RL. For example, the book starts with a non-trivial (in my view) proof that there exists an optimal stationary and deterministic policy for an MDP.

Foundations of Deep Reinforcement Learning: Theory and Practice in Python (Addison-Wesley Data & Analytics Series) 1st Edition
This book does not give a detailed background information on Markov Decision Processes, different Bellman equations and relationships between the value function and action-value function, etc. It focuses on Deep Reinforcement Learning and goes straight to Policy and Value - based algorithms using neural networks. It might be good for someone trying to quickly understand what Deep RL algorithms are out there and apply them.


text:
In layman's terms, what is stochastic computation graph?
I'm going through the distributions package on PyTorch's documentation and came across the term stochastic computation graph. In layman's terms, what is it?

Hi and welcome! It may be useful if you link us to the documentation that uses that term/expression.

Hi, the documentation is here: https://pytorch.org/docs/stable/distributions.html ; its detailed in https://arxiv.org/pdf/1506.05254.pdf , but I was looking for an intuitive comment on how its integrated by libraries like Tensorflow or Pytorch. Thank you!

Although I haven't yet read that paper (I will try to do it so that to provide a formal answer), I think that a stochastic computation graph (SCG) is just a computation graph that is able to handle stochastic or random operations, such as sampling from a normal distribution. Here, by "handle" I mean that it handles the back-propagation part of training a neural network that uses random sampling (e.g. VAEs). But, to be honest, this is just an educated guess given my experience with TensorFlow Probability and VAEs and after having read that introductory section of the documentation you linked.

Just to understand how much you know about machine learning, do you know what a computation graph is? Are you familiar with variational auto-encoders or similar models that use random sampling?


text:
Is there any application of topology to deep learning?
Is there any application of topology (as in math discipline) to deep learning? If so, what are some examples?

See A Topology Layer for Machine Learning by Rickard Brüel Gabrielsson.


text:
Is there a good book or paper on word embeddings?
Is there a good and modern book that focuses on word embeddings and their applications? It would also be ok to provide the name of a paper that provides a good overview of word embeddings.


text:
Can operations like convolution and pooling be discovered with a neural architecture search approach?
From Neural Architecture Search: A Survey, first published in 2018:

Moreover, common search spaces are also based on predefined building
  blocks, such as different kinds of convolutions and pooling, but do
  not allow identifying novel building blocks on this level; going
  beyond this limitation might substantially increase the power of NAS.

Has anyone tried that? If not, do you have any thoughts about the feasibility of this idea?


text:
How can I design a system that suggests physical exercises to a person while keeping into account the fatigue?
I want to create an exercise suggester. Each day either has a routine or is a rest day. A routine has 4 slots. For each slot we select an exercise. We constrain the legal exercises, only do upper-body today, etc. We, therefore, restrict the number of available exercises. This seems easy.
I want to know how can I extend this model to take fatigue into account. Fatigue is a state such that every exercise reduces it (specific value for each exercise) and it recovers with time.
Can this problem even be modeled as a constraint satisfaction problem? What's the best way of modeling and solving this problem
I'd like to suggest all the legal exercises taking fatigue into account.


text:
Would a different learning rate for every neuron and layer mitigate or solve the vanishing gradient problem?
I'm interested in using the sigmoid (or tanh) activation function instead of RELU. I'm aware of RELU advantages on faster computation and no vanishing gradient problem. But about vanishing gradient, the main problem is about the backpropagation algorithm going to zero quickly if using sigmoid or tanh. So I would like to try to compensate this effect that affects deep layers with a variable learning rate for every layer, increasing the coefficient every time you go a layer deeper to compensate the vanishing gradient.
I have read about adaptive learning rate, but it seems to refer to a technique to change the learning rate on every epoch, I'm looking for a different learning rate for every layer, into any epoch.

Based on your experience, do you think that is a good effort to try?

Do you know some libraries I can use that already let you define the learning rate as a function and not a constant?

If such function exists, it will be better to define a simple function lr=(a*n)*0.001 where n is layer number, and a a multiplier based on experience, of we will need the inverse of the activation function to compensate enough the gradient vanishing?

I see two possibilities: (a) copy and modify an existing optimizer like SGD or Adam to do what you want, (b) use SGD with lr=1 and multiply the gradients of each layer by whatever learning rate you want (this is mathematically equivalent to what you want).

I believe you might run into issues related to errors inherent in numerical computation. That is, the calculation of the gradient for these "compensated" layers might become increasingly inaccurate and prevent learning from occurring. This would be mostly true in deep networks because the compensated learning rate might result in very large numbers multiplied by very small numbers (a known challenge in numerical computation). It would be interesting if someone had insights into this.

Alternative idea: normalize each layer's gradient and scale by normalized gradient multiplier ngm, which is somewhat similar to learning rate.


text:
Is it a good idea to change the learning rate at each training step as a function of the loss?
Is it a good idea to change the learning rate at each training step as a function of the loss? i.e. for points with high loss value, put a high learning rate and for low loss value a low learning rate (using a tailored function)?
I know that the update of the parameters is done via $\gamma \nabla L$, where $\nabla L$ is the gradient and $\gamma$ the learning rate, and that points with high loss should correspond to a high gradient. Hence the dependency of the update of the parameters on the value of the loss should be already contained, although in a more indirect way. Is doing what I propose dangerous and/or useless?

Have you ever heard of adaptive learning rate? I think that's what you are looking for.

Yes, but I've never read of a learning rate scheduler that updates the learning rate explicitly depending on the value of the loss at each step. Usually, for example, you halven the learning rate at a plateau, or something similar. I want to change the learning rate at each step

When you adjust your learning rate based on the loss, you have more chance to get stuck in local minima more often. It could be more efficient, but in general, I don't feel it has a positive effect in more complex situations.

https://machinelearningmastery.com/learning-rate-for-deep-learning-neural-networks/ maybe this is a good resource to go to.

Why you think you have more chances of getting a local minima? experience?

@JuliaOrtali Yes, but also from a theoretical point of view perhaps. I'm not saying it is always the case, but many times, losses around the minima are quite small, so the result will be that you keep taking smaller steps and eventually will never escape them. I do think that adapting your learning rate in a 'smarter' way will help. That's why optimizers like Adam and RMSprops are used more often instead of GD.So I guess that when adapting your learning rate for better outcomes will eventually lead you to an Adam optimizer.

its an interesting idea, but i dont see an obvious implementation. these very clever variations of vanilla backprop usually involve an "obvious formulation", such as SGD's loss function, or the update value for Newton's method of optimization. i dont see an obvious way of changing the learning rate asides from what is already implicit in the standard formulation, but it can be open to formulation

but as always, perhaps the best way to test this hypothesis is by testing it on somthing like MNIST


text:
How does YOLO handle non-class objects?
I have been reading more about computer vision and I'm bothered by YOLO and similar deep learning architectures.
The thing I am confused about is how non-class image sections are dealt with. In particular, it's not clear to me at all why YOLO doesn't consider every part of an image a possible class.
What actually sets the cutoff for detection and then classification?

The output of YOLO is (x,y,w,h,confidence,class). The confidence value presents whether the rectangle holds an object, the rectangle is non-classed when confidence is low.
The class value will be used, only when confidence is high.

As of now, it is not possible to detect the non-class object. You have to train that non-class object again then the model will detect.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Should I use the discounted average reward as objective in a finite-horizon problem?
I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a total of $T$ maximally possible time steps (e.g., the usage rate of an app in each time-step), in each time-step, the reward may be 0 or 1. The goal is to maximize the daily average usage rate.
Episode length ($T$) is maximally 10. $T$ is the maximum time window the product can observe about a user's behavior of the chosen data. There is an indicator value in the data indicating whether an episode terminates. From the data, it is offline learning, so in each episode, $T$ is given in the data. As long as an episode doesn't terminate, there is a reward of $\{0, 1\}$ in each time-step.
I heard if I use an average reward for the finite horizon, the optimal policy is no longer a stationary policy, and optimal $Q$ function depends on time. I am wondering why this is the case.
I see normally, the objective is defined maximizing
$$\sum_t^T \gamma^t r_t$$
And I am considering two types of average reward definition.

$1/T(\sum^_{t=0}\gamma^t r_t)$, $T$ varies is in each episode.

$1/(T-t)\sum^T_{i=t-1}\gamma^i r_i$

How does your horizon relate to episode length (or is the target environment continuous)? What controls value of $T$? Is the value of $T$ known at $t=0$? What do you mean by "iteration" - is that {0, 1} reward per time step, or is it {0, 1} reward per $t=0$ to $T$ pseudo-episode?

@NeilSlater i added the edits

Thanks for the edit. So, to be clear, $T$ is not actually the length of an "episode", but the size of an observation window into a continuous process? There is essentially no natural episode, just observations, and some of them form into small trajectories where state progresses due to actions taken?

T can be thought of as episode since after maximum of 10 time steps, users will go to another category of the app, which is not what we care about. And yes, it can thought of as no natural episode; state progresses due to actions taken

For "I heard if I use an average reward for the finite horizon" I would be interested to know where you heard/read this from, as on the surface it looks incorrect to me for your problem (i.e. I agree with you). There may be some context to it that makes it true for some problems and not others, so I would like to see. Do you have a link?

i asked a person doing RL research, she said "In finite (small) horizon , the optimal policy is no longer a stationary policy, and optimal Q Function depends on time." but i did not get the reason.

I'd suspect she misunderstood the context of what you were asking about. Or you have misremembered. Seems wrong to me. Are you in a position to ask the researcher for clarification or a reason?

she emailed me this sentence when I asked if it is ok to use average reward as objective for q-learning and she said the following. If you think it does not matter to use q-learning in average reward setting, can you also kindly tell me why

I suggest you ask the researcher for clarification, otherwise it is not possible to fairly critique the statement. She may have had in mind something else based on your description, and that makes it a miscommunication. The statement "it does not matter to use q-learning in average reward setting" is missing too much context to answer for me, and possibly the researcher was faced with similar missing details and made an assumption about what you were doing.


text:
How can I derive n-step off-policy temporal difference formula?
I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as
$$V(S_t) = V(S_{t-1}) + \alpha \rho_{t:t+n-1}[G_{t:t+n} - V(S_{t-1})],$$
where $V(S_{t})$ is state value function of the state $S$ at time $t$ and $ G_{t:t+n} \doteq \sum_{i=t}^{t+n-1}\gamma^{i-t}R_{i+1} + \gamma^n V(S_{t+n})$ and $\rho_{t:t+n-1}$ is the importance sampling ratio.
I tried to prove this equation for $n = 1$ using the incremental update of the value function. Now I end up with this formula:
$$V(S_t) = \frac{1}{t} \sum_{j=1}^{t} \rho_{j}G_{j} $$
$$V(S_t)= \frac{1}{t}(\rho_{t}G_{t} + \sum_{j=1}^{t-1}\rho_{j}G_{j}) $$
$$V(S_t) = \frac{1}{t}(\rho_t G_t + (t-1)V(S_{t-1}))$$
$$V(S_t)=V(S_{t-1}) + \frac{1}{t}(\rho_{t}G_{t} - V(S_{t-1}))$$
I know I'm wrong because this does not match with the above equation. But can anyone please show me where I am wrong?

Do you remember which version of the book you were reading at the time?

It was a draft version


text:
Given the daily stock prices of the last 3 years, how should I sample the training data for episodic RL?
I am playing around with a stock trading agent trained via (deep) reinforcement learning, including memory replay. The agent is trained for 1000 episodes, where each episode consists of 180 timesteps (e.g. daily stock prices).
My question is concerning the sampling of episodes for training.
Assuming I've got daily stock prices going back 3 years, that's about 750 trading days/prices.
How should I sample this data set to get enough episodes for training?
With an episode length of 180 and an episode count of 1000, I'd need 180k "days" to choose from, if I wouldn't want any duplication.
Do I even need to sample 1000 non-overlapping windows from my dataset or can I sample my episodes using a sliding window approach? Could I even just randomly sample the dataset for episodes? For example, calculate a random date and build the episode from the 180 days following that random starting date?
The reward for each action is calculated as follows, p are the prices and t is the current timestep of the episode.

CASH: 0
BUY:  p(t+1) - p(t) - fee
HOLD: p(t+1) - p(t)

Are you want to predict the stock price using RL?

No, the model predicts one out of three actions (open/close/hold).

Why don't use the sliding window and iterative the whole data over and over. You can also go to investing.com and find more training data.

That is the question, whether it is "ok" to use a sliding window or not. I can't get more data for stocks which have only been traded for a few years ;)

I may be wrong about your problem, but I think you are trying to make action (open/clsoe/hold) based on previous days stock price. If this is the case the next state is independent of our current action if you take sliding window aproach. How you formulated the MDP? I think it's a contextual bandit problem for the sliding window approach. Same goes to other methods.

Yes the states should be independent of the action taken & from each other. I've edited the question to clarify how the reward for each action is calculated.

what is the fee?

Make the state as follows $s_{t} = (p_{t},p_{t-1},,...,p_{t-n}, cash \ in \ hand)$, then it will be a complete RL problem.

The fee models the transaction cost of buying a stock, e.g. 0.5% of the trade price.

Let us continue this discussion in chat.


text:
What is the representational capacity of a learning algorithm?
The definition I see for representational capacity is "the family of functions the learning algorithm can choose from when varying the parameters in order to reduce a training objective." (Goodfellow's Deep learning book).
However, to me this seems to be the same as the definition of the hypothesis space. Is the key difference the "in order to reduce a training objective" in that some functions may not be chosen in reducing a training objective? Or are these identical definitions.

I don't think there's any difference between representational capacity and hypothesis space.

Learning algorithms (some others too, like search) aim at generating functions that get as close as possible to the "shape" of the training data (so we can then feed values to the generated functions and get outputs like, say, a prediction).
In 2D, the "shape" may be easy to visualize. If the data in 2D seems to line up, learning algorithms generating linear/affine functions (e.g. y = ax + b), should fit fairly well. Their representational capacity extends to lines. If the data seems to form a parabola, a representational capacity bound to lines will do poorly. We then need more "capable" representations, which can cope with the quadratic terms.
So this should be what the "family of functions" Goodfellow's book refers to. The "training objective" should not be in the definition (to me), as a line fitting solution is unable to represent, say, a parabola, whatever the training objective is. However the book definition may mean a line can be fit to a parabola, strictly speaking, although it will do so very poorly.
The hypothesis space resembles the definition, perhaps (I have not checked it), but what matters is the "mindset". I would draw a parallel with prior and posterior distributions in Statistics, where we usually make some hypothesis on the shape of the distribution.
Concrete examples like linear regressions generate linear functions, as hinted by their names. Decision trees generate sets of linear functions, thus more complex and "capable". SVM generates functions depending on its kernel. The RBF kernel, for example, allows generating functions that can map quite complex, non-linear data "shapes". And arbitrary neural networks can map "arbitrary" data "shape" (easier written than actually done).

Could you also clarify how this differs from the hypothesis space of the learning algorithm?


text:
Can you convert a MDP problem to a Contextual Multi-Arm Bandits problem?
I'm trying to get a better understanding of Multi-Arm Bandits, Contextual Multi-Arm Bandits and Markov Decision Process.
Basically, Multi-Arm Bandits is a special case of Contextual Multi-Arm Bandits where there is no state(features/context). And Contextual Multi-Arm Bandits is a special case of Markov Decision Process, where there is only one state (features, but no transitions).
However, since MDP has Markov property, I wonder if every MDP problem can also be converted into a Contextual Multi-Arm Bandits problem, if we simply treat each state as a different input context (features)?

The main difference between an MDP and contextual bandit setting is time steps and state progression. If those are important to the problem you want to solve, then it is not possible to convert.
Essentially MDPs are a strict generalisation of contextual bandits. You can model a CB as an MDP but not vice-versa.
In some very specific cases you can convert MDP to CB - any one of these situations means that the MDP can be simplified to CB, and then you could use bandit solving algorithms to optimise it:

When there is only one time step for an episodic problem.

When the discount factor is zero.

When state transition rules are completely independent of action choice, but reward is not.

Thanks for your reply. If you have criteria #2 and #3, does #1 even matter any more? - I think that basically makes it a non-episodic problem? - Also MAB actually can have a non-zero discount factor.

It is any one of the criteria. The discount factor in MAB is not the same thing as in MDP


text:
Learning only using off-policy samples
When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a bunch of episodes using randomly sampled actions (or maybe by following an old policy) one time, and train multiple policies using this set of samples to find the most effective hyperparameters? Every time we train a new policy, does it make sense to replay all the episodes generated by the previous policy? I'm mostly interested in actor-critic methods.

What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy.
If you're interested in actor-critic methods then a popular off-policy method is the Deep Deterministic Policy Gradient.

authors: Ванек Огонек


text:
How should I choose the target's update frequency in DQN?
I have been dealing with a problem that I'm trying to solve with DQN. A general question that I have is regarding the target's update frequency. How should it change? Depending on what factor do we increase or decrease this hyperparameter?

As you said yourself, it is a hyperparameter. Hence, no one (even you) can say what is the ideal update frequency. You have to test and try.
Having said that, remember one thing the target NN should mimic the actual network as closely as possible. Hence if you update it after a long number runs, then I think you will start losing the accuracy. On the contrary, if you update it too often, then you lose the benefit of using the target network (which is to boost the training rate and reduce training time) and the training will take a larger amount of time.
My suggestion is to try updating after every 5 to 8 episodes.


text:
What do we mean by 'principal angle between subspaces'?
I came across the term 'principal angle between subspaces' as a tool for comparing objects in images. All material that I found on the internet seems to deal with this idea in a highly mathematical way and I couldn't understand the real physical meaning behind the term.
I have some knowledge of linear algebra. Any help to understand the physical significance of this term and its application in object recognition would be appreciated.

This is not a new question, but it would be nice if you can provide a link to the article or paper where you found this term/expression, so that to have specific context.

Let's consider the case where you have two photos, one base photo and one other photo which is a scaled version of the base photo. Consider then that you could create a 'mapping' from the base photo to the scaled photo as defined by a set of vector changes for each pixel in the base photo.
That is to say, if you had a pixel at point (0,0) in the base photo it would be in position (0,0) on the scaled photo. If you had a pixel at point (0,1) on the base photo, it would be at point (0,s) on the scaled photo, where s is the scale factor between them. The same would be true for (1,0) mapping to (s,0) and (1,1) mapping to (s,s).
We can understand each of these pixels as a subspace on the vector space of "all possible pixels" (possibly with an included vector for the 'value' of each pixel as a third+ dimension, based on how it's represented). Note though, that they are different subspaces. There are vectors that exist in the scaled picture for scale factors greater than 1 which are not in the original picture. For scale values less than one, there are vectors which are in the original picture which are not in the scaled picture. For scale vector equal to one, it's basically the identity and they're the same picture.
What's more, we can do the same for rotation. If you have base photo and rotated photo, then there is a mapping from one pixel on the base photo to one pixel on the rotated photo, I'll leave the math up to you. The same is further true for translating, skewing, and even changing the colors of the image. Each one denotes a type of mapping that you could make from one subspace to another.
So what does this mean? Well, one thing that you could do is take two images, one "base" photo and two "scaled" photos and determine "how different are these two photos from the base photo"? That is, if the first scaled photo has a scaling factor of 10 and the second has a scaling factor of 100, then you could say that the larger scaled-photo is "further away" from the base photo. That is, we can compare the two mappings together to determine which mapping is closer to the original.
What's more, these mapping functions are often composable and commutative. That is if you do a "scale-then-rotate", that is very similar if not exactly identical to doing "rotate-then-scale". Let's take a look at why this is. Let's consider that we take a unit vector from either one. We then apply "scale" which scales both the x and y components, and then rotate this unit vector about some axis (let's assume the origin, though maybe there's a translation that happens as well). You will then yield a new vector which represents all of these applied changes. What's more, we can do this to both of the "scale-then-rotate" function and the "rotate-then-scale" function.
Now we have two vectors, and we want to determine if they are the same. Well, one way to do that is to calculate the angle between two vectors. If the angle is 0, then we can say that the vectors are the same (with the assumption that they are also of the same magnitude). In this case, both would yield the same vector, the angle between them is 0, so we can say they are the same.
So then how do we get back to the question of "how different are these two objects"? Well, one way that we could do it is to attempt to answer the question "What set of composed functions maps from one object onto the other one?". If we can create a new "vector" which is the representation of the full set of changes from one picture to the other, we can then attempt to determine how different these two objects are, as represented in the dimensional spaces of their image.
But the question that you may then ask is "why is this useful"? Well, imagine that you are a neural network. You've been given a picture and tasked with answering the question "Is this a cat?". How would you go about answering this? Well, one thing as a neural net that you've learned is "cats have ears". You have a canonical representation of a "cat ear" encoded in the weights of your neural network, so you go about looking at different subsets of the picture to say "is this a cat ear"? You can do this, for example, by taking a cropped section of the picture, determining the angle and dimensions apart that cropped section is from your canonical "cat ear" is, and if the angle between your canonical "cat ear" and your sample is below some amount, you can say with confidence that what you have found is a "cat ear". A couple cat ears, eyes, a nose, some fur, and a tail (though maybe the tail is occluded), and you can be confident that you have found a "cat".
(Note: This is an oversimplification and this is usually done in deep networks by breaking pictures out into a continually composing series of features, for instance first "lines" vs "curves" vs "textures", which then yield "circles" vs "rectangles" vs "cross-hashes", which then yield "eyes" vs "fur" vs "skin", etc. Each different layer could be implemented with its own comparison in this same method though!)

So this is where some of the analogy may get stretched. In something like image analysis, the "dimensions" of an image can literally be the number of pixels, so scaling an image may change the literal "dimensions" if each pixel position is a dimension being analyzed (as opposed to something like the length vs height, which we often think of as the "dimensions" of an image). As an example, a black and white image of the same size has reduced dimensionality from a color image. This is how I've always understood it when studying CNNs, but I may be wrong. The terminology there was a bit different.

I realize that about length and width, I was clarifying that's not what I was talking about. As an example, a given convolutional layer may create a set of non-linear inputs from a sampling of a subset of pixels or color channels of a given image, creating a new matrix based on the values created. So if one layer is looking at base pixels, the next layer is looking at "textures". The dimensionality of the second layer is only tangentially related to the first, though is often lower. Scaling was just one example transformation that is easily understood. Feel free to add your own answer though!

So, I can link you to a couple of mathematical definitions, but so does the OP and that wasn't the question. To quote, "I couldn't understand the real physical meaning behind the term." I was trying to provide an intuitive understanding as to the use of the term and why it is valuable physically. These analogies can sometimes be stretched, but can also be useful. If you have a better answer, provide it and maybe I can learn something too! I'm excited to see.


text:
Given two neural networks that compute two functions $f(x)$ and $g(x)$, how can I create a neural network that computes $f(x)g(x)$?
I have two functions $f(x)$ and $g(x)$, and each of them can be computed with a neural network $\phi_f$ and $\phi_g$.
My question is, how can I write a neural net for $f(x)g(x)$?
So, for example, if $g(x)$ is constant and equal to $c$  and $\phi_f = ((A_1,b_1),...(A_L,b_L))$, then $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$.
Actually, I need to show it for $f(x)=x$ and $g(x)=x^2$ if this make something easier.

Did you mean $\phi_{fg} = ((cA_1,cb_1),...,(cA_L,cb_L))$ rather than $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$?

no because only the output should be multiplied with c

In the title you write $f(x) * g(x)$ and in the body $f(x)g(x)$. Is that a convolution or multiplication? If $g(x)$ is constant then you would have $(cA_L, cb_L)$ only if the activation function of the final layer is linear so it's not true in general.

@Brale Sorry, it was me who rewrote the title. It's multiplication in all cases.

Use another network h which takes f(x) and g(x) as input i.e. h(f(x), g(x)).
Training psuedo code(pytorch):
for epoch in epochs:
    for batch, x in dataset:
        train(f)
        train(g)

for epoch in epochs:
    for batch, x in dataset:
        # freeze f and g (using torch.no_grad in pytorch)
        with torch.no_grad:
            fx = f(x)
            gx = g(x)
        pred = h(fx, gx)
        loss = loss_fn(pred, (fx . gx))
        backpropagate()
        optimise()

thank you for your answer but i dont want to know how to implement it but how to write it theoretically as a neural net

You need to express the product of the activation functions of your neurons via some combination of individual activation functions.
For example, if the product of your activation functions is equal to the linear combination of such functions then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ exactly.
Let's assume for simplicity that $f(x)$ and $g(x)$ are approximated with trained networks that consist of just one neuron,
$$f(x) = A(w_f x + b_f)$$
$$g(x) = A(w_g x + b_g)$$
Then, if the activation function $A$ is such that
$$A(w_f x + b_f)A(w_g x + b_g) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$
then $f(x)g(x)$ is approximated by a neural network consisting of $N$ neurons:
$$f(x)g(x) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$
This is easily generalized to a network that consists of multiple neurons.
An example is when some of your neurons have activation functions $A(z)=\cos(z)$ and others are $A(z)=\sin(z)$. Then you can build the network computing $f(x)g(x)$ from the networks computing $f(x)$ and $g(x)$ using the product-to-sum identities, like $2\sin(\theta)\sin(\phi) = cos(\theta-\phi)-cos(\theta+\phi)$, etc.
If the product of your activation functions cannot be expressed as a linear combination of such functions (as in the case of sigmoids) then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ approximately. You need to figure out the parameters of the approximation
$$A(z_f)A(z_g) \approx \sum\limits_{i=1}^N \alpha_iA(z_i)$$
Such parameters are: the number of terms, $N$, the coefficients $\alpha_i$, and the range $[z_{min}, z_{max}]$ where this approximation works with a desired accuracy.
Linear combination is not the only opportunity. Say, your activation is $A(z)=a^z$. Then, $$A(w_f x + b_f)A(w_g x + b_g)=a^{(w_f+w_g) x + (b_f+b_g)}=a^{w_{fg} x + b_{fg}}$$ so the weights and the biases of the product are exactly expressed via the weights and the biases of the multiplicands.


text:
Black Box Explanations: Using LIME and SHAP in python
Recently, I came across the paper Robust and Stable Black Box Explanations, which discusses a nice framework for global model-agnostic explanations.
I was thinking to recreate the experiments performed in the paper, but, unfortunately, the authors haven't provided the code. The summary of the experiments are:

use LIME, SHAP and MUSE as baseline models, and compute fidelity score on test data. (All the 3 datasets are used for classification problems)

since LIME and SHAP give local explanations, for a particular data point, the idea is to use K points from the training dataset, and create K explanations using LIME. LIME is supposed to return a local linear explanation. Now, for a new test data point, using the nearest point from K points used earlier and use the corresponding explanation to classify this new point.

measure the performance, using fidelity score (% of points for which $E(x) = B(x)$, where $E(x)$ is the explanation of the point and $B(x)$ is the classification of the point using the black box.

Now, the issue is, I am using LIME and SHAP packages in Python to achieve the results on baseline models.
However, I am not sure how I'll get a linear explanation for a point (one from the set K), and use it to classify a new test point in the neighborhood.
Every tutorial on YouTube and Medium discusses visualizing the explanation for a given point, but none talks about how to get the linear model itself and use it for newer points.

Hello. Although this is an old post, to me, it's not fully clear what your question is. Could you please edit it you clarify what it is and write it explicitly (in the title)?

For LIME, the local model that is trained can be found at
lime.lime_base.explain_instance_with_data under the name "easy_model".


text:
Why are policy iteration and value iteration studied as separate algorithms?
In Sutton and Barto's book about reinforcement learning, policy iteration and value iterations are presented as separate/different algorithms.
This is very confusing because policy iteration includes an update/change of value and value iteration includes a change in policy. They are the same thing, as also shown in the Generalized Policy Iteration method.
Why then, in many papers as well, they (i.e. policy and value iterations) are considered two separate update methods to reach an optimal policy?

Policy iteration is made up of two steps. The first is a full policy evaluation, where a value function is calculated for the current policy. The second is policy improvement, where the policy is made greedy with respect to the value function.
Value iteration looks to speed things up by stopping policy evaluation after one iteration, make the policy greedy with respect to that value function, and repeat until convergence.
Clearly, these are two different algorithms, hence why they are considered to be different. They are, however, very closely linked, which is why you might consider them to be 'the same thing'. I guess you could say they belong to the same family of algorithm.

Policy iteration is based on the insight that for a given policy, it is straightforward to compute the value function (the long-run expected discounted value of being in a given stage) exactly -- it is a set of linear equations at that point.  So, we update the policy, then calculate the exact values of the states for always following that particular policy, and based on that we update the policy again, etc.
Value iteration, in contrast, does not use that insight.  It just updates estimates of the values of being in the states one step at a time.  If these values are initialized at 0, you can think of this of the $i$th iteration computing the value of what would be the optimal policy if we knew the MDP would end after $i$ iterations.  We never really have to think explicitly about policies (though we are in effect computing a policy each iteration), and never directly calculate the infinite sum of expected discounted rewards.
These are just the vanilla variants and it is possible to mix and match these ideas -- e.g., you might not evaluate a policy by explicitly solving a system of linear equations but rather just do some iterations -- but the vanilla variants are clearly distinct.


text:
How to compute the target for double Q-learning update step?
I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 possible actions (1,2,3)):

For each experience $e_{j}=(s_{j},a_{j},r_{j},s_{j+1})$ of the mini-batch (consider an experience where $a_{j}=2$) I compute the output through the main network in the state $s_{j+1}$, so I obtain 3 values.

I look which of the three is the highest so: $a^*=arg\max_{a}Q(s_{j+1},a)$, let's say $a^*=1$

I use the target network to compute the value in $a^*=1$ , so  $Q_{target}(s_{j+1},1)$

I use the value at point 3 to substitute the value in the target vector associeted with the known action $a_{j}=2$, so: $Q_{target}(s_{j+1},2)\leftarrow r_{j}+\gamma Q_{target}(s_{j+1},1)$, while $Q_{target}(s_{j+1},1)$ and $Q_{target}(s_{j+1},3)$, which complete the target vector $y$, remain the same.

Is there anything wrong?

$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$
The only difference between the "original" DQN and this one is that you use your $Q_\text{est}$ with the next state to get your action (by choosing the action with the highest Q).
Afterward, you just figure out what the target $Q$ is given that action, by selecting the $Q$ belonging to that action from the target_network (instead of using the argmax a directly on the target Q network).
About the formula

$\theta_{t}^{-}$ above it means frozen weights, so it represents the target Q network.

the other $\theta_{t}$ represents the "learnable weights" so the estimate Q network.

Yes of course, but the question is quite different, maybe I've not explained well. Try look at this question: https://stats.stackexchange.com/questions/306921/how-to-update-network-weights-in-dqn

In other words, "to construct a target vector" do I have to use the target network or the main network?

Your Q target vector that you want to converge to is the same as your output vector except for the Q at the index of the chosen action, that one is the Y_doubleDQN value described in the paper.


text:
Is there any real-time computer vision system that can learn to detect new objects of new classes?
Suppose you have a ground plane and can use a stereo vision system to detect things that are possibly separate objects.
Suppose also your robot or agent can attempt to pick up and move these objects around in real-time.
Is there any current system in computer vision that allows new objects to be learned in real-time?

By "real-time" do you mean "incrementally", i.e. learn new classes incrementally as they arrive, and not necessarily strictly "online" and "real-time" (i.e. fast)?

Both possibly, basically not require retraining of the whole system.

You are probably looking for incremental learning (sometimes known as lifelong learning) techniques, i.e. machine learning techniques that attempt to address the catastrophic forgetting effect of neural networks when trained incrementally, i.e. as new classes or data are added to the original training data.
There are different techniques and some of them store (or compress) the old data in order to fully or partially re-train the neural network with the new classes or data.  However, note that this is a relatively new area of research and significant progress still needs to be made to produce serious tools. If you are specifically interested in incremental class learning, maybe have a look at this paper Class-incremental Learning via Deep Model Consolidation (2020).


text:
What is the Preferred Mathematical Representation for a Forward Pass in a Neural Network?
I know this may be a question of semantics but I always see different articles explain forward pass slightly different. e.g. Sometimes they represent a forward pass to a hidden layer in a standard neural network as np.dot(x, W) and sometimes I see it as np.dot(W.T, x) and sometimes np.dot(W, x).
Take this image for example. They represent the input data as a matrix of [NxD] and weight data as [DxH] where H is the number of neurons in the hidden layer. This seems the most natural since input data will often be in tabular format with rows as samples and columns as features.

Now an example from the CS231n course notes. They talk about this below example and cite the code used to compute it as:
f = lambda x: 1.0/(1.0 + np.exp(-x)) # activation function (use sigmoid)
x = np.random.randn(3, 1) # random input vector of three numbers (3x1)
h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations (4x1)
h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations (4x1)
out = np.dot(W3, h2) + b3 # output neuron (1x1)

Where W is [4x3] and x is [3x1]. I would expect the weight matrix to have dimensions equal to [n_features, n_hidden_neurons] but in this example it just seems like they transposed it naturally before it was used.

I guess I am just confused about general nomenclature in how data should be shaped and used consistently when computing neural network forward passes. Sometimes I see transpose, sometimes I don't. Is there a standard, preferred way to represent data in accordance to a diagram like these This question may be silly but I just wanted to discuss it a bit. Thank you.

I don't think there's a "standard way" of expressing the forward pass: you use the transpose when you need to use it, and this depends on how you define the weights and inputs matrices, and on the architecture of your neural network. For example, in a fully connected feedforward neural network, you know that every neuron in the previous layer is connected to every neuron in the current layer, so, as long as this is satisfied when you multiply the matrices, it does not matter whether you use transposes or not, and I don't think that, in computational terms, it makes any difference if you use transposes or not. (By the way, if you are writing something, I suggest that you always specify the dimensions of your matrices and your conventions).
Of course, if you want to use a library like TensorFlow, you will probably need to follow the conventions of the library, but this is another story.


text:
What is the main contribution of the paper Disentangling by Factorising?
Considering the paper Disentangling by Factorising, in addition to introducing a new model for Disentangled Representation Learning, FactorVAE (see figure), what is the main theoretical contribution provided by the paper?

One of the core contributions presented in the paper consists of understanding at a deeper level the objective function used in Beta VAE and improving it
More specifically, the authors started from Beta VAE OF
$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-\beta K L\left(q\left(z | x^{(i)}\right) \| p(z)\right)\right]$$
which consists of the classical reconstruction loss and the regularization term which steers the Latent Code PDF towards a target PDF.
Developing the second term
$$\mathbb{E}_{p_{\text{data}}(x)}[K L(q(z | x) \| p(z))]=I(x ; z)+K L(q(z) \| p(z))$$
they observed there are 2 terms which push in different directions:
$I(x,z)$ is the mutual information between the input data and the code, and we do not want to penalize this as in fact we want this to be as high as possible, so that the decode contains as much information as possible about the input
$KL(q(z) || p(z))$ is the KL divergence term which if penalized pushes the latent code PDF $q(z)$ towards the prior $p(z)$ which we choose as factorized
So penalizing the second term in Beta VAE OF means also penalizing the mutual information hence ultimately loosing reconstruction capability
The authors then propose a new OF which fixes this
$$\frac{1}{N} \sum_{i=1}^{N}\left[\mathbb{E}_{q\left(z | x^{(i)}\right)}\left[\log p\left(x^{(i)} | z\right)\right]-K L\left(q\left(z | x^{(i)}\right)|| p(z)\right)\right] -\gamma K L(q(z) \| \bar{q}(z))$$
In the rest of the paper they run experiments and show they have improved the disentanglement of the representation (according to the new metric they propose in the paper) while preserving reconstruction hence improving the reconstruction vs disentanglement tradeoff with respect to the Beta VAE paper


text:
What are the real applications of hierarchical temporal memory?
What are the real applications of hierarchical temporal memory (HTM) in machine learning (ML) these days?

The following figure [1] is inspiring:

It means HTM can be used as a memory unit in different neural networks such as RNN. In the same reference, we can find the following advantages and applications of HTM:

Sequence learning:
  Being able to model temporally correlated patterns represents a key property of intelligence, as it gives both biological and artificial systems the essential ability to predict the future. It answers the basic question “what will happen next?” based on what it has seen before. Every machine learning algorithm should be able to provide valuable predictions not just based on static spatial information but also grounding it in time.
High-order predictions:
  Real-world sequences contain contextual dependencies that span multiple time steps, hence the ability to make high-order predictions becomes fundamental. The term “order” refers to Markov's order, specifically the minimum number of previous time steps the algorithm needs to consider in order to make accurate predictions. An ideal algorithm should learn the order automatically and efficiently.
Multiple simultaneous predictions:
  For a given temporal context, there could be multiple possible future outcomes. With real-world data, it is often insufficient to only consider the single best prediction when information is ambiguous. A good sequence learning algorithm should be able to make multiple predictions simultaneously and evaluate the likelihood of each prediction online. This requires the algorithm to output a distribution of possible future outcomes.
Continual Learning:
  Continuous data streams often have changing statistics. As a result, the algorithm needs to continuously learn from the data streams and rapidly adapt to changes. This property is important for processing continuous real-time perceptual streams, but has not been well studied in machine learning, especially without storing and reprocessing previously encountered data.
Online learning:
  For real-time data streams, it is much more valuable if the algorithm can predict and learn new patterns on-the-fly without the need to store entire sequences or batch several sequences together as it normally happens when training gradient-based recurrent neural networks. The ideal sequence learning algorithm should be able to learn from one pattern at a time to improve efficiency and response time as the natural stream unfolds.
Noise robustness and fault tolerance:
  Real world sequence learning deals with noisy data sources where sensor noise, data transmission errors and inherent device limitations frequently result in inaccurate or missing data. A good sequence learning algorithm should exhibit robustness to noise in the inputs.
No hyperparameter tuning:
  Learning in the cortex is extremely robust for a wide range of problems. In contrast, most machine-learning algorithms require optimizing a set of hyperparameters for each task. It typically involves searching through a manually specified subset of the hyperparameter space, guided by performance metrics on a cross-validation dataset. Hyperparameter tuning presents a major challenge for applications that require a high degree of automation, like data stream mining. An ideal algorithm should have acceptable performance on a wide range of problems without any task-specific hyperparameter tuning.

One of the applications of HTM is in Anomaly detection. See, for example, this paper Unsupervised real-time anomaly detection for streaming data (2017) which used from a network of HTMs to solve this problem in streaming data.
The implementation of HTM (NuPIC) can help to see more applications of HTM in ML.

I would like to note that HTM has evolved over the years. Also, note that, even though you're quoting an article by Numenta's visiting researcher, it's not true that HTM doesn't require hyper-parameter tuning. There are many parameters in Numenta's models. I think it would be nice to also provide an example of where HTM has been used in another traditional neural network, as the picture suggests. Apart from that, I think the content in the quote is correct or, at least, this is what Numenta wants to achieve (but I am not sure if they have already achieved that).

@nbro, thanks for your comments. Your second comment has been resolved.


text:
How to learn how to select a subgraph via reinforcement learning?
I have the following problem.
I am given a graph with a lot (>30000) nodes. Nodes are associated with a low (<10)-dimensional feature vector, and edges are associated with a low (<10)-dimensional feature vector. In addition, all nodes start out having the color white.
At every time step until completion, I want to select a subset of the nodes in the graph and color them blue. Then I receive a reward based on my coloring. I continue until all nodes are colored blue, and the total reward is the sum (maybe with a gamma factor) of my total rewards.
Do you have suggestions of papers to read where the task was choosing an appropriate subgraph from a larger graph?
Just doing a node classification task using a Graph Convolutional Network doesn't seem to do well, I suspect because, given that a good heuristic for reward is connectivity, it would need to learn to choose an optimal neighborhood in the graph and upweight only that neighborhood.
To contextualize, each of the nodes of the graph represents a constraint that will be sent to an incremental SMT solver, and edges represent shared variables or other relationships between the constraints.  I have found empirically that giving these constraints incrementally to the SMT solver when in a good order can be faster than just dumping the entire problem into an SMT solver, since the SMT solver doesn't have the best heuristics for this particular SMT problem.  However, eventually, I want to add all the constraints, i.e., color the entire graph. The cost is the amount of time the solver takes on each set, with a reward at the end for completing all the constraints.


text:
What is an auto-associator?
What is an auto-associator, and how does it work? How can we design an auto-associator for a given pattern? I couldn't find a clear explanation for this anywhere on the internet.
Here's an example of a pattern.

Hi and welcome to AI SE! Where did you hear of this term "auto-associator"? In which context? Are you referring to "auto-associative memory"? Please, edit your post to explain this.

Hi, I came across this term when trying to solve a problem in neural networks. I came across auto-associative memory also. But couldn't find what is meant exactly by an auto associator. so I asked here. :) look forward to some help.

Can you please provide a link to the source where you read the term "auto-associator"?


text:
How does SGD escape local minima?

SGD is able to jump out of local minima that would otherwise trap BGD

I don't really understand the above statement. Could someone please provide a mathematical explanation for why SGD (Stochastic Gradient Descent) is able to escape local minima, while BGD (Batch Gradient Descent) can't?
P.S.
While searching online, I read that it has something to do with "oscillations" while taking steps towards the global minima. What's that?

This is a popular question. You can find a good explanation in the answer here.

Where did you read that SGD is able to escape local minima?

The statement above is from a MOOC on Coursera, while I found more details here - http://proceedings.mlr.press/v80/kleinberg18a/kleinberg18a.pdf

@cogito_ai I wouldn't say that SGD can't really escape local minima, but that it can get stuck in other (better?) minima. It's not like simulated annealing, where sometimes you perform a random action to escape local minima. However, it's true that there's some form of stochasticity in SGD that probably helps to explore more the search space. And that's what people probably mean by that sentence. See also https://ai.stackexchange.com/a/20380/2444.

Anyway, can you please provide the link to the exact Coursera's lecture that claims that? Also, if you want, later, I could read that paper and try to provide an answer based on it. And don't forget to tag me with @nbro next time, so that I see the comment ;)

@nbro One of the quizzes in here claim my question's opening statement. https://www.coursera.org/learn/getting-started-with-tensor-flow2
Also, it'd be great if you could go through that paper when you have the time and write an answer based upon it! :)

SGD has greater mobility precisely due to its stochastic nature. Because the gradient can drastically change (or oscillate) due to random training items being picked, it might produce a gradient large enough to jump out the local minimum.
Take a look on the purple line on the illustration below (source):

Look how chaotic it is compared to the non stochastic ones. It is harder to keep a naughty child in place than a well-behaved one.


text:
How to prevent deep Q-learning algorithms to overfit?
I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time.
Did the model overfit the environment? It seems that the agent just memorized the environment.
What are the common techniques to prevent the agent to overfit like that? Is that a common problem?

In supervised learning, overfitting generally means that you learn the training data, but don't perform well on the test data. The concept of overfitting in reinforcement learning makes sense in the case if your training environment is different than your evaluation/test environment. However, if you train your RL agent in the same exact environment where you want it to behave, you actually want it to "overfit", i.e. learn the environment dynamics. See this answer for more details.


text:
Why is the accuracy of my model very low on a separate dataset from the training and test datasets?
I am working on stock price prediction project, I am using the support vector regression (SVR) model for it.
As I am splitting my data into train and test, I am getting high accuracy while predicting test data after fitting the model.
But now when I am trying to use another data which I separate out from the used dataset before start doing anything, it gives me very bad results. Can anyone tell me what's happening?

Looking forward to your response.

Rather than providing a screenshot of the code, it may be a good idea to explain how you obtained that separate dataset. Maybe you should also describe more in detail your datasets (e.g. how many entries they have, how many features, etc.). Also, note that in the source code that you're showing us, apparently, you're calculating the r2_score rather than the accuracy. You should clarify this (because in your post you say "accuracy").

The difference is very large. Have you used any type of scaling? Maybe you forgot to scale the newest data set?


text:
Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?
Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs?
Given that I have trajectories of $(s_t,a_t)$ pairs and the behavior policy distribution $\pi_b(a_t | s_t)$ can be approximated with a neural network.
A paper I came across where they say that IS can be used for continuous states whereas WIS cannot be used for function approximation. I am not sure why WIS cannot be applied to the continuous case whereas IS can be. Both of these techniques seem similar.


text:
Is there a general file type associated with AI projects?
This is a general question.
Is there a general file type associated with AI projects?
Photoshop = .psd
Excel = csv
Artificial Intelligence = ?

AI is a very broad term and there is no such "AI" kind of project. Do you mean a machine learning or a deep learning based project?

Hi and welcome to AI SE! Although you already have an answer, this question is a little bit unclear and needs more details. It seems like you're saying/assuming that AI is just one application that you use on your computer.

No, there is no file type associated with AI projects in general.
Your examples of Photoshop and Excel are specific corporate branded products. These store bespoke data that only works with those products (plus maybe a few converters that can read the files for competitor products).
Even more general examples such as .jpg for images or .txt for text documents are not a good match to AI in general. AI is such a broad field, it is next to impossible to define a standard set of components of an "AI project" in order to build a single file format that could handle the contents of all AI projects.
That said, practical work in AI is likely to include use of specific file formats and extensions, depending on what you do. For instance if you work with Python and TensorFlow you are likely to use .py files for your source code, .ckpt for neural network training checkpoints, .pb for saved models. The last two extensions - ckpt and pb - are only semi-formal naming conventions though.
This lack of a single file extension, and use of maybe a dozen ones that you would be familiar with on a project, is true of most coding and software development work. If you work with an integrated devlopment environment (IDE), then you might have a single "master" file that allows you to load up all the resources in the project to work on them. That is entirely optional though, and may have any of a number of extensions depending on which IDE you use.
As a quick example, the Transformers project on github is a popular resource for natural language processing, a topic often considered as part of AI. A quick look at that project shows:

Various development tool configuration files either starting with . or with various extensions - .cfg, .yaml
.py files for Python source code
Script files with .sh extension or with extension removed for convenience of using on command line
Dcoumentation written with .txt, .md and .rst extensions.
Some .ipynb files used for Jupyter notebooks - a format that bundles documentation, Python scripts and their output for sharing work.


text:
How to effectively crossover mathematical curves?
I'm trying to optimize some reflective properties of curves of the form:
$a_1x^n+a_2x^{n-1}+a_3x^{n-2} + ... + a_n + b_1y^n+b_2y^{n-1}+b_3y^{n-2} + ... + b_n = 0$
which is basically the curve that you get when you sum two polynomials of same degree in different variables:
$f(x) + g(y) = 0$
Anyways, I was wondering what would be a good way to do crossover on two such curves? I tried averaging the curves and then mutating them, but the problem is that the entire population quickly becomes homogeneous and the fitness starts to drop. Another typical method I tried is taking a cutoff point somewhere on the expansion above and mixing the left and right part from both parents.
Of course there are many ways to do a similar process. I could order the above expansion differently and then do a cutoff. I could seperate Xs and Ys and do two seperate cutoffs. Etc. The question is: in the context of algebraic curves, which method of generating offspring would be a good option considering I want to optimize some property of the curve (in this case I want it to have some reflective properties)?


text:
How do you prove that minimax algorithm outputs a subgame-perfect Nash equilibrium?
At every node, MAX would always move to maximise the minimum payoff while MIN choose to minimise the maximum payoff, hence there is nash equilibrium.
By using backwards induction, at every node, MAX and MIN player would act optimally. Hence, there is subperfect nash equilibrium.
How do I formally prove this?

Have you looked at this article http://math.ucr.edu/home/baez/games/games_18.html?


text:
Is there any good source for when the pole actually starts all the way at the bottom, in the cartpole problem?
There are a lot of examples of balancing a pole (see image below) using reinforcement learning, but I find that almost all examples start close to the upright position.
Is there any good source (or paper) for when the pole actually starts all the way at the bottom?

Almost certainly, there is no such paper since that would be a trivial problem. The pole lying flat is the definition of failure, hence game over. If you started in that position, you would be permanently in the game-over state and you would never learn anything.
The reason is that if the pole is lying flat, then, if you apply a force on the cart (in the same direction as the pole is pointing, say), the pole head moves in exactly the same direction as the cart (i.e. the directional vectors of the cart and pole head movements are identical). Hence, the pole head never moves upwards.
In fact, I am fairly certain, that below certain angle with the surface, the pole can no longer be stabilized. This should be possible to prove from the dynamic equations governing the movement of the cart and the pole. This may not be easy, though, and definitely not easy for me (these are second-order differential equations). Anyway, with this in mind, you can see why one needs to start close enough to the stationary point for the problem to have a solution.
If the pole could go below the surface and swing, however, as here, it would be similar to the acrobot problem mentioned by Neil and you could start anywhere.

"Almost certainly, there is no such paper since that would be a trivial problem." it might seem trivial after it has been solved. I have in the meantime found several papers: https://scholar.google.com/scholar?hl=nl&as_sdt=0%2C5&q=swing-up+cart&btnG=. There is also no need to start near the top, the bottom is fine. But yes, it is similar to the acrobat problem.

@ThomasW Trivial in mathematical sense, that was not a reference to anyone's intelligence. If you know that the pole does not move when you apply the force, then the solution is trivial (i.e. any action yields the same outcome).

@ThomasW The references you point to analyze the problem, where the pendulum can rotate 360 degrees. That I mentioned in my last sentence as a side note. My answer was about the pole-balancing problem where the pole can only deviate from the upright position by 90 degrees on either side, since this is the problem which you asked about.

It is difficult to prove a negative, but I doubt there will be a paper on that specific problem. It should be relatively easy to adjust the environment or write a new one that does this if you wished though.
A very similar environment that does have a lot more written about it is Acrobot, which does have a OpenAI Gym version. Instead of a cart on a track with forces applied and a free join to the pole, there is a longer pole fixed to a free joint, with an active joint in the middle (that the agent can apply forces to). It can be thought of as a very simple model of an acrobat on a trapeze swing (with poles instead of chains, so the swing is stable balanced upside-down)
The degrees of freedom and difficulty of the Acrobot task are similar to CartPole - I would rate it as a harder problem overall, but if you started CartPole with the pole hanging down in fact the two problems are very similar. Usually the joint motor is made too weak to achieve the goal of balancing in a single clean action, and the agent must learn to build momentum over a few swings, before moving to the balance point. That also makes both your alternate start position CartPole and Acrobot similar to MountainCar.
For a low-effort look at a very similar problem you could try Acrobot. Otherwise you will likely need to do some custom work on CartPole.


text:
Why do we use a delay when feeding our input data to the echo state network?
I'm new to working with neural networks and have recently began implementing neural networks for time series forecasting in some of my work. I've been particularly using Echo State Networks and have been doing some reading to understand how they work. For the most part, things seem pretty straight forward, but I'm confused as to why we use a 'delay' when feeding our input data (the delay concept mentioned in the paper Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication)?
I'm looking at some source code on Github, and they implement this delay as well (they feed two arrays inputData & targetData, into the network where one is delayed by one element relative to the other). I am noticing that the larger the delay, the worse the fit.
Why is this done?
My interest is eventually to forecast past sample data.

It is not a 'delay' but a 'look ahead' as some people call it.  This is how far in the future the model can forecast (or predict).
In reference to the GitHub source, the code uses a look ahead ('delay') of one.  The echo state network learns on the 'inputData' using a future version of itself called 'targetData'.
Here is an analogy.  It is early on Friday, June 5, and I have the last 51 days worth of Amazon stock prices.  I can use the first 50 days of prices, which go up to and include Wednesday, June 3, to train a network to predict Thursday, June 4.  If I have a lot of historical data on Amazon, I can train many 50 day windows.  Each 50 day window would have as its target a stock price one day in the future.
Performance generally degrades the further into the future you try to forecast.  One reason is due to the build up of error over time.  This is true in life.  We usually know fairly accurately what we will do tomorrow, less so next week, and it would be a wild guess to predict what we will be doing in ten years.


text:
Detect object in video and augment another video on top of it
I'm trying to detect an object in a video (with slight camera movement), and then augment another video on top of it. What is the simplest approach to do that?
For instance, let's assume I have this simple video of a couch HERE. Now I want to augment the right cusion with a human or a dog. The dog or human are video itself (let's assume transparency is not an issue).
What's the simplest approach to do that?


text:
What is the relation between multi-agent learning and reinforcement learning?
What is the relation between multi-agent learning and reinforcement learning?
Is one a sub-field of the other? For instance, would it make sense to state that your research interest are multi-agent learning and reinforcement learning, or would that be weird as one includes most of the topics of the other?

I think there is an intersection. There are problems that are in reinforcement learning and in learning in multi-agent systems. There are problems in reinforcement learning, but not exactly in multi-agent systems. And there is learning in multi-agent systems that is not through reinforcement learning. For sort you can say: multi-agent reinforcement learning. I recommend take a look at these references:

Bu, Lucian, Robert Babu, and Bart De Schutter. "A comprehensive survey of multiagent reinforcement learning." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38.2 (2008): 156-172.
Busoniu, Lucian, Robert Babuska, and Bart De Schutter. "Multi-agent reinforcement learning: A survey." 2006 9th International Conference on Control, Automation, Robotics and Vision. IEEE, 2006.

If you are a little bit familiar with RL, it's clear that there are RL algorithms that are not directly (or immediately) applicable to multi-agent systems, so there is no need to explain further that case. However, I think this answer would improve considerably if you can provide an example of a MAS that doesn't use RL and an example that uses RL. It would also be interesting to talk about hierarchical reinforcement learning (i.e. is HRL a multi-agent system?).

Thank you very much for your answer. The comment from @nbro is also very helpful and I agree.


text:
How can I increase the exploration in the Proximal Policy Optimation algorithm?
How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: https://stable-baselines.readthedocs.io/en/master/modules/ppo2.html