text:
Why does k-means have more bias than spectral clustering and GMM?
I ran into a 2019-Entrance Exam question as follows:

The answer mentioned is (4), but some search on google showed me maybe (1) and (2) is equal to (4). Why would k-means be the algorithm with the highest bias? (Can you please also provide references to valid material to study more?)

Hello. Can you please provide the link to the exam question?

I'm not an expert on clustering, but here's my take below. Note that this is only based on theoretical arguments, I haven't had enough clustering experience to say if this is generally true in practice.
K-means vs GMM
K-means has a higher bias than GMM because it is a special case of GMM. K-means specifically assumes the clustering is spherical (meaning each dimension is weighted equally important) and that the clustering problem is a hard clustering problem (each data point can only belong to one label). So, theoretically, K-means should perform equal to GMM (under very specific conditions) or worse. More info
K-means vs GMM (identity covariance matrix)
K-means has a higher bias than GMM (identity covariance matrix) because it is also a special case. K-mean specifically assumes the hard clustering problem, but GMM does not. Because of this, GMM has stronger estimates for the mean of the centroids. More specifically,

[GMM] estimates the cluster means as weighted means, not assigning observations in a crisp manner to one of the clusters. In this way it avoids the problem explained above and it will be consistent as ML estimator (in general this is problematic because of issues of degeneration of the covariance matrix, however not if you assume them spherical and equal).

In practice, if you generate observations from a number of Gaussians with same spherical covariance matrix and different means, K-means will therefore overestimate the distances between the means, whereas the ML-estimator for the mixture model will not.

So, theoretically, K-means should perform equal to GMM (identity covariance matrix) or worse. More info
K-means vs Spectral clustering
K-means has a higher bias then spectral clustering because spectral clustering effectively uses K-means after processing more information from the matrices.

Spectral clustering usually is spectral embedding, followed by k-means in the spectral domain.

So yes, it also uses k-means. But not on the original coordinates, but on an embedding that roughly captures connectivity. Instead of minimizing squared errors in the input domain, it minimizes squared errors on the ability to reconstruct neighbors. That is often better.

More info

K means tried to cluster data points into 0 and 1 rules for cluster assignment i.e. Data Point belongs to a class or it does not. But sometimes the data points comes from classes whose probability distribution functions have overlap with each other.
Now Hard Clustering of K means leads to truncation on the data generating\original probability distribution which lead to the Bias in K means algorithm.
Spectral clustering does soft assignment so they are less biased.
This has been proved by taking example in this paper : https://www.researchgate.net/publication/287829457_On_the_bias_and_inconsistency_of_K-means_clustering/link/567a04a608ae40c0e27df98c/download


text:
What is the difference between Attention Gate and CNN filters?
Attention models/gates are used to focus/pay attention to the important regions. According to this paper, the authors describe that a model with Attention Gate (AG) can be trained from scratch. Then the AGs automatically learn to focus on the target.
What I am having trouble understanding is that, in the context of computer vision, doesn't a filter from the convolutional layers learn the region of interest?
The authors say that adding Attention Gate reduces complexity when compared with multi-stage CNNs. But the job a trained AG would do is the same as that of a filter in a convolutional layer that would lead to the correct output, right?

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel.
Note, when I say "at every pixel" this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: https://arxiv.org/pdf/1709.01507.pdf
While this is one of the strengths of CNNs, since it drastically reduces the number of parameters of a network, you can imagine that it may not make sense to treat every part of the image the same regardless of the content. This is what the attention gate is for.
By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image.
CNNs without AG can typically learn the same things but may need more channels and layers. Attention gates allow to treat filters differently depending on the content and hence make it easier to learn with less filters.
What you use as the attention tensor can vary. In the paper you referenced, the tensor was one of the input tensors of the convolutional layer. In this case, it is called self-attention. However, with attention in general this gate tensor may also come from other information such as another network.

"By performing an element-wise multiplication of the output of a CNN layer with a gate tensor (typically clamped to the 0-1 range with a sigmoid) we can effectively down weigh or ignore features in certain areas of the image."
Isn't this similar to having a CNN layer followed by a fully connected one with sigmoid activation? The only difference is that we're performing element wise rather than tensor multiplication

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Why is the ideal exploration parameter in the UCT algorithm $\sqrt{2}$?
From Wikipedia, in the Monte-Carlo Tree Search algorithm, you should choose the node that maximizes the value:
$${\displaystyle {\frac {w_{i}}{n_{i}}}+c{\sqrt {\frac {\ln N_{i}}{n_{i}}}}},$$
where

${w_{i}}$ stands for the number of wins for the node considered after the $i$-th move,

${n_{i}}$ stands for the number of simulations for the node considered after the $i$-th move,

$N_{i}$ stands for the total number of simulations after the $i$-th move run by the parent node of the one considered

$c$ is the exploration parameter—theoretically equal to$\sqrt{2}$; in practice usually chosen empirically.

Here (and I've seen in other places as well) it claims that the theoretical ideal value for $c$ is $\sqrt{2}$. Where does this value come from?
(Note: I did post this same question on cross-validated before I knew about this (more relevant) site)

It's not the same question, but the answer to this question should contain the answer to your question. You may want to write a formal answer to your own question, once you understood the explanation in the paper and/or answer.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Would it make sense to share the layers (except the last one) of the neural networks in Double DQN?
Context: Double Q-learning was introduced to prevent the maximization bias from q-learning. Instead of learning a single Q-network, we can learn two (or in general $K > 1$) and our Q-estimate would be the min across all these Q-networks.
Question:
Does it make sense to share the layers of these Q-networks (except the last layer)?
So, instead of having 2 networks of size [64, 64, 2] (with ~8.5K parameters in total) we can have one network of size [64, 64, 4] (with ~4.3K params).
I couldn't see much of a downside to this, but all the implementations I've seen keep two completely different networks.


text:
How to deal with predictions for data outside the range of the training dataset in neural networks?
I’ve set up a neural network model to experiment with predicting foreign exchange rates based on various economic data. The model learned fine and the test data is OK ($R^2 = 0.88$).
But I can't figure out how to input data for scenarios where new data is outside the range of the datasets used to train the model. For example, if US debt is increased (again), then it will be outside the data range used for the training datasets, so, when normalised using the same parameters as the training dataset (0-1 scale), it will be greater than 1, so the model rejects it.
Everything I've read says to normalise the new data using the training data parameters (understandably), but I can't find anything that explains how to use the model to make predictions where new data is outside the range of the training datasets.
In parallel, I've used a regression model, but I'm fairly new to neural networks and would like to find a way of using these for this kind of prediction model. Any help gratefully received.

This part "when normalised using the same parameters as the training dataset (0-1 scale), it will be greater than 1, so the model rejects it." is not very clear. Are you normalizing the inputs? Can you describe more in detail 1. what and how are you normalizing, 2. which parameters of the model are you referring to?, 3. Why exactly you cannot normalise the test dat in the same way? I don't get why "it will be greater than 1".

Also, would be good to clarify what "the model rejects it" mean. Neural networks should be able to handle inputs outside of the (0-1) scales, regardless of the stability of prediction. What library are you using?

All inputs (7+target) are normalised using (x-min)/(max-min) hence the 0-1 scale from normalising the training data.   So if US debt max was, say USD18tn and the latest input is now USD24tn it will be greater than 1 using the normalisation paramaters from the training dataset.

In reply to what library - I'm using the Knime platform for now as a quick way to test how useful the model might be.  The Knime "node" is  RProp Multi-layer Perceptron which uses the logistic activation function for forward propogation.   I've been trying to run alternative model structures so will need to rebuild the model as before to replicate the exact error message.

I'll close this question, with thanks to nbro for his comments.  With reassurance that "Neural networks should be able to handle inputs outside of the (0-1) scales, regardless of the stability of prediction",  I dug a little deeper and the issue was being generated by the denormalisation step saying the value of the new prediction was out of bounds. ie, larger than the maximum value from the training data.  I've now resolved this by breaking down the steps in the model and double checking the output.by extracting the actual parameters used in the normalisation step.

Also, for info although not a programming point, I've changed the input data to use ratios of similar parameters rather than absolute values,eg, for EUR.USD, use the ratio of US debt to EU debt as the input rather than the two debt figures separately.  This should keep any out-of-range values a lot closer to the training data range unless the economic world gets really out of whack!


text:
What are the differences between Proximal Policy Optimization versions PPO1 and PPO2?
When Proximal Policy Optimization (PPO) was released, it was accompanied by a paper describing it.
Later, the authors at OpenAI introduced a second version of PPO, called PPO2 (whereas the original version is now commonly referred to as PPO1). Unfortunately, the several changes made between PPO1 and PPO2 are pretty much undocumented (as stated over here).
Someone associated with OpnenAI's baselines Deep Reinforcement Learning repository commented that the main advancement of PPO2 (compared to PPO1) was the use of a more advanced parallelism strategy, leading to improved performance. Unfortunately, the person omitted naming further changes made.
Now, I was wondering if anyone is aware of a (reliable) source of information or (preferably) even some published literature that lists all the numerous differences between PPO1 and PPO2.


text:
Are the relative magnitudes of the learned and optimal state value function the same?
I have been reading recently about value and policy iteration. I tried to code the algorithms to understand them better and in the process I discovered something and I am not sure why is the case (or if my code is doing the right thing)
If I compute the expected value until convergence I get the following grid:

If I compute the optimal value with value iteration I will get the following.

As you can see, the values are different but their relative magnitude is similar, i.e. the 3rd column in the last row has the greatest value in both computations.
I believe this makes sense, as the expected value will tend to accumulate values in "promising" cells. But I assume that the first computation won't help much because it does not tell us what is the optimal policy, whereas the second, does.
Is my understanding correct?


text:
Why is Adam trapped in bad/suspicious local optima after the first few updates?
In the paper On the Variance of the Adaptive Learning Rate and Beyond, in section 2, the authors write

To further analyze this phenomenon, we visualize the histogram of the absolute value of gradients on a log scale in Figure 2. We observe that, without applying warmup, the gradient distribution is distorted to have a mass center in relatively small values within 10 updates. Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Here is figure 2 from the paper.

Can someone explain this part?

Such gradient distortion means that the vanilla Adam is trapped in bad/suspicious local optima after the first few updates.

Why is this true?

The authors describe their belief in Section 3:

Due to the lack of samples in the early stage, the adaptive learning rate has an undesirably large variance, which leads to suspicious/bad local optima.

Diving further, in section 3.2:

Adam uses the exponential moving average to calculate the adaptive learning rate.  For gradients $\{g_1,\dots,g_t\}$, their exponential moving average has a larger variance than their simple average.  Also, in the early stage ($t$ is small), the difference of the exponential weights of $\{g_1,\dots,g_t\}$ is relatively small (up to $1−β^{t−1}_2$).

It seems like the root issue is that exponential moving average, while great with many data samples, have too large of a variance with few data samples. It is this variance that sometimes allow the gradient descend in a bad optima.


text:
Literature on the advantages of using an auto-encoder for classification
Given a supervised problem with X, y input pairs, one can do two things for obtaining the function f that maps X with y with Neural Networks (and in general in machine learning):

Deploy directly a supervised learning algorithm that maps X to y

Deploy a (variational) auto-encoder for learning useful features, and then using these for training the supervised learning algorithm

I would like to be pointed to some papers/blogs that explain which technique is better and when or where they conduct empirical benchmarking experiments.

Are you asking about the advantages of using auto-encoders to pre-train models in an unsupervised way that are later trained in a supervised way to solve a classification task? You mention variational auto-encoders. Are you interested specifically in VAEs or any AE?

Yes exactly, AE or VAE is the same


text:
What is the difference between object tracking and trajectory prediction?
In autonomous driving, we know that the behaviour prediction module is concerned with understanding how the agents in the environment will behave.
Similarly, in the perception module, the tracking algorithms are responsible for getting an estimate of the object's state over time.

Can you please provide a link to a reference that says that autonomous vehicles are equipped with a "behaviour prediction module" and a "perception module", which makes you think that they are different? Note that I'm not saying that this is or not the case. I just think that this post is not fully clear, especially because I don't immediately see the connection between what you wrote in the title and in the body of your post. Can you please connect the two in a clearer way?

Although both processes might be doing estimations (because data sources aren't perfect and/or have noise), there is a key difference:

Object tracking cares solely abouth where objects are now. That means that there is actual sensor data that can support the current position. For example: from a camera and a lidar, the computer predicts where a vehicle stands.

Trajectory prediction is done with the main purpose of predicting where objects will be in the future, meaning that there is no sensor data yet. For example: from past data, the computer predicts where a vehicle will be after 1 second from now.

However, both processes might need each other. In order to predict a trajectory, it could be easier to work with curated positions given by an object tracker than raw sensor data. On the other hand, past predictions might be given as input to an object tracker, in order to mitigate noise in the sensor data and acting so as a belief-based filter.

Your definitions look reasonable to me, but I am not an expert in this area, so I cannot assess the correctness of this answer. Are these definitions common in the context of computer vision (or whatever the context is)? Maybe you should provide a link to a reference for more details (given that the OP was also mentioning autonomous vehicles and it's not clear if your definitions apply to that context or, in general, apply to any computer vision applications), but I guess this is not necessary, if you more or less familiar with these topics.

there is a presentation made by kris kitani in robust video scene understanding workshop,start from 5:55 https://www.youtube.com/watch?v=bwIgpxmWnWM&t=22137s


text:
Is AlphaFold just making a good estimate of the protein structure?
In the news, DeepMind's AlphaFold is said to have solved the protein folding problem using neural networks, but isn't this a problem only optimised quantum computers can solve?
To my limited understating, the issue is that there are too many variables (atomic forces) to consider when simulating how an amino acid chain would fold, in which case only a quantum computer can be used to simulate it.
Is the neural network just making a very good estimate, or is it simulating the actual protein structure?

the CASP competition entered by deepmind is rigorous in its scoring & is unbiased wrt prediction technique. increasingly there is not really a technical difference between "estimation" vs "simulation" by ML. yes QM computers are thought to have an advantage here but actually applying them to real world problems is still relatively far off. the computational complexity of protein folding is very high but its a case where the "actual complexity" is not really known/ proven. so deepmind work is a demonstration/ "proof of principle" its within reach of existing systems but only via excellent engr.

AlphaFold (version 1 and 2) predicts (so estimates) the 3D shape of the protein from the sequence of amino acids. AlphaFold's performance is measured with the global distance test (GDT), which is a measure of similarity between two protein structures (the prediction and the ground-truth) that ranges from 0 to 100.
There is a short video and a longer one (both by DeepMind) that summarise the issue of protein folding, how it is important, how well AlphaFold approximately solves it (in the competition Critical Assessment of protein Structure Prediction (CASP)), i.e. AlphaFold 2 achieves a median GDT score of 92.4 (and 87 on the hardest proteins), which is a lot higher than AlphaFold 1's GDT score of 58 (which was the highest achieved score at the time), where, according to John Moult (president of CASP), a score around 90 is considered a satisfactory solution to the protein folding problem. You can find more details about AlphaFold 2 in this DeepMind blog post and about AlphaFold 1 in this other blog post or the associated paper published in Nature this year. You can find the code for AlphaFold 1 here, but there are other community/open-source implementations.
Despite the importance of the problem and achievement, there is clearly a lot of hype about this breakthrough (given also that it was achieved by DeepMind). This is also discussed in this video by Lex Fridman.

what is the difference between "hype" vs "celebration"? its a thin line sometimes. think most of the official announcement + much media coverage is not really "hype" because its a real breakthrough, there is strong scientific consensus on that, many authorities quoted endorsing that its a breakthru, and the worldwide multi-decade CASP competition format also underlines that.

@vzn This is an important milestone for the machine learning community, but I don't know exactly how useful it will really be for healthcare, medicine, or related areas. We will see in the next years. There's a lot of hype (i.e. as you put it, celebration), but that's not necessarily bad. It will probably help DeepMind get even more funds, which is probably not bad, given their achievements in the last 5-7 years (considering also that it has been funded in 2010).

the issue is that there are too many variables (atomic forces) to
consider when simulating how an amino acid chain would fold, in which
case only a quantum computer can be used to simulate it.

These many variables, taking as an example the ones you mentioned, the atomic forces, are somehow grouped in order to facilitate the calculations; thus, it is not necessary for absolutely everything to be simulated simultaneously.
A notable example of this is when the K computer (one of the most powerful supercomputers in Japan, but far from being an optimized quantum computer) calculated the force needed to untie the DNA strands of histones (without taking into account the interactions between each nucleotide).
Multi Scale Modeling of Chromatin and Nucleosomes
1:37 -

By treating multiple atoms as one single particle we can increase the
number of phenomena in our simulation

Basically, they start from the following question: "What is the maximum number of elements and interactions that we can exclude from an analysis object in a simulation in order to still preserve its properties without too much loss and with good accuracy?"
Another example of how this is done is when they apply the Finite Element Method to study hemodynamics in the heart.
They simply transform the heart into a set of regular tetrahedrons, yet still manage to simulate a very wide range of phenomena - such as the rate of consumption of ATP in each part of the heart and the variation in thickness of cardiac muscles during functioning.
Multi-scale Multi-physics Heart Simulator UT-Heart
In other words, they are far from wanting to simulate the heart in all its tissue details. On the contrary, they simplify as much as possible so that simulations are workable on our current supercomputers.


text:
What does "These designs employ skip connections to avoid a situation where the shortest path between time steps increases" mean?

Less popular alternatives include adding layers to the connections from input to the hidden state, between hidden states, or from the hidden state to the output. These designs employ skip connections to avoid a situation where the shortest path between time steps increases and training becomes more difficult.

This is from page 597 of the book Machine Learning for Algorithmic Trading. And this chapter is about RNN.
I don't quite understand what does 'These designs employ skip connections to avoid a situation where the shortest path between time steps increases' means. I haven't seen any materials about how to use skip connections in RNN.

My guess (based on my separate knowledge of RNNs and skip connections, so take this comment with a grain of salt) is that skip connections in RNNs have a similar purpose as skip connections in other neural networks. Here, by "shortest path between time steps increases", I think they mean that it's harder to keep track of longer dependencies between words (chars, or whatever is the unit of a sequence, in your case probably stocks) of the sequence as those dependencies are farther away from each other. So, I guess skip connections would help to overcome that to some extent.


text:
Is there a paper/article on contextual $\epsilon$-greedy algorithm?
I am reading the paper A Contextual-Bandit Approach to Personalized News Article Recommendation, where it refers to $\epsilon$-greedy (disjoint) algorithm. I suspect, that it is just a version of a K-armed bandit with regressors that estimate the average reward for an arm. However, I cannot find the description of this algorithm in the literature (papers, books, or other resources)


text:
Order of operations on sparse recurrent network alters the output. How to deal with it?
I'm working on an implementation of NEAT, which evolves neural networks with small and sparse topologies.
Evaluating a sparse and possibly recurrent network requires a different approach than the matrix operations of dense networks, and I'm trying to wrap my head around the order in which nodes should be evaluated.
I've set up a simple example:

Every node (except inputs) starts at 0. (t0)
Every weight is 1. There's no activation function or biases.

Assuming that the algorithm has the same intuition we humans do: evaluate the nodes connected to the input, then the "aggregation" node, then the output.
How should it decide whether to evaluate node [3] before node [4] or vice-versa?
By starting (t0) at [3], the value of [4] is 0. And vice-versa.
The side-effect of this behavior appears when comparing two networks. Say you have the network above, and a copy of it where [3] and [4] are inverted. I feel like both should return the same result for the same input in order for evolution to work properly.
Any thoughts?

I believe NEAT just loops through the nodes in the order they appear in a list.
Looking at the source (.cpp) and removing time features it looks like this in JS:
 do{
    // For each node, compute the sum of its incoming activation
    for(let i=0;i<nodes.length;i++){
        if(nodes[i].type!=='input'){
            nodes[i].sum=0;
            nodes[i].active=false;
            for(let f=0;f<nodes[i].from.length;f++){
                let {index,weight}=nodes[i].from[f];
                if(nodes[index].activeCnt>0)nodes[i].sum+=weight*nodes[index].activeValue;
                if(nodes[index].active||nodes[index].type==='input')nodes[i].active=true;
            }
        }
    }
    // Now activate all the non-sensor nodes off their incoming activation
    for(let i=0;i<nodes.length;i++){
        if(nodes[i].type!=='input'&&nodes[i].active){
            nodes[i].activeValue=sigmoid(nodes[i].sum);
            nodes[i].activeCnt++;
        }
    }
}while(!outputs.isActive());

On initialization the outer while loop might run >1 times. After this it will only run once on every time step.
From my understanding, on each time step NEAT does not "push each input value fully from input to output", only "pushes from one node to the next" except at intialization.

From the network.cpp(:170) file: Keep activating until all the outputs have become active. Your assumption is only right after the first activation. However, it's propagation strategy does seem to be topology agnostic. I wonder how it responds to larger/more complex networks.

what is the point of tracking activeCnt?


text:
What are the "per image" annotations that are generally used for image datasets in AI?
Computer vision is highly benefited by AI algorithms.  Image data is abundantly available. There are different varieties of tasks such as image classification, prediction, segmentation, generation, etc.
Although the collection of the folder(s) of image(s) is mandatory, it may not be enough. Different types of annotations are used in datasets.  Annotations can be treated as some extra information related to each image that helps for the AI algorithm under consideration.
I want to know the kinds of annotations at the individual image level that are generally used. Although the necessity of a particular type of annotations depends on the task under consideration. I want to know the requirements for the contemporary prevalent tasks including classification, prediction, segmentation, and generation. You are encouraged to provide for more tasks if you are aware.
I know the following types of annotations:

Bounding box(es)
Label

What can be the other kinds of annotations used for images in image datasets?

@nbro can you please check whether this question is on-topic?

I think it's on-topic because you're asking about how data is usually labeled (so prepared) in order to train ML models, but maybe a bit too broad. This type of question makes me think that you're curious about different tasks in computer vision and how data needs to be labelled for each of them, but what if there are millions of slightly different tasks? Of course, I don't think this is the case, but I see this potential problem. So, if you could somehow narrow the scope of this question, it would be better.

Maybe when you write "contemporary prevalent tasks." you could specify which ones. I think these would be "image classification, image detection, image segmentation, and semantic segmentation" (but what if there are more?) and I guess you're interested in supervised learning, so it may be worthing clarifying that too. Again, this type of question is a bit open-ended, which is an aspect that I dislike, as it can lead to many different answers.

True @nbro I also feel it is a little bit open since the tasks can be many. And many answers are possible. But, I think that these sorts of questions can be easily answered by experts as they see many available datasets and annotations available for them.

If many answers are possible, it's too broad, and can be closed as such. So, please, narrow the scope of this post or at least be more precise about what kind of answer you're looking for. You cannot expect us to list 100 possible labels, if there are 100 different tasks.  What do you want to achieve with this question?

Yeah @nbro narrowed down.


text:
What is the difference between the $Q_a$ calculated to update delta and those to select next action in the exploitation phase?
As the title suggests, I have a doubt about the computation of the $Q_a$ used to update the delta and the $Q_a$ used to select the next action in the exploitation phase, as shown below (source of pseudocode in Figure 8.9).

In both for loops enumerated in the image with 1 and 2, the $Q_a$ are calculated considering the new state, but while in 1 for each action the active features $F_a$ are calculated, in 2 they aren't. So what are the active features to consider in 2? Are they the same as those in 1? In that case, I could avoid recalculating the $Q_a$ by storing those calculated in 1.


text:
Why is old/off-policy data harmful to on-policy/online RL?
I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the optimal action to take is entirely dependent on the state.
So while I can understand the argument from prioritized experience replay that some data is more valuable for learning than others. I cannot understand how old data would be harmful.
Further Clarification:
The emphasis on an MDP is perhaps misleading. I don't mean to imply that optimal control should be dependent on previous states necessarily... Similarly, the emphasis on off-policy & on-policy is perhaps better placed on online vs offline RL.
Performance Gap:

This [offline RL] setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.

Consider a simple example: with Q-table 'learning'. It seems to me that Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance. This seems to me like a paradox and that's what confuses me.

I am not sure I understand this question and, in particular, I don't understand this "I ask because if optimal control were indeed a MDP" and  "performance gap between off/on-policy RL". What "gap" between on and off-policy RL are you talking about? A gap in terms of what? MDP is just a way to formulate a problem. Optimal control is an area of study. Also, "I cannot understand how old data would be harmful." doesn't seem to be related to the title. Please, clarify what you're really asking here.

@nbro Neil Slater understood what I was asking so you can see his answer for context. I'm saying that off policy/offline does worse than on-policy/online, and I'm saying that I don't get why because formulating the problem as a MDP implies that any episode rollout data will always remain valid training data. So in theory excluding old data like online-RL does shouldn't increase performance yet it seems it does.

@profPlum: I need to understand what you mean by "performance" and what examples you have that online is "better" than offline by that measure? It surely is not by all measures - do have examples, or links to relevant statistics? Also, if my answer has not resolved your question, it would be useful to get some feedback on what you are still stuck on

@NeilSlater I didn't have a specific metric in mind, but lets say expected accumulated reward per episode after training. Please see the citation/quote I have from Pieter Abeel stating that offline RL is 'harder'.
Also you have a pretty good answer, I may accept it but just wanted to wait & see what other people say first.

@nbro I edited the original question to try to state what I mean more clearly. I'm sorry I probably shouldn't have placed emphasis on optimal control & MDPs. I think the Q-table example I gave is perhaps a better way of stating the question.

@NeilSlater Actually probably the only thing I'm still stuck on is the Q-table example I gave. In such an admittedly trivial/hypothetical situation, it is so obvious that extra (correct) data shouldn't be harmful. Is the difference here specifically because we are using a NN to approximate the table?

@profPlum There are many ways to store and update using historical data from trajectories. You would need to be more precise about the algorithm you intend to use before I could answer you about potential harm. For instance, a simple implementation of single-step SARSA would have problems because the next action $a'$ is required, and may be incorrect in the historical data, causing the TD target to be incorrect. You could resample $a'$ or used Expected SARSA, but that changes your algorithm from on-policy to off-policy.

@profPlum Now that you think you know what you want to ask, can you please remove all the unnecessary parts of this post and just leave what you think is clear and your actual question? Right now, I still don't understand what's really being asked and I also don't understand your example. How is "Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance" true? It makes no sense to me this statement. Do you know the difference between online and on-policy? If not, read this before proceeding.

From the title, it seems that you're asking "Why can't we use old data for on-policy algorithms?". If that is your question, I guess you're trying to understand the difference between on-policy and off-policy. If that's not your question, then I have no idea of what you're asking.

there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent on the state.

The desired end-state of the trained model is that it can identify the optimal action. When comparing reinforcement learning (RL) methods, you are looking at how they use data from experience. This data does not directly identify optimal actions, and all RL control approaches perform significant processing of the raw data in order to learn about the environment and make this identification.
Any rollout may contain useful data that can be learned from for control, but it requires processing in order to drive a meaningful policy. When comparing on-policy and off-policy, you are looking at two different approaches to process this data.
The differences between on-policy and off-policy methods are to do with the effectiveness of the learning algorithm in particular scenarios, and not related to using MDPs as a framework. Instead the existence of the two approaches (and other ways to solve reinforcement learning problems) are a logical consequence of the MDP framework.

I cannot understand how old data would be harmful.

When learning for optimal control, then the policy must change in order to improve. Data collected under old policies has a different distribution and results from data expected under the current policy. You can often adjust for this and still learn something (which may or may not be useful to the rest of the task) - that is exactly what off-policy learning does.
Any data can be harmful in machine learning if your learning model makes incorrect assumptions about it. That is the case when an on-policy learning algorithm attempts to use data collected under a different policy. This also applies to many other types of machine learning - if something is different in an old data set to your target distribution, the trained model will be biased towards old data and make less accurate predictions on new examples.

if optimal control were indeed a MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data

This is not true in general. Even off-policy algorithms need there to be some useful relationship to the current target policy. Data from completely random rollouts in complex problems is rarely useful for learning, except as an initial bootstrap when nothing is known about the environment.
Assuming there is some useful data that could be extracted, then there is no problem training an agent, provided that agent adjusts correctly for the difference in policy. That requires off-policy agents, because on-policy agents do not have the ability to make the adjustment.


text:
In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?
In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated

actions [may be] chosen from a normal (Gaussian) distribution.

However, I can't seem to find the justification for choosing a Gaussian distribution. It seems somewhat presumptive. I understand we choose a distribution we can sample from to model a continuous action space but not why we choose a specific distribution.
Why not fit a more complicated distribution instead of just learning the mean and std. dev. of the Gaussian?
I've noticed throughout different implementations and papers, the distribution is assumed Gaussian.

you may find this unsatisfactory but the answer is down to simplicity. the gaussian has lots of nice properties and is well understood. personally I think one of the main reasons is for the ease in implementing a re-parameterisation trick if needed, which is typically done in Stochastic Actor Critic. It is also nice to be able to learn directly a parameter of a distribution which corresponds to the mean IMO (but of course this isn't unique to the Gaussian distribution)


text:
What method is better to use for a two-player reinforcement learning environment?
I want to create an RL agent for a mancala-type two-player game as my first actual project in the field. I've already completed the game itself and coded a minimax algorithm.
The question is: how should I proceed? Which is the better way: to create a custom OpenAI Gym environment and use stable baselines algorithms or create an AlphaZero-like Monte-Carlo Tree Search algorithm from scratch?
People here suggested that it is easier to create MCTS that use Gym, since the latter does not natively support multiplayer games. But I thought I could use my minimax algorithm and incorporate it into my custom environment, and since I have both the game and the minimax algorithm, it's easier to use Gym than MCTS.
Are there any pitfalls I should avoid?

The Method for solving a multiplayer game and the environment should be independent. In other words, if you design an environment (let's say an openai gym object) then it should be compatible with model-based and model-free methods. From your question it seems that the env is going to define the method which is incorrect. An env can be repreresented as an MDP (or a multiagent MDP (MMDP)) or other decision process. Then the algorithm (method) will solve the specific MDP by interacting with the env (2 separate objects interacting).

If you're asking "which libraries should I use for my project?", this seems to be off-topic here. Programming questions as well as questions asking for tools are off-topic. However, if you're looking for an "approach" or "algorithm", that may be on-topic. Please, take the time to read our on-topic page.


text:
Is it effective to use deep learning method to produce a 1D signal as output from a 2D image as input?
I have a 1D signal that will produce a 2D image after some image processing algorithm. Would it be possible and effective to use deep learning method to reproduce the 1D signal if I have the 2D image instead? If yes, what kind of deep learning neural network will be effective in this case?
In my application, I am looking at the noise images of the 2D synthetic aperture radar (SAR) images. Each image should look like random noise image (eg. tv static noise). I have the image processing algorithm that enables me to convert a 1D (or even a 2D) complex signal into that noise image. I am curious if it is possible for me to do the reverse process using deep learning instead because reversing the image processing algorithm is a complicated task. For dataset wise, I will randomly generate the 1D (or 2D) complex signals as the ground truths and the noise image as the training data. With the image processing algorithm, there should be no issue in generating the datasets.
Is there any relevant research / Github working on this method?

Can you give more details about your specific problem? Why do you want to convert the image to the 1d signal? What task are you trying to solve? What have you found so far?

Thanks for the reply. I am trying to retrieve a 1D pulsed signal (radar) from a 2D synthetic aperture radar (SAR) image. Nothing has been done on my part yet. I am merely trying to establish a proof of concept.

I am not familiar with "2D synthetic aperture radar (SAR) images", but if there are patterns that can be used to convert the 2d images into 1d signals, it may be possible to do it with machine learning. I think it could help if you show us an image of "a 2D synthetic aperture radar (SAR)" and what you wish to obtain from that image, i.e. what exactly would be the 1d signal that you expect. Having said that, remember that ML is about learning from data. So, how much data do you have? Do you have labelled data?

In my application, I am looking at the noise images of the 2D synthetic aperture radar (SAR) images. Each image should look like random noise image (eg. tv static noise). I have the image processing algorithm that enables me to convert a 1D (or even a 2D) complex signal into that noise image. I am curious if it is possible for me to do the reverse process using deep learning instead because reversing the image processing algorithm is a complicated task. For dataset wise, I will randomly generate the 1D (or 2D) complex signals as the ground truths and the noise image as the training data.

I would recommend that you edit your post include these details because comments are temporary and are only meant to ask for clarification. In addition, after editing, your question will pop up as an active post and people will maybe read it again.

One thing that I think is still missing from your description in the post is how the image processing algorithm that converts 1d to 2d noise images works. If your 2d images are just noise and there's no pattern in them, I doubt you can do what you want. It may also be a good idea to provide the example of a 2d image that you have in mind and the corresponding 1d signal. Just to have a better idea of the types of inputs and outputs that you have.

@nbro, yes thanks for the suggestions. I will edit accordingly.

I am working on the same problem like 2d image to 1d signal. I also generated a 2d synthetic image from a 1d signal in plasma science and now try to reverse this. Is there any deep learning modal that can help with this? Please guild me.

@DhruvilGorasiya Hi there. It has been quite a while. Eventually, I decided to make use of this repository in GitHub https://github.com/LLNL/macc. The research was also built upon a plasma physics problem. The main idea of the deep learning model is to recreate a 2D image from a set of parameters. To do this, a subnetwork is trained to take in a 2D image and then convert it into a 1D latent space before converting back in the 2D image. Although this may not be specifically what you need, I hope this can give you a direction.


text:
Is there an approach where the output of one neural network is used to choose the next neural network?
I'd like to design a deep learning architecture in which the output of a primary neural network $M_{\theta}$ determines which neural network $N^i_{\alpha}$ in a set of secondary networks $\mathcal{N}$ to use next. For example, $M_{\theta}$ could be a multiclass classifier, where the predicted class determines $N^i_{\alpha}$. The networks may have different dimensions and activation functions. Is there a name for this type of architecture?

Mixture of Experts might be what you are looking for.
A Mixture of Experts model (MoE), divides a task into subtasks and designs seperate models for each of the tasks (This would be N in your case). It also defines a gating model to decide which expert to use, and during inference it uses the gating model output to pool/select predictions and makes the final decision.

https://en.wikipedia.org/wiki/Mixture_of_experts
https://www.cs.toronto.edu/~hinton/csc321/notes/lec15.pdf

Can you please provide more details about these "Mixture of Experts"? On our site, we expect users to provide more details than just a name and a link to an external resource. If you don't feel like providing more details, I can convert this answer to a comment.

Sorry for the late response, I was busy last night and completely forgot about this question. I added some more details as you said.

I am not sure if "mixture of experts" is a widely used term. Maybe you mean "ensemble learning"?

I think mixture of experts is more suitable for the op's question. In Ensemble Learning, the seperate models doesn't necessarily have divided problem spaces.

Thanks. This is what I was seeking. The main drawback here is that each of the expert networks is evaluated for each input. In my architecture, these networks could be arbitrarily complex, and we may only want to use one expert for a given input. Anyway, good suggestion.

I am not sure if "mixture of experts" is a widely used term.

The term is used widely in ML literature. Mixture of experts is a type of ensemble learning.


text:
What is the difference between Mean Teacher and Knowledge Distillation?
I recently read two papers:

BYOL Bootstrap your own latent: A new approach to self-supervised Learning

DINO Emerging Properties in Self-Supervised Vision Transformers.

I am confused about the terms Mean Teacher in BYOL and Knowledge Distillation in DINO.
Is KD the same as MT but using the cross-entropy loss instead of mean square error (since MT has preditor head while KD only has softmax head)?

Knowledge Distillation refers to using a teacher model and distilling its knowlege to a student model, mostly done by the teacher providing soft labels for the student model to create loss. So basically it defines the action of using a teacher model to teach the student.
Mean Teacher, on the other hand, is one way of how one would define/train the teacher (especially in self supervised learning). Usually in supervised learning, one can train the teacher using existing labels, but in self supervised learning this is not possible. Mean Teacher defines the teacher as a model that utilizes the weighted average of the student's past weights. So basically it is one methodology of defining the teacher in self supervised knowledge distillation.

Thank you for your comment ... So in MT, there is no distilled knowledge (e.g : transfer set of soft labels) from the teacher to the student because the teacher also needs to learn? And both teacher and student are trained together via parameters update (from loss  backprogation for the student and EMA for the teacher)

@ĐặngHuyHoàng correct

@ĐặngHuyHoàng this is an answer, and not a comment; please see What should I do when someone answers my question?


text:
What is the expression for projective transformation?
The following are the two types are projections that are generally used in image processing

Affine transformation
Projective transformation

Affine transformation is a backbone operation in neural networks also. It is expressed as
$$\mathbf{wx+b}$$
where $\mathbf{w, x, b}$ are matrices. In general, $\mathbf{x}$ is treated as an image in image processing.
Projective transformation is also a type of transformation on images and it may be different from affine transformation. I want to know whether it can be represented in terms of mathematical expression.
If yes, what is the expression for projective transformation?

What do you mean by "projective transformations"? Are you talking about homographies? Maybe you could provide a little bit more context. Projective transformations definitely appear in the context of computer vision and computer graphics, but if you provide more context, it may be better. Anyway, maybe later I provide an answer.

I explain in this answer what a projective transformation (aka projectivity or homography) is. It's a function $h$ of the form $$h: \mathbb{P}^2 \rightarrow \mathbb{P}^2,$$ where $\mathbb{P}^2$ is a projective space, so, essentially, a 3-dimensional Euclidean space of homogenous vectors.
You can also represent a homography as a $3 \times 3$ matrix $\mathbf{H}$, so that, when we apply this projective transformation to some input $\mathbf{x} \in \mathbb{P}^2$, we get $\mathbf{x}' \in \mathbb{P}^2$, so we can represent a projective transformation as follows.
$$\mathbf{H}\mathbf{x} = \mathbf{x}'$$
So, basically, a projective transformation is a linear transformation between projective spaces.
You can generalize these ideas to higher-dimensional projective spaces, i.e. $\mathbb{P}^n$.
Although you can represent a projective transformation as a matrix multiplication, there's more to it. In fact, it's a linear transformation, with 8 degrees of freedom, between projective spaces. You can also view a homography as a generalization of other transformations, like isometries, similarities, and affinities. This is explained more in detail in chapter 2 of the book Multiple View Geometry in Computer Vision (2nd edition) Richard Hartley and Andrew Zisserman.


text:
Is it true that batch size of form $2^k$ gives better results?
I am confused among the following in selecting the batch size for my model.
#1: powers of 2
I generally see that batch sizes are in powers of two: 32, 64, 128, 256.
#2: maximum GPU
Suppose my GPU allows a maximum batch size of 61. And it is not a power of two.
Which one should I apt? Is there anything like the powers of 2 will give relatively good results?

It seems that this question is a duplicate of this, which, however, seems to be more than one question, so it might be closed as too broad. I would recommend that you search a little bit on the site before asking a question. Many questions have already been asked, but, of course, the answers may not be good and even the questions may be poor, and can be deleted. If you think they are poor, you can ask the question again and state that.

@nbro Yeah, I missed the question because of tags assigned for that question. I think it is related but not duplicate. The intention of this question is to select which one: either #1 or #2, if can't both.

What do you mean by select #1 or #2? #1 is the batch size, #2 is the GPU, so what does that mean? I think they are exact duplicates. Here, it seems that you're asking why do we use powers of 2. If that's not the question, then the answer below doesn't answer your question. Your question in the title is "Is it true that batch size of form $2^k$ gives better results?", which seems to be the same as "why do we use powers of 2?"

I mean, suppose my GPU allows at most 61 as batch-size. Then should I opt for 32 or 61 is the question. @nbro

Here, you're actually asking multiple questions. One of your questions is: "Is there anything like the powers of 2 will give relatively good results?", which is a duplicate of that one, which the answer below answers. Your other question is "Which one should I apt?", which is unclear, but your second formulation in the comment above is better, so I would recommend that you edit your post to leae that formulation "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?"

"Which one among the both should I consider for the batch size" is the first title I kept. But, for readability, I changed the title. @nbro

Actually, I would recommend you ask that question in a separate post. I think that question "Suppose my GPU allows at most 61 as batch size. Then should I opt for 32 or 61?" is distinct enough to be asked in a separate post, taking into account the explanation in the answer below.

Okay @nbro.  I will do accordingly.

But maybe it's not worth it, if you ask the OP below to address that question too.

The choice of the batch size to be a power of 2 is not due the quality of predictions .
The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for batch_size=61 and batch_size=64, since the amount of stochasticity would be of the same order of magnitude.
The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of CUDA streaming multiprocessors).
If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.
So the reason for batches to be powers of 2 is about efficient utilization of GPU.

When you write "However, there is not much different say in the behavior for training with", I would write "For the purposes of optimization with SGD, there might not be much different in the behavior for training with say .... However, this is in theory. In practice, GPUs might have memory whose size is a power of 2..." etc

The main reason to use powers of 2 is in the way existing hardware and software are made, there isn't any purely mathematical reason. CPUs, GPUs, memories, and internal buses all use a size that's the power of 2 since that's the most efficient way to address them.


text:
Can I use a Mask R-CNN to detect a skin texture?
I'm trying to implement a solution in python to detect skin in an image.
I'm evaluating the Mask R-CNN model to create a mask on the skin (not on clothes). The problem is that every solution I have encountered using Mask R-CNN uses it to classify objects. I'm afraid that using it trying to classify texture might be a problem. Is it the case?
My dataset is actually pretty good, composed of the

original image,
precise mask on the skin, and
grounding box.

Can I use a Mask R-CNN to detect a skin texture?


text:
How does Horn–Schunck method for Optical Flow solve the aperture problem?
This is regarding the details stated in Wikipedia.
I am reading optical flow in Computer Vision. I understood the Horn–Schunck method as such, but did not get how it is related to the aperture problem, and how it is solved using Horn–Schunck method.
Also, why is Horn–Schunck method invented/used where a simpler "Lucas–Kanade method" is already there (Reference)?

The estimation of optical flow by the Horn-Schunk method can be written in matrix form as (for the sake of clarity, assume no regularization, that is, $\alpha = 0$)
$$
\left(
\begin{array}{ll}
I_x^2 & I_xI_y \\
I_xI_y & I_y^2 \\
\end{array}
\right)
\left(
\begin{array}{ll}
u \\ v
\end{array}
\right) =
\left(
\begin{array}{ll}
-I_x I_t \\ -I_y I_t
\end{array}
\right)
$$
where $(u, v)$ is the estimated displacement between two frames.
You can solve this linear system of equations if the matrix on the left is not singular, i.e. its eigenvalues are not zero. But this means that gradients in both directions are not zero. And this is what the aperture problem is about: when the matrix is singular.
As an example, consider a black square aligned along the coordinates axis, moving horizontally (along x axis) on a white background, and you want to estimate the displacement of the square from the optical flow. Furthermore, you do not consider the whole image, but just some rectangular area of the image.
Now, if your rectangle does not contain any of the edges perpendicular to the movement direction, you will not be able to estimate the movement because your gradient estimates will be $I_x = I_y = 0$.


text:
What is the relevance of the concept size to the time constraints in PAC learning?
My question is about the relevance of concept size to the polynomial-time/example constraints in efficient PAC-learning. To ask my question precisely I must first give some definitions.
Definitions:
Define the input space as $X_n = {\left\{ 0,1\right\} }^n$ and a concept $c$ as a subset of $X_n$. For example, all vectorized images of size $n$ representing a particular numeral $i$ (e.g. '5') collectively form the concept $c_i$ for that numeral. A concept class $C_n$ is a set of concepts. Continuing our example, the vectorized numeral concept class $\left\{ c_0, c_1, \dots c_9\right\}$ is the set of all ten vectorized numeral concepts for a given dimension $n$.
As an extension to include all dimensions we define $\mathcal{C} = \cup_{n \geq 1} C_n$. A hypothesis set $H_n$ is also a fixed set of subsets of $X_n$ (which might not necessarily align with $C_n$) and we define $\mathcal{H} = \cup_{n \geq 1} H_n$.
The following definition of efficient PAC-learnability is adapted from An Introduction to Computational Learning Theory by Kearns and Vazirani.
$\mathcal{C}$ is efficiently PAC-learnable if there exists an algorithm $\mathcal{A}$ such that for all $n \geq 1,$ all concepts $c \in C_n$, all probability distributions $D$ on $X_n$, and all $\epsilon, \delta \in \left(0,1\right)$, the algorithm halts within polynomial time $p{\left( n, \text{size}{\left(c\right)}, \frac{1}{\epsilon}, \frac{1}{\delta}\right)}$ and returns a hypothesis $h \in H_n$ such that $$ \underset{x \sim D}{\mathbb{P}}{\left[ h{\left(x\right)} \neq c{\left(x\right)}  \leq \epsilon\right]} \geq 1 - \delta.$$
Question:
Now, in the polynomial $p{\left(\cdot, \cdot, \cdot, \cdot \right)}$, I understand the dependence on $\epsilon$ (accuracy) and $\delta$ (confidence). Additionally, I understand why the polynomial should depend on $n$ - the concept of learnability should be invariant to the time burden incurred from increasing the dimension of the input space (e.g. increasing the resolution of the image). What I do not understand is why the dependence on the size of the target concept (which I believe is usually taken to mean the smallest encoding of the target concept)?

The size of a concept representation is important because the time to write/process this representation is a lower bound on the running time of the learning algorithm.
For example, you can represent the boolean parity function with a circuit composed of $\land$, $\lor$ and $\neg$ such that its size is polynomial in $n$, while, if you used a disjunctive normal form (DNF) representation, you might need an exponential number of terms. This may give you some intuition behind this example.
For some concept classes, $\text{size}(c)$ is already bounded by a polynomial in $n$, so the only practical requirement that you need in those cases is that the algorithm runs in time polynomial in $n$. For example, if you consider the class of DNF with at most 3 terms, then you know that the number of literals is at most $3n$ (i.e. each term can contain at most $n$ literals, either $x_i$ or $\neg x_i$ but not both, for all literals $x_1, \dots, x_n$).


text:
Why do we use $q_{\phi}(z \mid x^{(i)})$ in the objective function of amortized variational inference, while sometimes we use $q(z)$?
In page 21 here, it states:

General Idea of Amortization: if same inference problem needs to be solved many times, can we parameterize a neural network to solve it?
Our case: for all $x^{(i)}$ we want to solve:
$$
\min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right.
$$
Amortized formulation:
$$
\min _{\phi} \sum \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)
$$

One thing I am trying to wrap my mind around is $q(z)$ in the 1st formulation vs $q_{\phi}(z \mid x^{(i)})$ in the second.
Why is one a conditioned on $x^{(i)}$ and the other is not? I know in the 1st formulation, we are trying to find a different $q(z)$ for each datapoint.
I also recall that in VAEs,which uses amortized inference we consider $q(z)$ to be aggregated posterior, like
$$q_{\phi}(z)=\int q_{\phi}(x, z) d x \quad \text { Marginal of } q_{\phi}(x, z) \text { on } z$$
$$q_{\phi}(x, z) \equiv p_{\mathcal{D}}(x) q_{\phi}(z \mid x)$$
(formulas taken from here)

The reason is indeed the difference between traditional variational inference and the amortized variational inference.
In traditional VI, the objective is to solving the following problem for every datapoint $x^{(i)}$
$$\min _{q(z)} \mathrm{KL}\left(q(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$
Pay attention that if we write out explicitly the variational parameter  $\phi$, it should depend on i-th datapoint (hence we call them local parameters), i.e. we are in fact solving N problems provided N is the number of datapoints.
$$\min _{\phi_i} \mathrm{KL}\left(q_{\phi_i}(z) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$
while for amortized VI, we try to find a global variational parameter that serves for the inference of all the datapoints, i.e.
$$\min _{\phi} \sum_{i=1}^N \operatorname{KL}\left(q_{\phi}\left(z \mid x^{(i)}\right) \| p_{\theta}\left(z \mid x^{(i)}\right)\right)$$
This paper explains the difference with better notations.


text:
Is there any relation between the recursive neural network and recurrent neural network?
Recurrent neural networks, abbreviated as RNNs, are widely used in deep learning literature, especially for text processing.
Are they related to recursive neural networks in any way?
I am asking for the general/special relationship that enables us to view the one in terms of another if possible.

The Wikipedia articles on the two types of network look like a good place for someone to start an answer.

@NeilSlater They seem to be different based on that, but at some places (like proofs) they are referring to the same...

Form the articles, recurrent are a sub-type of recursvive. I hope that someone who knows more about this can answer in depth.

Yes, recursive neural networks (recursive NN) are related to recurrent neural networks (RNNs), because they generalize the latter (at least, structurally), as stated in section 10.6 of the Deep Learning book, similar to the way that a tree generalizes a list/array.
So, in a list/array, element $e_1$ is connected to element $e_2$, which is connected to element $e_3$, and so on. In a tree, element $e_1$ can be connected to more than one other element (node), and the same applies to the other elements, provided there are no cycles, otherwise, you get a graph, which is a generalization of a tree.
If these explanations weren't yet clear, the best way to understand the difference between the two is to look at the diagrams of an RNN and a recursive NN.
Here's an RNN.

The caption should describe the diagram (in case it is not clear). The important thing to note is that, on the left, we have an RNN, which looks like a list.
Here's a recursive NN.

which looks like a tree.
There are many variations of RNNs (e.g. Bidirectional RNNs, multi-layered RNNs, LSTMs, or GRUs) and recursive NNs (e.g. that attempt to have balanced trees, similar to the way that a Red-Black Tree is a balanced binary search tree), so we cannot give all the details about the differences between these two approaches.
However, a few things should be kept in mind

recursive NNs might be used in the context of parse trees
recursive NNs have been used to process data structures as input to neural nets
nodes in recursive NNs don't necessarily perform a linear operation followed by a non-linearity


text:
Is the capability of RNN more than the capability of MLP?
Consider the following excerpt paragraph taken from the section titled "Recurrent Neural Networks" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al on the computational graph of some computations.

The recurrent neural network of ..... is universal in the sense that
any function computable by a Turing machine can be computed by such a
recurrent network of a ﬁnite size. The output can be read from the RNN
after a number of time steps that is asymptotically linear in the
number of time steps used by the Turing machine and asymptotically
linear in the length of the input. The functions computable by a
Turing machine are discrete, so these results regard exact
implementation of the function, not approximations. The RNN, when used
as a Turing machine, takes a binary sequence as input, and its outputs
must be discretized to provide a binary output. It is possible to
compute all functions in this setting using a single speciﬁc RNN of
ﬁnite size. The “input” of the Turing machine is a speciﬁcation of the
function to be computed, so the same network that simulates this
Turing machine is suﬃcient for all problems. The theoretical RNN used
for the proof can simulate an unbounded stack by representing its
activations and weights with rational numbers of unbounded precision.

This paragraph clearly explains that RNN is capable of computing any computable function exactly and is the same as the Turing machine in terms of capability.
Afaik, MLP is capable of approximating any continuous, bounded function.
So, it seems to me that RNN is more powerful in terms of the capability of computing functions than MLP. RNN can learn any function that MLP can learn and RNN can learn more than that can be learned by any MLP in general.
Am I correct? Or is there any issue in my interpretation or do more details need to be considered?

The way MLP works usually work with sequences restricts to the fixed length sequence, since feature vectors from multiple timesteps are concatenated to single vector of big dimension. RNN can handle arbitrary long sequence on the other hand.

@spiridon_the_sun_rotator But, but I think it is possible to assume an MLP with infinite input length.


text:
How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?
I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected blocks and your goal is to complete all the required objectives (collect X color blocks, destroy 30 balloons, etc).
I have tons of levels and the ML solver seems to perform very well for all kinds of obstacles - except Quicksand.
Quicksand is a special object that replicates itself to a nearest tile whenever a user makes a move. If the user destroyed a quicksand in his turn, a quicksand won't be replicated. So basically the fastest way to destroy quicksand is to make sure you destroy as much quicksand as you can in each turn so it won't replicate and cover your board.
I use ML-Agents from Unity (https://github.com/Unity-Technologies/ml-agents) and I just give the agent reward=1f whenever it completes an objective (destroy 1 obstacle) and I subtract 1f from reward whenever it performs a move.
For simpler non-replicating obstacles it works perfectly. For example, you click 2 blocks next to a balloon - it will pop a balloon, add 1f as a reward and at the same time remove 1f for using a move.
This way the agent learns to make as few moves as possible.
Below you'll find how the Quicksand works and some simple obstacle - Balloon.
Quicksand preview (sorry for bad quality, 2mb max size)

Balloon preview

My issue is that no matter what, I can't teach it to solve quicksand. With the above rewarding approach, I think this strange creature learned that by actually REPLICATING quicksand it's gaining more reward because it can destroy it later (actually it's not because moves give -1f so it's more-less equal).
I've tried not giving reward for the quicksand, so it only loses rewards by using moves. But it doesn't work either, I'm not sure why.
Do you guys have any idea how this kind of things should be taught?

Do you know which RL are you using under the hood with ml-agents? It may be a good idea to investigate that if you don't know it. Having said that, if I understand correctly, you have 2 distinct environments: Ballon and Quicksand. If the objective in these environments is different, you should probably change the reward function, but it seems that you're using the same reward function for both. Right?

I wonder if it would help to make the reward a function of the change in obstacle tile count following the move. So allowing an obstacle tile to replicate is punished more severely than simply neglecting a non-replicating obstacle.


text:
How to do testing for an RNN that was trained with teacher forcing only?
If an RNN is trained using only the teacher forcing, then the network takes the actual output from the previous time step as input to the hidden state the next time step.
We know that the actual outputs cannot be given to the model while testing, then what information passes from a time step to the next time step in the test phase?

I don't think there's any difference between making predictions when you use or not teacher forcing during training. So, let me describe one way of doing that.
During testing, as you noticed, you don't know the ground-truth labels, so the only way of predicting a sequence is to feed into the model the predictions for the previous time-steps.
So, let's say that your model is denoted by $f$, which attempts to approximate the language model $p(x_t \mid x_{1:t-1})$, a conditional probability distribution over characters.
To perform inference, you first need to provide the initial character of the sequence, usually, a special character, like $\langle \text{start} \rangle$, so your model predicts the first real character of the sequence as follows $$f([\langle \text{start} \rangle]) = \hat{x}_1.$$ Afterward, you feed to it $\hat{x}_1$, in order to produce the next character, i.e. $$f([\langle \text{start}\rangle, \hat{x}_1]) = \hat{x}_2,$$ and so on, until the model $f$ produces another special character that denotes the end of the sentence $\langle \text{end}\rangle$.
Now, in practice, rather than all the previously predicted characters, you may just feed the character $x_{t-1}$ to predict $x_t$, and, at the same time, pass the previous last state of the RNN, so something like $$f (\hat{x}_{t-1}, h_{t-1}) = \hat{x}_t.$$
The following diagram illustrates the concept.

Here the first char is $T$ and the model predicts that the next char is $e$. Then we pass $e$ and the previous state of the model to the model to predict $n$, and so on.
Here you have a TensorFlow example that generates text with an RNN, uses teacher forcing, and then performs inference in the way I described (of course, there are other details, like using a temperature).

Then the architecture of RNN is changing, right?

@hanugm No, the architecture doesn't change, at least, not usually. What changes is what you pass as input to the model. The hidden state of a NN can be thought of as containing the information that would allow you to make a prediction based on the previous inputs or history, at least, this is the intuition. If you look at the linked code (TF example), maybe you will get a better sense of what's happening.

In that example, basically, you can make the RNN (in that case, a GRU) return you the last state of that layer after having taken the input. Then you use that last state as some kind of "context vector" for predicting the next char, and that GRU produces another "context vector" (i.e. the hidden vector), which you use to make a prediction at the next time step, and so on.


text:
Generative systems based on Schmidhuber's compression framework
In Driven by Compression Progress:
A Simple Principle Explains Essential Aspects of
Subjective Beauty, Novelty, Surprise,
Interestingness, Attention, Curiosity, Creativity,
Art, Science, Music, Jokes Jürgen Schmidhuber describes, using the idea of using compression progress on perceived data as an intrinsic reward signal:

[Artists] create action sequences yielding interesting inputs, where interestingness is a measure of learning progress, for example [...] the saved number of bits needed to encode the data.

Has such a (purely generative) system been created before? Say, one that creates 2d images?

Hello. Welcome to Artificial Intelligence Stack Exchange. Are you familiar with or heard of (variational) autoencoders?

@nbro Although variational autoencoders inherently use compression (and better compression arguably leads to better results in many cases), the loss signal is not directly related to degree of compression, but fidelity of reproduction using a fixed size compression. I think someone could answer using VAEs as an example, but it should be caveated as these don't quite reach Schmidhuber's design.

@NeilSlater Well, the KL divergence and cross-entropy (which form the ELBO objective function to train VAEs) can have an information-theoretic perspective. See my answer if you are interested in more details. However, I don't know the details of Schmidhuber's design, but this answer also doesn't provide more details. That's why I was asking if the OP was aware of VAEs. If yes, maybe he had in mind something different and could provide more details about Schmidhuber's design.

@NeilSlater By the way, if I remember correctly, World Models (co-authored by Schmidhuber)  use the VAE. So, maybe that goes more in this direction, as the OP is talking about rewarding and intrinsic rewards. Anyway, I think this question is vague, so it needs more details to be answered appropriately, so I don't plan to give an answer now.


text:
What is the best machine learning algorithm for clustering dots based on coordinates $(x,y)$ with consideration of weight of the points?
I'm looking for a machine learning algorithm for clustering points based on their coordinates. Furthermore, I want to take into consideration the weights of each point. Suppose there is a weight in each point. Then we take the sum of the weight of all the points in a cluster. I want the sum in different clusters to be close and balanced. What algorithm is best for this? Any suggestion?

Maybe try to use weights as 'z' coordinate and perform k-means clustering. I don't know if that's the best approach though.

if z is weight, then all the points with greater weight will be packed together isn't it? Then the sum of them will be the biggest? Am I right?

Oh, I misunderstood Your question. You just have (x, y) coordinates and want to perform basic clustering? Just google clustering algorithms like k-means, DBSCAN, BIRCH or mean-shift. You don't need to add any weight to Your data.

ok, this solves my first problem. But what if each point has a weight. So how I make the total weight in a cluster is similar to others? Using kmeans,.... will only consider their locations and cluster them together and not considering the weight of the points.

just assume the weight as a third parameter. However, I cannot simply put it as 'z' coordinates and do clustering. Because this will pack those 'heavier' points together and making clusters not balanced

Okay, from the beginning. You have (x, y) points and want to cluster them. You've done this by k-means and that's great. Now You want to add weight as an 3rd parameter. If You want to perform clustering on that points, but not taking weights as an 'z' coordinate pls provide more information what this 'weight' means to You. Maybe some more specific information about dataset and problem itself, because there could be some better solution than simple clustering.

ok, let say each point is a cell station and has traffic (this is my 'z'). (x,y) is the latitude and longitude. So with pure k mean I can cluster the stations that are close to each other. But some clusters may have very large total traffics compared to others. (if the heavy traffic stations are closely packed.) Is it possible i can make the clusters balanced by minimizing the difference of the total traffics between clusters.

Okay, and how about looping over possible k for k-means and calculate the mean of the clusters in respect to weights. After that You could calculate std dev of collected means. Choose number of clusters with the smaller std dev. That might be not computationally efficient, but You could find good solution in this way.

this idea is pretty inspiring, thanks for your help.


text:
Can teacher forcing in RNN ensure Turing completeness?
RNN has the same capability as a universal Turing machine. But I am confused whether RNN holds the same capabilities if we use teacher forcing.
Consider the following excerpts from paragraphs taken from the section titled "Teacher Forcing and Networks with Output Recurrence" of the chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al.

The network with recurrent connections only from the output at one time step to the hidden units at the next time step is strictly less
powerful because it lacks hidden-to-hidden recurrent connections. For
example, it cannot simulate a universal Turing machine. Because this
network lacks hidden-to-hidden recurrence, it requires that the output
units capture all the information about the past that the network will
use to predict the future....... Models that have recurrent
connections from their outputs leading back into the model may be
trained with teacher forcing.

The quoted portion says that the RNN in which recurrent connections only exist from the output at one time step to the hidden units at the next time step are less powerful and are not as capable as a universal Turing machine. And those neural networks can be trained with teacher forcing. Even though they are not trained using teacher forcing, they are not as capable as the universal Turing machines. But, I want to get clarity on the relation between the capability of RNN trained using teacher forcing and the capability of universal Turing machine.
Is it true that if an RNN is trained with teacher forcing then it cannot simulate a universal Turing machine?

For an RNN to be trained using teacher-forcing, the recurrence in time must be in the form of output-to-hidden. This is a special case of the more general RNN structure that allows hidden-to-hidden recurrence. The latter can approximate a universal Turing machine with a large enough hidden state, however, the output-to-hidden RNN cannot do so.
This is fundamentally because the sequence of output does not uniquely specify the future behavior of a Turing machine. The same argument can be made for autoregressive Transformers and time-delay neural networks (TDNN). They are less expressive.


text:
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, s_t, \theta)
\end{align*}
where $\hat A_t$ is the advantage estimate for step $t$. For example, $\hat A_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ in the one-step actor-critic algorithm given in section 13.5.
In the answers to the linked question, it is claimed that the extra discounting is "correct", which implies that it should be included.
If I look in the literature to a seminal paper such as Proximal Policy Optimization Algorithms by OpenAI, they do not include the extra discounting factor, i.e. they use a gradient defined as
\begin{align*}
\hat A_t \dfrac{\nabla_{\theta}\pi(a_t \, | \, s_t, \theta)}{\pi(a_t \, | \,s_t, \theta_{\rm old})}
\end{align*}
which does not include the discounting factor (of course, it's dealing with the off-policy case, but I don't see how that would make a difference in terms of the discounting). OpenAI's implementation of PPO also does not include the extra discounting factor.
So, how am I supposed to interpret this discrepancy? I agree that the extra discounting factor should be present, from a theoretical standpoint. Then, why is it not in the OpenAI code or paper?

If you want to maximize the expected reward
\begin{align*}
\mathbb{E}\bigg[\sum_{t=1}^nr_t \bigg]
\end{align*}
and are using a score function based gradient estimator (as opposed to a SAC/DDPG style update), you have the unbiased gradient estimator
\begin{align*}
\sum_{i=1}^n \sum_{k=i}^n r_k\nabla_{\theta}\log\pi(a_t) \tag{1}
\end{align*}
Then, you can add discounting as a variance reduction technique; the gradient estimator
\begin{align*}
\sum_{i=1}^n \sum_{k=i}^n \gamma^{k-i}r_k\nabla_{\theta}\log\pi(a_t) \tag{2}
\end{align*}
will have a lower variance than Eq. (1) (see this answer).
If you want to maximize the discounted expected reward
\begin{align*}
\mathbb{E}\bigg[\sum_{t=1}^n \gamma^{t-1}r_t \bigg]
\end{align*}
you get the unbiased gradient estimator
\begin{align*}
\sum_{i=1}^n \sum_{k=i}^n \gamma^{k-1}r_k\nabla_{\theta}\log\pi(a_t) \tag{3}
\end{align*}
So in Sutton and Barto they are essentially presenting the formulation Eq. (3). The difference between (2) and (3) is the factor of $\gamma^{i-1}$ which is what I was confused about in the question.
Thus in summary, the formulation (2) is an biased estimator of the expected reward; whereas (3) is an unbiased estimator of the expected discounted reward.

I believe you will find the answer in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation, which is the basis for the advantage function used in the PPO paper that you referenced.
From the paper, the estimate of the advantage function is defined as:
\begin{align*}
\hat{A}_{t}^{GAE(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+1}^{V}
\end{align*}
where $\delta_{t}^{V}$, the TD residual of $V$, is defined as:
\begin{align*}
\delta_{t}^{V} = r_{t}+\gamma V(s_{t+1})-V(s_{t})
\end{align*}
where $V$ is an approximate of the value function.
If you look closely at these two equations you will see that the discount $\gamma$ is applied twice.
I never went through the code of the whole OpenAI implementation of PPO, but if I am not mistaken the implementation of the above equations can be found here in ppo2/runner.py.

This does not answer the question. I am familiar with GAE and the equations you give are consistent with the PPO paper. The question is whether or not it is supposed to be $\gamma^{t-1}\hat A_t$ (which is what Sutton and Barto give and what I would expect accordingly to theory); or whether it should be just $\hat A_t$, which is what is present in the PPO paper, PPO code, and GAE paper.

Apologies for not being clear in my answer. From my understanding of the GAE and PPO implementation, the discount factor is there twice like in S&B, it is just not applied in the same order.

You have given the definition of $\hat A_t$. I agree with the definition. The question is whether the gradient of the (log probability) should use the coefficient $\hat A_t$ or $\gamma^{t-1} \hat A_t$.

No, I don't believe that $\gamma$ should be applied to the gradient in $\hat{A}{t}\frac{\nabla{\theta}\pi(a_t , | , s_t, \theta)}{\pi(a_t , | ,s_t, \theta_{\rm old})}$ for the reason that it is applied in the calculation of the estimate of the advantage and in the calculation of $\delta^{V}_{t}$ .


text:
How can I calculate the parameter $w$ in the third condition of LVQ 2.1 algorithm?
I'm developing a neural network software using several NN architectures including LVQ family. I met a parameter that is used in the 3rd condition of LVQ2 and later versions. It's named $w$ and is used to calculate whether the input vector falls in the $w$-length window gap or not. For example, for LVQ 2.1 it's used to calculate the 3rd condition as follows:
\begin{equation}
\min(d_{i}/d_{j},d_{j}/d_{i})>\frac{1-w}{1+w},0<w<1.
\end{equation}
where

$d_{i}$ is the minimum Euclidean distance from input vector (1st winner neuron's distance)
$d_{j}$ is the 2nd minimum Euclidean distance from input vector (2nd winner neuron's distance)

This parameter $w$ is a user-determined one like learning rate or momentum, etc. I give the value of $w$ to training algorithm manually but this is a trial and error style. During my research, I found an information about this parameter may be calculated via training samples count.
Are there any information about how to calculate this parameter by training samples count or something else like represented training samples count to the network?

i asked,i answered:

Teuvo KOHONEN, "The Self-Organizing Map", Proceedings of The IEEE, Vol. 78, No. 9 September 1990 page.1471

Can you please at least copy and paste the actual text that you're showing us rather than pasting a screenshot? In addition to that, you should also summarize the main points of the quote that answer your question.


text:
Why should one focus on spectral operations as a computer vision researcher?
While reading about various types of mathematical operations on tensors, I encountered spectral operations for the first time.
The description is as follows (p. 53 of this book)

Spectral ops - Functions for transforming in and operating in the frequency domain, like stft and hamming_window

While going through different codes in computer vision, I have experience of using all other mathematical operations: Pointwise ops, Reduction ops, Comparison ops, and some others. But, I personally never come across any code using spectral operations on tensors.
So, I am interested in knowing whether one has an advantage in learning spectral operations in the domain of computer vision?

The main advantage of using spectral operations comes from the Covolution theorem, which states that the Fourier transform of the convolution of two functions is equal to the pointwise multiplication of the individual Fourier transforms of those functions.
Mathematically:
$\mathscr{F}|\it{f} \; * \it{g}| = \mathscr{F}|\,\it{f}\,| \mathscr{F}|\,\it{g}\,|  $
Of course multiplying is much more convenient than convolving, and the Fourier transform is also a well known operation, and fast to compute, so that does not represent a downside.
In practice, when it comes to computer vision, the convolution theorem become fundamental for many kinds of analytic de noising and filtering methods. Instead of applying a kernel on an image on teh spatial domain using convolution, you can simply move to the frequency domain through the Fourier transform and multiplying both, image and kernel.
Moreover, when moving to the frequency domain, it become easier to access specific characteristics of images. Sharp lines  and edges for example are characterized by high frequencies, so we can extract them with a simple masking operation, instead of having to came up with a tough kernel in the spatial space. This is the core idea of low and high pass filters, below here an example, where you can see that by masking the center (low frequencies) area of the spectra of two images, we can easily mix low frequency properties (e.g. shadows) and high frequency properties (e.g. edges) of two images. The same operation in the spatial domain would be tough if not impossible.

It may be also worth mentioning graph neural networks. Some of them exploit the convolution theorem.

thanks for the hint, not an expert on graph networks but I'll check!

In the Fourier analysis, an image is represented by frequencies and phases. This allows to filter or transform certain frequencies or phases.
For example, Alan Oppenheim and Jae Lim have shown that phases contain information about the object contours in the image. Their paper was published in 1981, so it is poorly scanned, and the images there are not clear. Better examples are in the following books:

Danilo P. Mandic, Vanessa Su Lee Goh. Complex Valued Nonlinear Adaptive Filters (2009)
Igor Aizenberg. Complex-Valued Neural Networks with Multi-Valued Neurons (2011)

Here is an example from the second book.
Consider two images, "Lenna" and "Airplane",

Take their Fourier transform, swap magnitudes and phases of their Fourier spectra, and then perform the inverse Fourier transform. The result is as follows.
Magnitudes from the "Airplane", and phases from the "Lenna":

Magnitudes from the "Lenna", and phases from the "Airplane":

So, the information about objects is contained mainly in the phase part of the spectra.
This is what happens when amplitudes are replaced by the constant 1:

This can be used to design algorithms for image processing.
More examples of applying Fourier transform to image processing is shown in the lectures of Guillermo Sapiro where he makes really impressive things with images of HIV: part 1, part 2. That Prof.'s algorithms were used in Mars rovers.
A very good explanation of spectral techniques is given in the book, Wilhelm Burger, Mark J. Burge. Digital Image Processing. An Algorithmic Introduction Using Java (2016).