text:
Why do activation functions in neural networks have to be non-polynomial to approximate any function?
Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.

Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs the derivatives of all the activation functions and thus possibly leads to unbounded gradient if any of these are mere polynomials.
And from the abstract of your referenced paper threshold plays an important role in universal function approximator, an obvious easy example is the 2 layer perceptrons (aka threshold logic unit [TLU], Adaline [Widrow, 1962]) neural network implementation of the classic Boolean even-parity function $y=x_1x_2+\bar x_1 \bar x_2$ which is not linearly separable and thus a single layer of perceptrons cannot achieve as pointed out by Minsky in the 60's，but it could be easily solved by below 2 layer perceptrons with all (bounded) step functions assigned with appropriate thresholds as shown below (you can manually try to verify). If all activation functions are mere polynomials, no matter how many hidden layers you add, these Boolean functions cannot be modeled exactly, and, thus, most practical classification problems could not be solved by neural networks.

Do you think you can provide a simple example why a polynomial nonlinearity would lead to an unbounded gradient? I don't see how that could be the case. (The derivative of a polynomial is just a polynomial, not infinity)

@StevenSagona Here unbounded just means without a pre-assigned fixed threshold, (higher order) polynomial activation functions may contain bilinear terms which could complicate and amplify the (partial) derivatives. For example see this paper about tensor network: the mathematical explanation of neural tensor networks remains a challenging problem, due to the bilinear term... we associate NTN with Taylor’s theorem and find that each slice of NTN could be represented as a 2nd order multivariate Taylor polynomial.


text:
Why could there be "information leak" if we do not use fixed horizons?
In this page Limitations on horizon length from the Imitation library, the authors recommend that the user sticks to fixed horizon experiments because there could be "information leak" otherwise.
I'm having problems understanding this term, how can the information leak?
Can someone please explain this with an example, or something that might help me get some intuition about the issue?

In the standard Cartpole environment, the episode ends either at 500 timesteps or when the Cartpole falls down. Expert demonstrations show how to keep the Cartpole upright for 500 timesteps.
One degenerate way for Generative Adversarial Imitation Learning or Adversarial Inverse Reinforcement Learning to train an expert policy is to assign a positive reward to every action taken by the generator policy, regardless of whether it mirrors the expert actions. With a positive reward, the RL algorithm that is training the generator policy learns to favor prolonging the episode.
Hence the episode length is a form of side-channel information that makes the imitation learning task easier, and we want to exclude it by choosing environments with fixed episode length.

Thank you for your answer @twink_ml. This puts me in a difficult situation, the enviroment I´m using in my project cannot have a fixed length, it could finish well before any imposed length.

You can use something like https://seals.readthedocs.io/en/latest/common/util.html#seals.util.AbsorbAfterDoneWrapper
to make your environment fixed horizon T, given that the horizon is upper bounded by T.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are some non-RL-based approaches to solving a typical bin assignment problem?
What are some non-RL-based approaches to solving a typical bin assignment problem, i.e., given a set of items (can be multidimensional), find the bin/knapsack/target which best packs (with minimum wastage of space) the items?
I tried looking for some papers on this but most of them are RL-based methods or haven't described the approach lucidly, hence am reaching out to the community. Any help in this would be greatly appreciated

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Machine Learning Methods commonly used when data are scarse
It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification problem. However, you don't always have lots of data to train your machine learning algorithm.
Which classical machine learning methods work well with little data? I'm thinking of things like KNN, linear regression, support vector machines, random forests, etc.
Is there a paper that systematically investigates machine learning methods when data are scarse? If not, are there some rules of thumb one can follow?

Currently to train state-of-the-art robots for bicycling needs millions of trials from a blank slate on average compared with much less trials for average young aged human. You may contemplate the root cause here. There seems no other way around this issue until we have true AGI to have agent continuously evolved to learn new objectives/rewards based upon previous learned policies in the RL formulation.

You are asking for sample efficiency of learning algorithms

Generally speaking, the number of training examples needed to train a neural network for your classification task cannot be determined a priori.  Generalizations such as the claim that 10K training examples needed are obtuse.  This question is complicated, but here are some considerations:

Technically, you can generate a model with very few training
examples (equal to the number of classes), but model performance
will be poor.  Thus, it is not sufficient to ask how
many training examples are needed.  It would be more apropos to ask
how many are needed to achieve a target performance level.
The number of examples depends on the complexity of your task, the expressivity of your model, and the quality of your data.

How many classes do you have (binary OR 100 classes)?
How different are the classes from each other (snake vs plane OR python vs boa)?
What types of features do you have (imaging, time series, tabular data)?
How much label noise is in the data?
How much feature noise is in the data?
How predictive are the features of the classes (relates to #1)?
What is the capacity (model complexity) of your model to learn the pattern?

In a more practical sense, I search the literature for publications that worked on similar tasks as my task and see what models they used, how many examples they used, and what type of performance they achieved.
Consider expanding your training data with Data Augmentation or GANs or public data sets or working with collaborators.
Consider transfer learning if you have scarce data.
Besides neural networks, there are again no a priori methods for determining which traditional ML model is going to perform better.  You have to empirically test them out.
Consider N-shot learning.
Depending on whether you have very little data over all or whether you have very little labeled data, considered semi-supervised learning.

I agree with most! Only a minor point: data augmentation can be harmful. Only in the setting of multi objective optimization (like optimizing e.g. recall of multiple classes etc.) data augmentation is a very indirect way of changing the tradeoffs


text:
Where exactly is permutation happening in equation 5 of the paper "Learning with Sets in Multiple Instance Regression Applied to Remote Sensing"?
I am reading the article Learning with Sets in Multiple Instance Regression Applied to Remote Sensing about creating an embedding which is order-invariant to inputs ($m_{l}$). They referred to order-invariance in eq (5) as follows:

The order-invariant operation happens in (5), where permuting $m_l$
with $m_{l}'$ yields the same vector $r_t$, due to the summation.

First, what exactly is $m_{l}'$? Is it the transpose of $m_l$?
Second, how is permutation happening?
$$r_t=\sum\nolimits_{l=1}^{L_i}a_{l,t}m_l$$
Here, equation (5) in the paper, $m_l$ points to the $l^{th}$ vector, $a_{l,t}$ is merely some scalar weight, and $r_t$ is the vector returned.


text:
How can I encode angle data to train neural networks?
I am training a neural network where the target data is a vector of angles in radians (between $0$ and $2\pi$).
I am looking for study material on how to encode this data.
Can you supply me with a book or research paper that covers this topic comprehensively?

Many suggestions here: https://stats.stackexchange.com/questions/565038/why-doesnt-mean-square-error-work-in-case-of-angular-data/565057#565057

The main problem with simply using the values $\alpha \in [0, 2\pi]$ is that semantically $0 = 2\pi$, but numerically $0$ and $2\pi$ are maximally far apart. A common way to encode this is by a vector of $\sin$ and $\cos$. It perfectly conveys the fact that $0 = 2\pi$, because:
$$
\begin{bmatrix}
    \sin(0)\\
    \cos(0)
\end{bmatrix}
=
\begin{bmatrix}
    \sin(2\pi)\\
    \cos(2\pi)
\end{bmatrix}
$$
This encoding essentially maps the angle values onto the 2D unit circle. In order to decode this, you can calculate
$$\text{atan}2(a_1, a_2) = \alpha,$$
where $a_1 = \sin(\alpha)$ and $a_2 = \cos(\alpha)$.
Here is a nice detailed explanation and here are two references, where this is applied:

Heffernan et al.
Pizzati et al.

EDIT As it was noted in the comments: The values $\sin(\alpha)$ and $\cos(\alpha)$ are not independent and the following naturally holds: $\sqrt{\sin(\alpha)^2 + \cos(\alpha)^2}= 1$, i.e. the euclidean norm is one. In a situation where your Neural Network predicts the sin and cos values, this condition isn't necessarily true. Therefore, you should consider adding a regularization term to the loss that guides the neural network toward outputting valid values (with unit norm) which could look like this:
$$
r_\lambda\left(\hat{y}_1, \hat{y}_2\right)\; = \lambda \left(\; 1 - \sqrt{\hat{y}_1^2 + \hat{y}_2^2}\right),
$$
where $\hat{y}_1$ and $\hat{y}_2$ are the sin and cos outputs of the network respectively and $\lambda$ is a scalar that weights the regularization term against the loss. I found this paper where such a regularization term is used (s. Sec. 3.2) to get valid quaternions (Quaternions must also have unit norm). They found that many values work for $\lambda$ and they settle for $\lambda = 0.1$

Do you know if there is a general approach for that? The closest thing that comes to my mind are group convolutional networks. Personally though, I would prefer a simpler, less number crunch heavy method

I'm not sure what exactly you are looking for when you ask for a general approach - on the one hand, there is general angular data (which you can encode with the method described). And then there is the context of those values (i.e. the domain). If the domain involves rotational symmetry then you should look into group equivariant convolutional networks (which is probably what you mean by group convolutional network, right?). Without knowing the domain, it is not clear if group equivariant convolutions are a reasonable thing to use. What is the dataset you are using?

You only need to use $sin(\alpha)$ (or $ccos(\alpha)$) as the label, right?  No need (and undesirable) to have both as multilabel outputs given their strict dependence.

Actually, you should use both otherwise you would get singularities, e.g. for $\sin$ this would happen at $\sin(0) = \sin(\pi) = \sin(2\pi)$. Whereas, if you encode it as the tuple, you get a unique vector for all $\alpha \in [0, 2\pi)$ with the property that at $0$ and $2\pi$ the encoding is the same. You are right that the values are dependent. I've often seen that there is a regularization term in NN training like this: $\parallel [x_1, x_2]^T \parallel_2 = 1$, which guides the network toward outputting valid numbers for the two outputs $x_1, x_2$ where the norm is one.

Thanks for pointing that out @SnehalPatel, I added the regularization thing to the answer

@Chillston I didn't have a specific data set in set. I was wondering how one would handle the case, when the given data has some symmetries, structure or invariances towards specific transformation. In case of angular data as above, a neural network can learn to treat the regions are around an angle 0 and angle 2pi equally, however it is not guaranteed that the network will pick up on that structure. So, how can one simplify this process in general, as done with the angle problem above?

@Imago The general form of that would be group-specific neural networks. So when you have a symmetry w.r.t a specific group, then you can design neural operations that obey these symmetries. For 2D rotation invariant data, you'd want a kernel that itself is invariant to the orthogonal group O(2). However, this is a different concept to what the question is about. Proper angle encoding doesn't necessarily mean that you have rotational symmetry. Maybe you find the protobook about Geometric Deep Learning a good read (or you already know it).

You might want to look at the von Mises Distribution, it defines a probability distribution over angles.
See Pattern Recognition and Machine Learning, Christopher Bishop, Appendix B (pg 693), or alternatively Wikipedia has an article on this.
You could certainly use this as a loss function in a neural network. My only reservation is that the input parameter $\theta_o$ is itself periodic which might not play well with standard neural network architectures? Therefore previous answers are also worth considering.
I only mention this as if you are interested in distributions over angles, the von Mises distribution is something you should probably be aware of.

I'm not sure I understand what you mean by "You could certainly use this as a loss function in a neural network?"  Can you provide an explanation of how it the probability density function can be used as a loss function?

Von Mises distribution defines a probability density. You take the logarithm of this function and then take the negative of that; that is the loss function.   This provides what I think is a pretty good explanation of the basic approach:
link


text:
Instead of accumulating the gradient, can we accumulate loss values?
I have read and used Gradient Accumulation as a method to handle large batch size on smaller memory restrictions. It is described as following:
for step, eachBatch in enumerate(dataloader):
   ...
   loss = loss_func(ytrue, ypred)
   loss.backward()
   if step % 5 ==0:
      # update weights every 5 steps
      loss.step()
      loss.zero_grad()

However, instead of accumulating the gradient, can we accumulate loss values? Just like multi-task trainings, but I have never seen people recommending the following method, but wonder why it that the case, what are the potential problems with this. If there really exist a problem with the following method, then why does loss accumulation work for multi-task training?
totalLoss = 0
for step, eachBatch in enumerate(dataloader):
   ...
   loss = loss_func(ytrue, ypred)
   totalLoss += loss # accumulate the loss
   if step % 5 ==0:
      totalLoss.backward()
      totalLoss.step()
      totalLoss.zero_grad()
      totalLoss=0

Accumulating the loss like that doesn't improve the memory requirements, because the memory consumption depends on the size of your computational graph. In other words, each time you add a term to the loss, the overall function of your loss grows as well, and with it grows the memory consumption.
Specifically, every time you add a term to the loss value, you run the forward step with a new input/output pair $x_i; y_i$. In order to backpropagate, all the hidden activations $h_{i,l}$ must be stored until you call loss.backward(). So with each new loss added, that cache of latent activation vectors grows. Check out PyTorchs walkthrough of what exactly happens there.
In contrast, when you have already computed the gradient, then the cached activation vectors can be deleted and the only things that are saved are the gradient values for your model parameters. In comparison, accumulating the loss would lead to a memory consumption of $\mathcal{O}(n * I)$ and accumulating gradients is $\mathcal{O}(n + |\theta|) \sim \mathcal{O}(n)$, where $n$ is your single-pass memory requirement, $I$ is the number of accumulation steps, and $|\theta|$ is the number of model parameters. I hope that was somehow clear.

So in short, loss accumulation does not achieve the effect of reduce memory requirement. And in the case of multi-task training, what is your opinion if I change loss accumulation to gradient accumulation?

It is mathematically equivalent whether you backpropagate a loss $L(x, y) = l_1(x, y) + l_2(x, y)$, or you accumulate the gradients from separate backpropagation of $l_1$ and $l_2$. So if you can save memory using gradient accumulation I'd say go for it :)


text:
What models/algorithms besides variational autoencoders can I use to transform a discrete input into a differentiable latent space?
Let's say I have a discrete input and want to transform it into a differentiable latent space. What models/algorithms besides variational autoencoders can I use?

I think any kind of matrix factorization technique would do the trick.
You can find a lot of resources related to natural language processing about it. Of course everything is applicable outside the NLP realm.

Thank you for your reply, I will look into the resources you provided. My idea is indeed that I want to apply this to NLP. So let's say I want to project whole sentences into the latent space through matrix factorization, do small, linear movements in that latent space always lead to a meaningful result?

I am asking because I want the latent space to be differentiable.


text:
What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?
What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?
I know one possibility is to subtract a baseline as a running average of rewards from past mini-batches. Another is to compute the mean and variance of each trajectory over one mini-batch and standardise the values. A third one is to use large batch sizes.
What is considered the most effective? What other methods are there?

Eligibility traces and actor-critic method would have much less variance compared to REINFORCE while eligibility traces often works best in practice.


text:
Where does the proximal policy optimization objective's ratio term come from?
I will use the notation used in the proximal policy optimization paper.

What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?
Put another way, we start with the vanilla policy gradient objective and aim to optimize $L^\text{PG} (\theta) = \mathbb{E}_t \left [  \log \pi_\theta(a_t|s_t)A_t \right]$. Where in $L^\text{PG}(\theta)$ do I make an approximation to derive equation (6)?
I could not find this objective following the reference KL02.

I am not sure either. It doesn’t make sense. Im confused because its not symmetric(unclipped). If the advantage is positive and the ratio is greater then 1, you get a big positive gradient. But if you have a negative advantage and a ratio less then 1(same scenario but flipped in my mind), then you get a small negative gradient. In my mind, given the same advantage magnitude, opposite signs, the gradient of a ratio of 2 would be the negative of a gradient of a ratio of 0.5. Does that make sense?

No approximation here. The formula for $L^{CPI}$ can be directly derived from the formula for $L^{PG}$.
\begin{align}
\nabla L^{PG} &= \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a_t | s_t) A_t \Big] \\
&= \int_{a, s} \pi_\theta(a | s) \nabla \log \pi_\theta(a|s) A_t \\
&=  \int_{a, s}  \frac{\pi_{\theta_{old}}(a | s)}{\pi_{\theta_{old}}(a | s)} \pi_\theta(a | s) \frac{\nabla \pi_\theta(a | s)}{\pi_\theta(a | s)} A_t \\
&= \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\nabla \pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \\
&= \nabla L^{CPI}
\end{align}
Now:
\begin{align}
L^{PG} &= \int \nabla L^{PG} = \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \log \pi_\theta(a_t | s_t) A_t \Big] \\
L^{CPI} &= \int \nabla L^{CPI} = \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big]
\end{align}
The difference between the two is that in $L^{PG}$ states and actions are sampled using the original policy $\pi_\theta$. This means that once you perform one gradient update step you have to throw away the data and collect new.
In $L^{CPI}$ the states and actions are collected under $\pi_{\theta_{old}}$, which means that you can perform multiple updates with the same dataset. Obviously, if you perform only one update step with PPO you will get the vanilla PG. After the first update step the two objectives will differ.
There is a bit more theory regarding why you want to clip, but since this is not part of the question I will just refer you to this blog post that I wrote:
https://pi-tau.github.io/posts/actor-critic/#ppo

Thank you for the answer! I was just wondering - I was checking the CPI paper but I dont see where they propose it as an objective? Any idea?

The ratio term appeared between Eq. (13)&(14) in Sch+15b, as a result of importance sampling:
$$
L^{\theta^{\prime}}(\theta)=E_{\left(s_t, a_t\right) \sim \pi_{\theta^{\prime}}}\left[\frac{\pi_\theta\left(a_t | s_t\right)}{\pi_{\theta^{\prime}}\left(a_t | s_t\right)} A^{\theta^{\prime}}\left(s_t, a_t\right)\right].
$$


text:
Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?
In some RL notes, I encountered the following equation, which I am trying to prove:
$$
V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S
$$
Here is my attemption:
Firstly, I only need to prove "$\geq$" as "$\leq$" is obvious.
Suppose that $\exists S_0 \subset S$ such that $\forall s_0 \in S_0,V^{\pi^*}(s_0) < \max_{a \in A}Q^{\pi^*}(s_0, a)$.(W.L.G, soppose there is only single state $s_0 \in S_0$), i want to construct a new policy $\pi_{\text{new}}$ to derive the contradiction to $V^{\pi^*}(s) \geq V^{\pi}(s), \forall s \in S$ and all policy $\pi$.
I'm trying to create the following policy:
$$\pi_{\text{new}}(a\vert s)=\begin{cases}
1& s=s_0,a=\text{argmax}_{a\in A}Q^{\pi^*}(s_0,a)\\
0& s=s_0, a\neq \text{argmax}_{a\in A}Q^{\pi^*}(s_0,a) \\
\pi^*(a\vert s)& s \in S - s_0
\end{cases}
$$
Next, I want to show
$$
V^{\pi_{\text{new}}}(s_0) = \max_{a\in A}Q^{\pi^*}(s_0, a)
$$
And thus $V^{\pi_{\text{new}}}(s_0) > V^{\pi^*}(s_0)$ to get contradiction. But when I derived it, I encountered some trouble as follow:
$$
V^{\pi_{\text{new}}}(s_0) = \sum_{a \in A}\pi_{\text{new}}(a \vert s_0)Q^{\pi_{\text{new}}}(s_0, a) = Q^{\pi_{\text{new}}}(s_0, \text{argmax}_{a\in A}Q^{\pi^*}(s_0, a)) \overset{?}{=} \max_{a \in A}Q^{\pi^*}(s_0, a)
$$
I guess the last equation holds, but I can't find a way to prove it. Because the policy changed. Could you please help me?

So, we've got these Bellman equations for optimal $V^*$ and $Q^*$:
$$ V^*(s) = \max_a\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)V^*(s')\right]\qquad (1)$$
$$Q^*(s,a) = R(s,a) + \gamma \sum_{s'}T(s',s,a)\max_{a'}Q^*(s',a')\qquad (2)$$
Taking $\max_a$ over $(2)$:
$$\max_a Q^*(s,a) = \max_a\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)\max_{a'}Q^*(s',a')\right]$$
We see that this is equivalent to $(1)$ if we denote $V^*(s) = \max_aQ^*(s,a)$.


text:
What makes TRPO an actor-critic method? Where is the critic?
From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and old policy.
NPG is a modification to "vanilla" policy gradient algorithms that modifies the direction of the update using the Fisher information matrix such that the direction is in that of maximal improvement per unit of KL divergence between new and old policies.
Now, I've skipped a lot, but throughout most introductory material on these two algorithms, I see no mention of a critic (i.e. auxiliary evaluator value function), yet  know that at least TRPO is considered actor-critic. Why is that? Where does the critic come in?

TRPO still uses the advantage function, which computed using the critic (aka value function).

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Can AI simulate fake inputs, train itself on them, and elaborate them like they were real ones generated from an outside source?
Can AI simulate fake inputs, train itself on them, and elaborate them like they were real ones generated from an outside source?
This idea is similar to how human brain works, indeed it can "think" whatever feeling, such a voice, a sound, an image, a smell, a flavour without really get them from outside. It can work on it like the real ones and give an output. It looks to me this is how the consciousness works.

How does "human brain(...) think whatever feeling, such a voice, a sound, an image, a smell, a flavour without really get them from outside"? I am human but never know such ability.

Can AI simulate fake inputs, train itself on them, and elaborate them like they were real ones generated from an outside source?

Yes, this is exactly what generative adversarial networks do.  Take a look at "A Beginner's Guide to Generative Adversarial Networks (GANs)"  The GAN has a generator and discriminator.  The generator "simulates fake inputs".  The discriminator (police) tries to determine if it is fake or real.  When the discriminator catches the fake ones, the generator learns from its mistakes.  The generator keeps creating fake images until it starts fooling the discriminator into thinking they are real.  In other words, the AI (i.e., the generator/discriminator combo) "trains itself" to create fakes that are so good that they start looking real


text:
Why are policy gradients popular in RL when there exists a dual LP formulation in terms of occupation measures that can be solved easily?
Why are policy gradient methods popular in reinforcement learning when there exists a dual LP formulation in terms of occupation measures that can be solved easily?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Can you please clarify what you mean by "dual LP formulation in terms of occupation measures". I know what linear programming is, but what do you mean by "occupation measures" and "dual LP formulation in terms of occupation measures"?

Occupation measure refers to an MDP's discounted state-action visitation frequency under a given policy. A detailed explanation of how this can be used for obtaining policies and the corresponding dual formulation, you can refer to Section 6.9 Linear Programming in the book Markov Decision Processes(1994) by Martin L. Puterman.

Policy gradient methods are popular in reinforcement learning because they are fast and easy to implement. Additionally, policy gradient methods often work well for simple problems. for example, if the Hellinger distance between two measures is small.
Dual LP methods may be more accurate if the problem is not too simple. However, they can be more difficult to implement and may require more computational resources. they are preferable if you have more information about the underlying distribution of the data.


text:
Can I use a dataset with real-world images and corresponding actions that the expert took to train an IRL algorithm?
Offline Reinforcement Learning approaches like Inverse Reinforcement Learning/ Batch RL/ imitation learning/ behavior cloning allow us to use previous demonstrations by an expert to learn a policy.
Many of the papers that I have found use expert data generated from a simulated environment for model training. However, I have some real world images and corresponding action that the expert took in these states e.g. images from a camera mounted on the car and the steering angle as an action. The action space is continuous meaning how fast the steering angle was applied.
Can I use this dataset as an expert demonstration in place of simulated dataset for training an Inverse Reinforcement learning algorithm?


text:
Is there a reward function that would encourage exploration in this case?
I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is
reward = (variable1 + variable2)*-1
The agent is providing actions with values that are close to -1 for higher iterations. Instead of attempting to improve the environment, it only tries to avoid high negative rewards. I've tried normalizing observations and rewards, but it's still been ineffective.
Is there a reward function that would encourage exploration rather than the same behavior?

There is no one-size-fits-all answer to this question, as the best reward function for encouraging exploration will vary depending on the specifics of the AI system. However, some possible reward functions that could encourage exploration include giving the AI a small reward for each new state it visits, or giving the AI a large reward for discovering new states that are particularly valuable.


text:
Are there Reinforcement Learning algorithms specialized for the case $\gamma=0$?
I have a Reinforcement Learning problem where the optimal policy does not depend on the next state (ie gamma equals 0). I think this means that I only need an efficient exploration algorithm coupled with a way to backpropagate the rewards.
I know that contextual bandits are specialized for this situation, except they only work for discrete action space, and I still need my policy network to make complex decisions (train a Deep neural networks, whereas most contextual bandits algorithms I found learn linear policy decisions). I also think deep active learning does not work because I can not trace how the reward is computed (it is not model-based).
Therefore I am looking for algorithms, or ideally a python implementation, that solves RL for continuous action spaces when $\gamma=0$.
Reposted from here.


text:
What exactly is non-delusional Q-learning?
Problems occur when we combine Q-learning with a function approximator.
What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-learning and value-iteration.
I have trouble understand the term "Policy Commitments", and "consistency". What are they talking about?
PS: a related post

Unfortunately I don't think I'll have time to write up a comprehensive answer very soon. A video of the authors' own presentation of their paper at the conference is freely available here though: https://www.youtube.com/watch?v=fwSEyyEJmkc
Hopefully that can already help you, or anyone else who may want to write up a detailed answer

Non-delusional Q learning is a type of machine learning that can be used to learn how to optimally make decisions in environments where there is some uncertainty. It is related to reinforcement learning, but differs in that it does not require a model of the environment and can be used in environments where the reward function is unknown.


text:
What is the best distance measure between policies that are not probability distributions?
This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a state as input and return an action vector, what would be the best method to measure how close two policies are from each other?
A naive approach that came to my mind would be to:

Run both policies A and B to produce a trajectory each and record all states visited.
For each state encountered by policy A, ask policy B which action it would take (and vice-versa). Hence we would have, for every state encountered, both A and B action vectors.
For each state, compare action vectors of A and B by using a common distance (Euclidean distance?)
Take the average (maybe maximum) of those distances.

Does it make any sense from a theoretical point of view?

Can you clarify what the action vector produced by the policy actually represents? If it's not a probability distribution, then why do you have an action vector? Is it a policy that produces multiple actions, all of them need to be taken at time step $t$? Is this a multi-agent system? Please, edit your post to clarify this.

One way to measure the distance between two policies is to use the Kullback-Leibler (KL) divergence. This measures the amount of information lost when one policy is used to approximate the other.
Another way to measure is to use the Wasserstein metric. This measures the amount of "work" or "energy" required to transform one policy into the other.
A third way is to use the Mutual Information (MI) metric. This measures the amount of information shared between the two policies.

yes! Euclidean distance would way to measure the distance be between the two policies action vectors

It does, sort of. The idea is that you have a reward function that encodes the task you want to solve. The agent is trying to find an optimal policy that maximizes this reward function. In order to do this, the agent must learn the reward function.
One way to think about this is that the agent is trying to find a mapping from states to actions that will maximise the reward. This is the same as finding a mapping from states to a goal state that maximises the reward. So, if you have two different agents, they will both be trying to find a mapping from states to actions that maximises the reward. However, they will likely find different mappings, because they have different starting

OP’s question was regarding deterministic policies. I’m not sure how useful KLD is in this instance as they are trivially dissimilar (I think, happy to be told otherwise).


text:
What features for a content-based recommendation system might give the best results?
I'm working on a hybrid recommendation system (collaborative and content-based) for an online ordering/shopping app. So far I've managed to identify a data-source for the collaborative model (likely item-based) but I'm having trouble deciding on features and algorithms for the content-based model.
I know it's probably specific to the problem I'm working on, but if anyone could share their experience or recommendations on what features give the best results for a content-based system it would be really helpful.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Some features that have been found to work well for content-based recommender systems include:

Item category (e.g. food, clothing, electronics, etc.)

Item sub-category (e.g. type of food, type of clothing, etc.)

Item brand

Item price

Item description

Item reviewer ratings

Number of reviews

Overall sentiment of reviews

Keywords associated with the item

Any other relevant metadata


text:
How should I choose the depth for minimax if I have a strict time constraint?
I am working on a controller that plays Ms. Pac-Man using a minimax algorithm. The controller has a limited time amount in which it can choose a move on each round, otherwise when the time runs out the system chooses a random action instead.
My goal is to select a value for the depth that is large enough to consent Ms.Pac-Man to achieve a decent score, but not so large to cause a lot of timeouts. Leaving the number of timeouts aside, basically I did the most intuitive thing: I increased the depth until I noticed a drop on the score values, but that drop could have occurred casually.
Is there a more scientific/statistical way to do that (that ideally addresses both the score e the timeout issues)?
(I fantasized about performing repeated hypothesis testing, but I quickly learned how silly that idea was when I heard about multiple testing problem).

Can you do a breadth first search using all available time (leaving just enough to collapse the tree and find the optional action)?

I don't think that I can do that. Every round lasts only a few milliseconds, and the agent chooses its moves on each round.


text:
Is there a name for this model?
I have an image autoencoder model trained as follows:
Step 1) train a GAN to obtain a generator capable of drawing from the data manifold by sampling a normal distribution in latent space
Step 2) train an encoder on the front, keeping the generator frozen, effectively mapping images into the normally distributed latent space.
Is there a name/reference for such a model?
I know it's not really a VAE-GAN because it has no variational component, and the decoder and encoder are trained separately. It's not quite a vanilla AE either since the latent space is structured and constrained in the GAN training step. The models I get from this approach, while not SoTA, do have desirable qualities like disentangled representations but without the instabilities of VAE-GAN training.
I do use a perceptual loss when training the encoder, using intermediate feature maps from the GAN-trained discriminator, but I don't think it makes a big difference to the question I'm asking.


text:
Why do transformer Key Query Value layers not have biases or activations?
Transformers use just matrices to transform input embeddings, which is halfway to being a connected dense layer (add a bias and activation). So, why don't transformers have dense layers for encoding input into Query Key Value?

The following is about the same question and provides a suitable answer: https://ai.stackexchange.com/questions/40252/why-are-biases-typically-not-used-in-attention-mechanism/40256#40256

I suppose the main reason is that they were proposed as a more parallelizable alternative to RNNs that is quicker to compute. Hence, making them just big matrix multiplications achieves this and the lack of intermediate activations is just replaced by the sheer number of transformers that focus their attention on different parts of the input sequence.


text:
Is item-based collaborative filtering the same thing as content-based filtering?
According to this Google dev page

content-based filtering
Uses similarity between items to recommend items similar to what the
user likes.
collaborative filtering
Uses similarities between queries and items simultaneously to provide >recommendations.

With the items and query defined here

Items (also known as documents)
The entities a system recommends. For the Google Play store, the items
are apps to install. For YouTube, the items are videos.
Query (also known as context)
The information a system uses to make recommendations. Queries can be
a combination of the following:
user information
the id of the user
items that users previously interacted with
additional context
time of day
the user's device"

But I also see the term "Item-based Collaborative Filtering" thrown around sometimes, e.g. here, here and here.
Is this item based collaborative filtering the same thing as content based filtering?


text:
Today's Practicality of Bayesian Neural Networks
Just having heard lately about BNNs (wow, ANNs and CNNs are clear; now there's a B? What's that? Ahh, Bayesian ;-)) and quickly getting their main idea and focus, that is, weights not being pure static numbers anymore, but instead described by a distribution function, allowing the net to be "uncertain" about its answers (for more about BNNs, see, for example, this post, or, if you want to know a lot about them, check out this introduction and survey),
I'm now curious about any practical examples of where BNNs are used nowadays, maybe even more successful than other (deep learning) alternatives?
Thanks to nbro's comment here, pointing out that asking this question is not unknowingly, but relevant.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

https://arxiv.org/pdf/2002.00269.pdf

@Community I am questioning today's practicality of BNNs and asking (see tags) for state-of-the-art and/or references. I think it's clear what I am asking and how an adequate answer would look like. Please correct me if I am mistaken here.

@dangom I skimmed that article but I honestly could not find any real value about my question in it. It's morge like the introduction and survey I linked above; but not about its daily use.


text:
What do we mean by the notation $\mathbf{x}_{p} \in \mathbb{R}^{N \times\left(P^{2} \cdot C\right)}$?
I was going through this VIT paper, what will it look like in torch , if we are trying to write this expression.

This itself isn't really an expression but a description of what $x_p$ looks like. Specifically, $x_p$ is a real-valued vector with the shape [N, P^2 * C].
Of course, that data comes from your dataset, but as an example torch.ones(N, P^2 * C, dtype=torch.float32) will give you a vector with the same shape. Note that this will be a float32 vector, which makes it real-valued ($x_p \in \mathbb{R}$).

@Chillston You could write that as an answer instead of a comment.

Of course, sorry

This itself isn't really an expression but a description of what $x_p$ looks like. Specifically, $x_p$ is a real-valued vector with the shape [N, P^2 * C]. Of course, that data comes from your dataset, but as an example
torch.ones(N, P^2 * C, dtype=torch.float32)

# an example output for values N=2, P=2, C=1
>>> [[1., 1., 1., 1.],
     [1., 1., 1., 1.]]

will give you a vector with the desired shape. Note that this will be a float32 vector, which makes it real-valued ($x_p \in R$).


text:
How to form 10 and 20 actions in corridor environment, in the paper "Dueling Network Architectures for Deep Reinforcement Learning"?
I am just reading a paper titled "Dueling Network Architectures for Deep Reinforcement Learning". In this paper, 4. Experiment (4.1 Policy evaluation), I just wonder how to form 5, 10, and 20 actions.
According to this paper, the 10 and 20 action variants are formed by adding no-ops to the original. I can't understand this... Does it mean that actions are available: go up, go down, left, right, no-op, no-op, no-op... and no-op?
Am I correct?

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

What do you mean by "to form 5, 10, and 20 actions"?

I just borrow that expression from the paper.
I mean how to modify 5 action(up,down,left,right) in corridor env to 20 actions.

I think you are right. But then the next doubt would be why to include so many no-ops instead of just original action space+ one no-op space rather. And for this, I think it is to intentionally increase the chances for the agent to choose no-op action. And the reason to increase the chance might be to have some randomness or variety for the agent so that it is kind of better challenged to learn the correct actions.
The accepted answer in the following link tells you the significance of having no-op actions that might help you to correlate and possibly deduce why that specific number of no-op actions in your paper exaample. https://stats.stackexchange.com/questions/351035/why-does-deep-q-learning-have-do-nothing-actions

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
How does YOLO detect the object when the object is in multiple grid cells?
I have been reading various articles and watching videos on YouTube, but I can't seem to understand one thing.
How does YOLO make a bounding box for an object if it is in multiple grid cells? For example, in the picture given below, how does it predicts the bounding box for the classes, because they fall in multiple cells? How does it know what object is in a grid cell even when it sees a small part of it?
It's been very difficult for me to get these answers.

Lets say you split the image by 13x13 (SxS) cells. For each grid cell, you predict 5 (older YOLO) or 9 (newer versions) bounding boxes.
These (anchors) are supposed to have various shapes and scales, so they vertical, horizontal and squared rectangle at small scale, and middle scale and bigger scale.
So for each cell (13x13) and for each anchor (9) , you predict (width offset ,height offset,center x,center y, objectness, class1 ,class2,... class k ). You actually predict scaling of the width and height of the box, so you can end up with variable sizes.
Ojectness marks if there is any object in this cell for this anchor, if not, it is discarded later.
Since you end up with lot of predicted bounding boxes of various shapes, you need to filter them out, this is called non maximum supression.
For each anchor you take class with highest confidence.
EDIT:
So to answer your questions, If an object is in multiple cells, it is contained in multiple predicted boxes, and you take the one with maximum confidence and discard others.
Each cell does not see only part of the image, it sees the whole image. Before you get to 13x13 cells, you have 50 or more layers of conv & max pooling layers, which gives each cell receptive field of the whole image.

Firstly, thank you very much for taking your valuable time to answer my question. Secondly, can you please brief me a little more about anchors boxes? Are bounding boxes and anchors boxes mean the same thing? Are their positions are fixed for each grid? What if the objects that we are looking for doesn't falls in any of the 9 anchors boxes? What do we do then? Do we get any prediction if objects doesn't falls in those 9 anchors boxes or we don't?

TL;TR
The location of the input that a neuron sees is called receptive field.
grid cell is merely used to demonstrate the concept of anchor. It has nothing to do with the receptive field of the network structure.
The actual receptive field size (RF) depends on the paramters of yolo structure, different size of the yolo network actually has different RF.
The RF is definitely not the full size of the input image, but it is also way larger than the grid cell. Usually several times bigger.
The smaller the feature map, the bigger the ratio of RF/cell
For yolov8 tiny, the RF of P3 is 12 times of P3 grid cell size. P4 is 7x, P5 is 4x.
yolo use each P3-P5 to predict exactly one bonding box per neuron.
For P3 it is 80x80 neurons, for P4 it is 40x40 neurons, for P5 it is 20x20 neurons.
The size of the object can be predict for each neuron is not constrained at all. It only limit the location of the object. see detail section for more discussion.
During training, one grid cell only provide a single box, but same ground truth box can be assigned to multiple grid cells.
Thus during inference it is possible to have multiple grid cells per feature map predict a single instance.
To handle this multiple-to-one problem, we need a postprocess step to select one box for each instance. During inference, the model predict additional score: class-quality score which roughly means the class probability multiple by box quality. This is then used as likely-hood score or confidence for corresponding box:

we first use a threshold of the confidence to filter out the low quality bbox.
for each bbox it choose the argmax of class confidence as the instance class.
If we are lucky there is no further postprocess is required. However, it is very common, that we still have some redundant boxes per instance. There are boxes per each class have large overlaps. To reduce the redundancy we use a heuristic algorithm called NMS. It is heuristic, because in high dense scenario it is possible to have multiple instance closely overlapped, but with NMS we can only detect one of these instances.
the NMS basically applied to each class separately. It basically start from high confidence box then remove the boxes has lower confidence but high overlap ratio

some details about the relationship of grid cell, predict box and receptive field.
A neuron can only predict a box contains the grid cell centre, but the box size can be much larger or much smaller.
The extreme case is demonstrated in following image.
The predicted box can be as big as the image itself.
There is no limit of the predicted box size relative to either grid cell or receptive field.

So why we allow it to predict a box larger than the input (receptive field)?
Because the network can "guess" the actual box via seeing partial of the object. For example, even though we can only see part of the bicycle, we can still estimate how big the actual bicycle is:

more details about the receptive field
Let's take the modern yolov8 as example. To simplify stuff we only look at the backbone.
It is a CNN structure.
In CNN the receptive field only changed when the conv kernel size > 1 or stride size > 1:

The conv has k=1, s=1, is only a channel mixer which does not blend any lateral spatial info.
when k>1 each neuron see kxk input
when s>1, the current layer skip some input while scanning. Thus equivalently it makes the next layer's neuron seeing wider input.

Thus the RF can be calculate as:
$$
RF_{l} = RF_{l-1} + (K_{l} - 1) \times \prod_{j=1}^{l-1} S_{j}
$$
Where:

$RF_{l}$ is the receptive field of layer $l$.
$K_{l}$ is the kernel size of layer $l$.
$S_{j}$ is the stride of layer $j$.

Here is the actual structure overview. Focus on the left side backend part.
the C2f is tricky, it contains n bottlnecks each of the bottleneck has 4 conv modules. For tiny model the d = 0.33
We can roughly see it as

stem conv
C1 conv
C2 C2f n=1
conv
C3 C2f n=2
conv
C4 C2f n=2
conv
C2f C2f n=1
C5 (SPPF) (9)

So the kernel size and stride for each layer can be represent as a list of list: blocks
# each bottleneck blcok has 4 conv that k>1
btn = [(3,1)]*4
# stem, C1 (1), C2 (2), conv, C3 (4), conv, C4 (6), conv, C2f, C5 (SPPF) (9)
idx_Cs = [1,2,4,6,9]
blocks = [
  [(3,2)], # conv
  [(3,2)], # conv
  btn*1, # C2f n=3*d
  [(3,2)], # conv
  btn*2, # C2f n=6*d
  [(3,2)],
  btn*2, # C2f n=6*d
  [(3,2)],
  btn*1, # C2f n=3*d
  [(3,1), (5,1), (5,1), (5,1), (3, 1)] # SPPF
]

For the backbone we already have recptive field for each feature map like following:
C1 has RF= 7x7
C2 has RF= 15x15
C3 has RF= 35x35
C4 has RF= 55x55
C5 has RF= 83x83

However, we still have FPN which continues to fuse the feature map to form P3, P4, P5.
This further increases the receptive field, because for each Pn, the longest path is longer than C5.
I will not put details here but I can show the final results:
P3 has RF= 99x99
P4 has RF= 111x111
P5 has RF= 123x123

detection heads then add another 2 convs that can change RF. Resulting the final RF:
DET_P3 has RF= 103x103
DET_P4 has RF= 115x115
DET_P5 has RF= 127x127

Remember, for P3, P4, and P5, the reducing factor is 8, 16, 32.
This means the grid cell size in pixel is: 8x8, 16x16 and 32x32.
We can see they are far smaller than the actual receptive fields.
Now you also understand the limit of the yolo structure, it usually struggling to predict big object which cover most of the image, since the RF of P5 does not cover the entire input.
This is also why there is enhanced version that has P6 layer to add a bigger feature map.
You can find the code used to compute the RF here.


text:
Is the described Q-table considered large?
I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can answer maybe. I assume I can visit all state-action pairs enough times but is it efficient in terms of space complexity/memory? Or is a deep neural network required?
I asked a related connected question, in case anyone wants to answer this current question better w.r.t that question context.

There a couple of "rules of thumb" you might apply to decide whether a Q table is large enough that some kind of approximation would help:

Does it fit into memory?

Does the rate of gathering experience mean that the agent can reasonably sample enough from each relevant state/action pair to assess policies?

For a state/action space of 2500, it should trivially fit into memory on any modern device, even if you need to use some kind of string description of the states and actions, using them as keys to a dictionary lookup of value. Even if you are short of space in some embedded device, the space required by code for approximation is probably larger than the table.
In my experience I would rate the state/action space of 2500 as very small. If you can simulate the environment on a computer, then you could reasonably expect to find the optimal policy in under a second. For fast simulations, or with full models using value iteration, Q tables with millions of values to calculate may be feasible.
However, you have not stated what it takes to sample a time step (measuring start state, action, reward, next state) in your environment, or how long you have to run the training process. This could still make a difference in how you want to represent the action value function.


text:
Why and when do we use ReLU over tanh activation function?
I was reading LeCun Efficient Backprop and the author repeated stressed the importance of average the input patterns at 0 and thus justified the usage of tanh sigmoid. But if tanh is good then how come ReLU is very popular in most NNs (which is even more odd when the authors didn't mention about ReLU at all)

Are you reading the original 1998 paper?  That would be my guess as to why ReLU isn't mentioned.

Does this answer your question? Choices of activation functions

For a discussion about the advantages of ReLU, see the original paper by Glorot (2011) "Deep sparse rectifier neural networks".
"Efficient Backprop" is a 1998 paper. At the time the use of rectifiers was uncommon and sigmoid was the standard choice of activation.


text:
What is single object localization?
Object detection is said to be combination of object localization and image classification. However, when reviewing localization, I often come across the term "single-object" localization, but I am having trouble finding an accurate definition. What is "single-object" referring to?

Can you please provide more context? Where did you find the term '"single-object" localization'?

its in the picture that I have attached?

Yeah, but maybe provide the paper, article, or whatever, where you found that picture?


text:
Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?
In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state:
$$
\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}
$$
In Q-learning, we try to find a policy that maximize the state-action value function Q:
$$
\pi_q^*=\underset{\pi}{\operatorname{argmax}} {Q_{\pi}(s,a)}
$$
However, does maximizing the value function and maximizing the state-action value function generate the same optimal policy? In the general case of continuous, stochastic action, $V_{\pi}(s)$ is connected to $Q_{\pi}(s,a)$ by:
$$
V_{\pi}(s)=E_{a\sim\pi}[Q_{\pi}(s,a)]
$$
So
$$
\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}=\underset{\pi}{\operatorname{argmax}} E_{a\sim\pi}[Q_{\pi}(s,a)]
$$
And if $\pi_v^*=\pi_q^*$, mathematically I'm not sure why the expectation $E_{a\sim\pi}$ before $Q_{\pi}(s,a)$ can be simply dropped.

"we try to find a policy that maximize the state-action value function Q", really? Or are we trying to find a policy that is assumed to be greedy wrt to the optimal state-action function? In an MDP, there's a unique optimal value function  and state-action value function . I am not sure I understand your question or confusion. It seems that you're mixing the theory (mathematical results) with the practice (algorithms). However, asking if two algorithms lead to the same optimal policy is reasonable because there can be more than one. So, can you clarify what you're really asking here?

Your first equation has a problem - an optimal policy is not the one that maximizes the value of a particular state $s$ - rather it maximizes the value of any state. The second equation has the same problem, along with having $a$ is unspecified - here, you should take the expectation w.r.t $\pi$.

I am not sure your equations are correct or I understand very well your question but I write the main equations of $V$ and $Q$ functions from Sutton book that may help you:
$ v^{*}(s) = max v_{\pi}(s) $
$ q^{*}(s,a) = max q_{\pi}(s,a) $
$  v^{*}(s)=max\mathbb{E}[R_{t+1}+\gamma v^{*}(s+1)|S_{t}=s, A_{t}=a] $
$  q^{*}(s,a)=\mathbb{E}[R_{t+1}+\gamma max q^{*}(s+1,a^{'})|S_{t}=s, A_{t}=a] $
$  v^{*}(s)=max q^{*}_{\pi}(s,a) $
As you can see, the optimal value function is equal to the maximum Q function on all feasible actions. We will reach optimal policy if we calculate the $argmax$ of these functions on possible actions. Also, the policy found by the value function is not always the same as the Q function because their features are different. The Q function considers both actions and states that not only decrease the speed of the process but also choose the shortest optimal path. For example, you can test simple graphical navigation on below Github repository:
https://github.com/pouyan-asg/global-path-planning
In this example, you will see the path that agent chooses to reach the destination is not similar in both cases, and the timing in value function mode is higher (because the agent does not consider the action). I hope my answer is clear.

As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.


text:
How does one detect training instabilities in DQN?
I am curious what training instabilities look like in a standard dqn, with or without a target network. I'm assuming the loss function would never converge since the difference between the predicted q-values and td-target would not reduce. I'd like it if someone could give more identifying symptoms of this problem.


text:
What are the possible ways to handle imbalance in multi-class image datasets?
Image imbalance is one of the major factor in the performance of DL model. Some of the methods that I found to tackle this are oversampling, under-sampling, SMOTE. Over-sampling has cons as it makes model to be overfit.undersampling results in loss of useful information.Again using SMOTE technique also won't works well on image datasets(as per web references).
What is the right way to handle imbalance in image datasets (multi-class problem)?

Yes this is certainly possible...
What you want to do is apply a weight to particular classes by proportion of the imbalance(assuming nothing else related to the problem is of note).
See this post for details


text:
Is there any proper literature on the types of features that different layers of a deep neural network learn?
Let's consider a deep convolutional network. It seems that there is some consensus on the following notions:
1. Shallow layers tend to recognise more low-level features such as edges and curves.
2. Deeper layers tend to recognise more high-level features (whatever this means).
While I usually come across various online articles and blogs that state this, no one ever cites literature that supports this claim. I am not seeking the question as to why this phenomenon happens, I'm only seeking whether it has actually been experimented on and documented. Also, I am barely able to find any peer-reviewed literature that provides evidence of this on sites such as Google Scholar or ResearchGate.
Could anyone point me to the right direction?

https://arxiv.org/abs/2106.14587 is the most exhausting, most fundamental work on this - it connects the architecture of NN with neural manifolds (manifolds of activities) and with categories of theories and languages. All is one system, layers (graphs, neural manifolds, languages-theories) are category theoretic fibrations. Just define learnability meatrics on this structurue and this can solve the problem of finding the optimal architecutre for the task.

It is assumed that NNs build up a hierarchical representation, whereby each layer combines features from the lower-level layers. The layers could be understood as representing a cascade of stacked features:
edges -> texture -> patterns -> parts -> objects
So from lower-level patterns to the more abstract higher-level concept like representation. This Distill article as far as I can tell is one of the most cited sources (740 citations) and provides an in-depth explanation of the features and how to visualize them. The journal is peer-reviewed.
The post also points to some older references such as: this, this or this.   The website of Chris Olah one of the authors of the Distill article is also a great source for finding visualizations for different deep learning architectures.


text:
Should importance sample weighting be compensated for by dynamically increasing learning rate?
I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along with the PER samples to downplay each sample's loss contribution according to its probability of selection. I've observed that typically these sample weightings vary from $~0.1$ to $<0.01$, as the buffer gradually fills up (4.8M samples).
When using this compensation, the growth of the maximal Q value per episode stalls prematurely compared to a non-weight-compensated regime. I presume that this is because the size of the back-propagation updates is being greatly and increasingly diminished by the sample weights.
To correct for this I've tried taking the beta-adjusted maximum weight as reported by the PER (the same buffer-wide value by which the batch is normalized) and multiplying the base learning rate by it, thereby adjusting the optimizer after each batch selection.
My question is two-fold:

Is this the correct interpretation of what's going on?
Is it standard practice to compensate for sample weighting in this way?

Although it seems to be working in keeping the Q growth alive whilst taming the loss, I've not been able to find any information on this and haven't found any implementations that compensate in this way so have a major doubt about the mathematical validity of it.


text:
Is it effective to concatenate the results of mean-pooling and max-pooling?
Is it popular or effective to concatenate the results of mean-pooling and max-pooling, to get the invariance of the latter and the expressivity of the former?

I haven't seen it as you describe and I don't think it would be much useful. Pooling layers are being gradually phased out of networks, because they don't seem to be that useful anymore. With the emergence of more and more conv-only architectures, I don't see that likely.


text:
What is the difference between machine learning and quantum machine learning?
What is the difference between machine learning and quantum machine learning?

The following questions on Quantum Computing SE Is there any potential application of quantum computers in machine learning or AI?, Will deep learning neural networks run on quantum computers? may also be useful.

The difference is much simpler than you might have anticipated: In the quantum computing community, machine learning algorithms designed to be used on quantum computers as opposed to classical computers, would fall under "quantum machine learning". There's really nothing more to it!
There is a short paper published in Nature called "Quantum Machine Learning" which was mentioned before, and it might give you all the answers you need about what "Quantum Machine Learning" is.

And this is different from "machine learning (on a classical computer) for quantum mechanics / quantum chemistry / quantum properties)"

@GeoffHutchison A very important distinction indeed! "Machine Learning for studying Quantum" is very different from "Quantum being used to do Machine Learning"!


text:
CBIR Evaluation on contextually different data
How good would a CBIR system trained on a dataset, for example, DELF, trained on the Google Landmarks dataset, perform when evaluated on a contextually different dataset such as the WANG or the COREL dataset without retraining?

Hi. You say "trained on a dataset, for example, DELF, trained on the Google Landmarks dataset", but a dataset cannot be trained on another dataset. Please, fix the wording of your post and add links to the relevant datasets and models.


text:
Can maximum likelihood be used as a classifier?
I am confused in understanding the maximum likelihood as a classifier. I know what is Bayesian network and I know that ML is used for estimating the parameters of models. Also, I read that there are two methods to learn the parameters of a Bayesian network: MLE and Bayesian estimator.
The question which confused me are the following.

Can we use ML as a classifier? For example, can we use ML to model the user's behaviors to identify the activity of them? If yes, How? What is the likelihood function that should be optimized? Should I suppose a normal distribution of users and optimize it?
If ML can be used as a classifier, what is the difference between ML and BN to classify activities? What are the advantages and disadvantages of each model?

Hi Atena and welcome to this community. Please, ask just one question per post!

If you read nothing else, maximum likelihood estimate => chance that the data predicted is the data observed. If you have a range of points (2, 3, 4, 5, 71) your MLE is going to favour ~4.5 because of means and standard deviations. MLE speeds up finding good input parameters, usually for a different classifier.
To answer your question:
1) Columbia University have a great example of using MLE classifiers, where everything is broken down into bitesize (or bytesize) chunks. Read this. Seriously.
2) In short, MLE is best used for simple, univariable distributions. It doesn't scale well to big problems but is waaay faster than a Bayesian network for simple tasks like predicting your height based on the heights of your immediate relatives. If you want to get technical, the conditional probability network of the Bayesian model reveals insights faster than the chain multiplication of the more primitive MLE.
Hope it helps!

Yes, it's called hypothesis testing but normally you need a little bit more than pure MLE.


text:
How do we determine whether a heuristic function is better than another?
I am trying to solve a maze puzzle using the A* algorithm. I am trying to analyze the algorithm based on different applicable heuristic functions.
Currently, I explored the Manhattan and Euclidean distances. Which other heuristic functions are available? How do we compare them? How do we know whether a heuristic function is better than another?

In the A* algorithm, at each iteration, a node is chosen which minimizes a certain function, called the evaluation function, which, in the case of A*, is defined as
$$f(n)=g(n)+h(n)$$
where $g(n)$ is the length (or cost) of the cheapest path from the start node to the current node $n$ and $h(n)$ is the heuristic function that estimates the cost of the cheapest path from current node $n$ to goal node.
There is potentially more than one path to the goal from a given node $n$. However, one of these paths is the cheapest path. An admissible heuristic function is a heuristic function that does not overestimate the cost to reach the goal node, that is, it estimates a cost to reach a goal that is smaller or equal to the cheapest path from $n$, which is denoted by $h^*(n)$. Therefore, an admissible heuristic $h$ satisfies $h(n) \leq h^*(n), \forall n$. Given that the goal is to find the cheapest path from a start to a goal node, intuitively, an admissible heuristic is an optimistic predictive function.
A* is guaranteed to find the optimal solution (or path) if it uses an admissible heuristic. In section 2.4 of the book Principles of Artificial Intelligence (1982), Nils J. Nilsson provides the proof of this fact.
However, not all admissible heuristics give the same information, so not all admissible heuristics are equally efficient. For instance, a heuristic function that is trivially admissible is $h(n) = 0, \forall n$. However, in this case, the only actual information that is used to choose the next node to expand is only based on $g(n)$, that is, $f(n) = g(n)$. This evaluation function corresponds to the evaluation function of the uniform-cost search algorithm, which is an uninformed-search algorithm (as opposed to A*, which, nonetheless, is considered an informed-search algorithm).
Which admissible heuristic is thus more informed? Consider two versions of A*, each with a different admissible heuristic function
$$
f_1(n) = g_1(n) + h_1(n)
$$
and
$$
f_2(n) = g_1(n) + h_2(n)
$$
where $h_1(n) \leq h^*(n), \forall n$ and $h_2(n) \leq h^*(n), \forall n$. Then A* with the evaluation function $f_1$ is more informed than A* with $f_2$ if, for all non-goal nodes $n$, $h_1(n) > h_2(n)$. See section 2.4.4. of the cited book where an example that attempts to show this is given.
The admissibility of a heuristic depends on the problem. For example, in the case of the Fifteen Puzzle problem, both Manhattan and the Hamming distances are admissible heuristics. However, in other problems, these distances might not induce an admissible heuristic.


text:
How does the network know which objects to track in the paper "Label-Free Supervision of Neural Networks with Physics and Domain Knowledge"?
I was reading the paper Label-Free Supervision of Neural Networks with Physics and Domain Knowledge, published at AAAI 2017, which won the best paper award.
I understand the math and it makes sense. Consider the first application shown in the paper of tracking falling objects. They train only on multiple trajectories of the said pillow, and during the evaluation, they claim that they can track any other falling object (which may not be pillows).
I am unable to understand how that happens? How does the network know which object to track? Even during the training, how does it know that it's the pillow that it's supposed to track?
The network is trained to fit a parabola. But any parabola could fit it. There are infinite such parabolas.

They are not tracking anything, instead they are trying to find an object which satisfies free fall equation. Gravity acts the same regardless of object's properties - at least in vacuum.
"In this paper, we model prior knowledge on the structure
of the outputs by providing a weighted constraint function
g used to penalize “structures” that are not
consistent with our prior knowledge."
There is a restriction to the possible parabola that is given in the last equation on page 2. They are training the Network to learn time dependence of that equation, which is the same for all objects.


text:
Is Mask R-CNN suited to solve a multi-class classification problem where the classes are related?
I want to create a model to solve a multi-class classification problem.
Here are more details about my problem.

Every picture contains only one object

The background is very simple

All objects belong to the same family of objects (for example, all objects are knives), but there are different specific subtypes

the model will learn and predict the name of the object (example: the model learn all types of knives, and when it get an image it will tell us the name of the knife)

To be clear, let's say I have 50 types of knives, and the output of the model has to recognize the correct name of the knife. Knife name could be:

Chef's Knife,
Heavy Duty Utility Knife,
Boning Knife, etc.

To solve this problem, I have started to use annotated, segmented (masked) images (COCO-like dataset) and the Mask R-CNN model.
As a first step, I got a prediction, but I really don't know if I'm on the right way.
For this problem, Mask R-CNN could be the solution, or it is impossible to recognize a tiny difference between two objects from the same class (for example Chef's Knife, Heavy Duty Utility Knife)?

Mask RCNN can be a very heavy function for a simple class classification. It is designed to handle multiple object in a single image. So I would suggest you could use much simpler models like VGGnet or Resnet which are backbones of the MaskRCNN.
The biggest hurdle you might face is the dataset. If you are trying to capture even small difference between knives and classify them you will need atleast 2000 images per knife type. which might be difficult to get. You may have to do a data augmentation.


text:
Are there RL techniques to deal with incremental action spaces?
Let's say we have a problem that can be solved by some RL algorithms (DQN, for example, because we have discrete action space). At first, the action space is fixed (the number of actions is $n_1$), and we have already well trained an offline DQN model. Later, we have to add more actions for some reasons (and the number of action is now $n_2$, where $n_2 > n_1$).
Are there some solutions to update the value function or policy (or the neural network) with only minor changes?

Is there some solutions to update the model with only minor changes?

In general, assuming the new action choices are meaningful - in at least some states, the expected return from taking one of the new actions is higher than the current optimal policy using just the old action selection - then the answer here is "no".
At the very least you will need to re-train your agent such that it explores the new action choices, and learns the new value function and policy. You can of course start this re-training using data and internal representations learned from the earlier environment, and that may help if the new actions have not changed things too radically.
There are a couple of things that might help improve performance on this re-training:

If the actions are not entirely discrete, but have some features that could be generalised from, you could base your value function or policy function estimators on those features instead of discrete actions. So for example in DQN your input to the neural network would be concatenated state, action feature vectors, and output a single value. Then it may generalise to the new actions quickly, in some cases perhaps even getting close to the correct value estimates from the start.
If you were training using DynaQ+, this includes an exploration term (which is added to planning assessments of immediate reward) that will prioritise exploring new state/action pairs when they appear. Other planning algorithms may have similar adjustments, although I am not aware of specific ones that could be dropped straight into a DQN agent.

If you know in advance which states the new actions are likely to be most useful in, you may be able to insert that knowledge into some initial action selection or value estimate helpers to avoid the need to train from scratch.


text:
Why do both sine and cosine have been used in positional encoding in the transformer model?
The Transformer model proposed in "Attention Is All You Need" uses sinusoid functions to do the positional encoding.
Why have both sine and cosine been used? And why do we need to separate the odd and even dimensions to use different sinusoid functions?


text:
How is the actor-critic algorithm guaranteed to converge?
From my understanding, the critic evaluates the policy (actor) following dynamic programming (DP) or approximate dynamic programming (ADP) scheme, which should converge to the optimal value function after sufficient iterations. The policy (actor) then updates its parameter w.r.t the optimal value function using gradient methods. This policy evaluation and improvement circle are repeated until neither the critic nor the actor changes anymore.
How's guaranteed to converge as a whole? Is there any mathematical proof? Is it possible that it may converge to a local optimal point instead of a global one?

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms where the critic is a neural network (function approximation). Most convergence proofs assume that the actor and the critic operate at different time scales, but, for example, in the paper A Convergent Online Single Time Scale Actor-Critic Algorithm (2010) this assumption is not made.
In the paper Incremental Natural Actor-Critic Algorithms (2007), the authors propose four different AC algorithms that use function approximation (neural networks) to represent the critic. Three of these proposed AC algorithms are based on natural policy gradients. In section 6 of the extended and more technical version of the mentioned paper, Natural Actor-Critic Algorithm (2007), the authors prove the convergence of the parameters of the policy and value function to a local maximum of an objective (or performance) function, which corresponds to the average reward (equation 2) plus a measure of the temporal-difference (TD) error inherent in the function approximation.

This is the correct answer. I will note that convergence guarantees will also vary dependent on discrete vs continuous state/action spaces.


text:
Why will the sigmoid function be 1 (and 0), if we use a fully connected layer that produces a big enough positive (or negative, respectively) output?
I am using a fully connected neural network that uses a sigmoid activation function. If we feed a big enough input, the sigmoid function will finally become 1 or 0. Is there any solution to avoid this?
Will this lead to classical sigmoid problems vanishing gradient or exploding gradient?

I think I am stuck with the use of sigmoid(since I am looking for a probability ) but I wonder how to increase the precision to 256 what do you mean by that ?

The other problem is when using sigmoid in an FNN with more than 1 hidden layer is that backpropagation degrades and weights in the first layer will only be updated by small values that might not have a significant impact. Relu fixes this problem. I agree with @DuttaA use softmax if you want to get a probability.

Can you elaborate on the task you're intending to solve with your network? As others have already said, ReLUs are the best option in intermediate layers and it's probable that you could do with logit (pre-sigmoid) outputs instead of going through the sigmoid for backprop purposes.

In general, it's better to not use the sigmoid function in any hidden layer. There are many other great options such as ReLU and ELU. However, if for any reason you have to use a sigmoid-like function, then go with the Tanh function, at least it has ~0 mean.


text:
How to calculate the false positives and negatives?
I have a huge amount of data and I want to calculate my false positive and false negative. Is there a software that can help me determine it?

Yes, you can use sklearn's confusion_matrix. To explicitly extract the false positives and negatives, you can do
from sklearn.metrics import confusion_matrix

y_true = [0, 1, 0, 1]
y_pred = [1, 1, 1, 0]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()


text:
What are the pros and cons of using standard deviation or entropy for exploration in PPO?
When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations :
Exploration with std

Collect trajectories on $N$ timesteps, by using a policy-centered distribution with progressively trained std variable for exploration
Train policy function on $K$ steps
Train value function on $K$ steps

For example, the OpenAI's implementation.
Exploration with entropy

Collect trajectories on $N$ timesteps, by using policy function directly
Train policy and value function at the same time on $K$ steps, with a common loss for the two models, with additional entropy bonus for exploration purpose.

For example, the PPO algorithm as described in the official paper.
What are the pros/cons of these two algorithms?
Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general?

Both implementations may be closer than you think.
In short:
PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising actions/policies. And there is a term added to the loss function aiming to prevent a collapse of the noisiness, to help ensure exploration continues and we don't get stuck at a bad (local) equilibrium.
In fact, for continuous action, the term for entropy in the loss function you describe in Ex. 2, can make sense only when the actions are stochastic, i.e. when there action choice has some standard deviation the way you describe in Ex. 1.
More detail:
On one hand, PPO (at least for continuous action), trains a central/deterministic value (say the mean policy or close to mean) targeting a most profitable action path. On the other hand, along with it, a standard deviation making the actions a random draw with noise around the deterministic value. This is the part you describe in Example 1. The noise helps explore new paths and to update the policy according to the rewards on these sampled paths.
Entropy itself is a measure of the noisiness of the draws, an thus also an indirect indicator for the trained standard deviation value(s) of the policy.
Now, entropy tends to decay as training progresses, that is, the random draws become progressively less random. This can be good for reward maximization - really the best draws are taken for reward maximization - but it is bad for further improvements of policy: improvement may halt or slow down as exploration of new action paths fades.
This is where entropy encouragement comes in. PPO foresees the inclusion of entropy in the loss function: we reduce the loss by x * entropy, with x the entropy coefficient (e.g. 0.01), incentivizing the learning network to increase the standard deviations (or, to not let them drop too much). This part is what you describe in Example 2.
Further notes:

During exploitation, we'd typically turn off the noise (implicitly assuming action std = 0) and pick deterministic actions: in normal cases this increases the payoffs; we're choosing our best action estimate, rather than at a random value around it.
People are not always precise when referring to the model's entropy vs. the entropy coefficient added to the loss function.
Other RL algorithms with continuous action tend to use noisy drwas with standard deviations/entropy too.

the entropy of a constant distribution is zero.


text:
Do the eigenvectors represent the original features?
I've got a test dataset with 4 features and the PCA produces a set of 4 eigenvectors, e.g.,
EigenVectors: [0.7549043055910286, 0.24177972266822534, -0.6095588015369825, -0.01000612689310429]
EigenVectors: [0.0363767549959317, -0.9435613299702559, -0.3290509434298886, -0.009706951562064631]
EigenVectors: [-0.001031816289317291, 0.004364438034564146, 0.016866154627905586, -0.999847698334029]
EigenVectors: [-0.654824523403971, 0.2263084929291885, -0.7210264051508555, -0.010499173877772439]

Do the eigenvector values represent the features from the original dataset? E.g., is feature 1 & 2 explaining most of the variance in eigenvector 1?
Am I interpreting the results correct to say that features 1 and 2 are therefore the most important in my dataset since PC1 represents 90% of the variance?
I'm trying to map back to the original features but am unsure how to interpret the results.

The principal components (eigenvectors) correspond to the direction
  (in the original n-dimensional space) with the greatest variance in
  the data.
The corresponding eigenvalue is a number that indicates how much
  variance there is in the data along that eigenvector (or principal
  component).

Thus, feature 2 is the most important (based on eigenvalue alone). Then feature 1. The other 2 features have little impact and theoretically could be removed as a part of your data reduction effort.
Also, it is important to point out that

When performing PCA, it is typically a good idea to normalize the data
  first. Because PCA seeks to identify the principal components with the
  highest variance, if the data are not properly normalized, attributes
  with large values and large variances (in absolute terms) will end up
  dominating the first principal component when they should not.

IOW, if you didn't normalize your data then your PCA analysis is quite likely meaningless.
*The above quoted text is from http://www.lauradhamilton.com/introduction-to-principal-component-analysis-pca


text:
Why are model-based methods more sample efficient than model-free methods?
Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two reasons for this question:

We can potentially obtain more samples from the learned model, which may speed up the learning speed.
Models allow us to predict the future states and run simulations. This may lead to more valuable transitions, whereby speeding up learning.

But I heavily doubt this is the whole story. Sincerely, I hope someone could share a more detailed explanation for this question.

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.
Main difference between those two is like you said, that the model helps in finding the correct path more efficiently because of the existence of the model. Maybe you cannot find new samples (point 1) but you know better the whole inner logic about the system and instead of just knowing what to do with the specific sample, you can relate it to the whole picture (sort of like in point 2: you can play with the choices) and make more profound calculations.
The article had a comparison, which told that you are writing a map in a city about every possible direction you can take when you are model-based and while in model-free you can enter specific places and remember which direction was best based on last visits but you still never know where you are coming nor going exactly.
In other words, if you think you're teaching a taxi driver on a big city with many signs and rules, model based guy would drive more precisely sooner because the sign language (the inner logic and model of the city) helps them on understanding the map sooner than just reacting crossing by crossing somewhat by chance all the time.
Sample efficiency tells what is the amount of information fetched from one sample [2]. Model-based machine can adjust model, maybe make some calculations about expected rewards AND after that the same as model-free, adjust the common policy. Model-free does only have the policy. Again the taxi guys: model-free guys know that last time and second last I stopped in crossing, model-based guy knows also it was due to red lights in pole. Third time model-free guy is the first in row and BANG - hits the crossing car. Next time rule is there comes sometimes cars and model-based guy knew that from the first place.

My sources:
[1] https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698
[2] What is sample efficiency, and how can importance sampling be used to achieve it?


text:
How can I cluster based on the complementary categories?
K-means tries to find centroid and then clusters around the centroids. But what if we want to cluster based on the complement?
For example, suppose we have a group of animals and we want to cluster Dogs, Cats, (Not Dogs and Not Cats). The 3rd category will not arise from mean clustering.

generally clustering (as you examplify with k-means) requires a representation. What would not refer to in this space?

Why do you say that the 3rd category will not be arisen from the mean clustering? What is the 3rd category? Not Dogs? Btw, what do you mean by "mean clustering"? Do you mean "k-means cluster algorithm"?

Note: K-means does not assume an interpretation/label of the clusterings - in fact it is an unsupervised algorithm. The interpretations are a result of human analysis after running K-means.
For example, in the case of cats and dogs one would most definitely chose k = 2 - which provides an easy interpretation. However, what would it mean if we set k = 1000. We no longer have a "clean" interpretation of the centroids.
Note: how I keep saying "interpretation." The algorithm simply assigns a data point to a cluster and calls it a day. Humans then look at the results and try to understand them with an interpretation.
Continuing with the example where k = 2. One could easily interpret "is cat" as "not dog" and "is dog" as "not cat." The idea here is that the data is unlabeled beforehand and humans try to fathom the results retrospectively by assigning the resulting clusters with an understandable label.
I hope this clarifies the issue.

The purpose of clustering is to look for the same characteristics in the data and group them into clusters. The number of clusters we take is based on how our clustering algorithm evaluated.
For example, we use the Elbow Method for evaluates. We take optimal number of clusters that distortion start decreasing in a linear fashion. In the picture below, the optimal number of clusters for the data is 3.


text:
Wasserstein GAN with non-negative weights in the critic
I want to train a WGAN where the convolution layers in the critic are only allowed to have non-negative weights (for a technical reason). The biases, nonetheless, can take both +/- values. There is no constraint on the generator weights.
I did a toy experiment on MNIST and observed that the performance is significantly worse than a regular WGAN.
What could be the reason? Can you suggest some architectural modifications so that the nonnegativity constraint doesn't severely impair the model capacity?


text:
When to use RMSE as opposed to MSE and vice versa?
I understand that RMSE is just the square root of MSE. Generally, as far as I have seen, people seem to use MSE as a loss function and RMSE for evaluation purposes, since it exactly gives you the error as a distance in the Euclidean space.
What could be a major difference between using MSE and RMSE when used as loss functions for training?
I'm curious because good frameworks like PyTorch, Keras, etc. don't provide RMSE loss functions out of the box. Is it some kind of standard convention? If so, why?
Also, I'm aware of the difference that MSE magnifies the errors with magnitude>1 and shrinks the errors with magnitude<1 (on a quadratic scale), which RMSE doesn't do.

Thanks @DuttaA , I think the comment you've given is quite good, that it can be one of the answers to this question. So, please post it as an answer below :)

I don't believe there is any practical difference. Someone should provide a reference to a source that says there is a difference if you think there is one.

Answers in the comments are decent, particularly DuttA.
DuttA gives these, approximately

ease of derivative
Don't have to worry about ~0 in denominator causing huge gradient
But to me the most important is mathematical convenience, someone
might easily make the mistake of RMSE is just equal the difference
y−y′ instead of root of mean square of y−y′. The answer this might
start depending on conventions.
In maths (Don't know the reason and might be inaccurate) we mainly
work with variances instead of standard deviation.

Here are my reasons for using the MSE instead of RMSE:

Doesn't have the sqrt operations, so it computes faster
the square root isn't easy, its Newtons method, so it could be a
dozens steps per iteration
MSE has all the information of RMSE, there is 1-to-1 mapping, so no loss
the storage of the square root typically doesn't save any memory in
IEEE-784 and compute vs. memory is big thing in complexity
tools like gradient boosted machine can "recycle" the squared error
computation for speedup and working on O(n) complexity
there is hidden scaling and regularization because many gpu hardware
elements are fundamentally 8-bit, so if you can make your code more
8-bit in its guts then you don't have as much in the back-conversion
and it runs a lot faster


text:
What does an oscillating validation error curve represent?
I have been training my CNN for a bit now and, while both the training loss and the training error curves are going down during training, both my validation loss and my validation error curves are kind of zig-zagging and oscillating along the epochs. What does this represent?

This is an old question, but if it would be great if you could provide a plot of the training and validation loss and error. Moreover, you should provide more context, such as provide the architecture of your CNN. If you're using TF, you could do plot_model(my_model).

Without more information the best diagnostic is:
You potentially have a bug in your code.
Justification
Neural network systems are capable of failing silently. The system can still appear to "learn" even in the presence of said bug - a while back I had a very similar issue with a toy CNN project. My network could get 99% accuracy on the training set but always achieved 8-13% on the validation set. This looked like overfit but none of the methods solved the issue. Finally, I found that I wasn't feeding the data correctly into the network during train time but I was feeding the data correctly during validation time.
Conclusion
Provide the following for better diagnostic:

Is this a custom CNN you hardcoded?
are you using python?
what's the objective function?
can you provide the loss curve for the training set?
show us some code too (this will be very helpful for working toward a solution)

I hope this helps, and best of luck!


text:
How do I generate structured light for the 3D bin picking system?
I want to know how to generate the structured light which projects different patterns of light on a 3D object which is under scanning.


text:
How to implement fisherface algorithm and how much time will it take?
I found on the web that fisherface is the best algorithm for face detection. Before investing deeply into it, I just want to know how hard is it to implement it and how much time will it take.
I am new to this website and I welcome any suggestions.


text:
Feasibility of using machine learning to obtain self-consistent solutions
I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way coupled systems where each system is described by a partial differential equation. The equations are generally unfolded in a numerical grid of interest and then solved iteratively until a self-consistent solution is obtained.
A well-known example is the Schrödinger-Poisson solver. Here, for a given nano/atomic structure, we assume an initial electron density. Then we solve the Poisson equation for that electron density. The Poisson equation tells us the electrostatics (electric potential) of the structure. Given this information, we solve the Schrödinger equation for the structure which tells us about the energy levels of the electrons and their wave functions in the structure. But one would then find that this energy levels and wavefunctions correspond to a different electron density (than the one we initially guessed). So we iterate the process with the new electron density and follow the above mentioned procedures in a loop until a self-consistent solution is obtained.
Often times the iteration processes are computationally expensive and extremely time-consuming.
My question is this: Would the use of deep learning algorithms offer any advantage in modeling such problems where iteration and self-consistency are involved? Are there any study/literature where researchers explored this avenue?