text:
How robust are deep networks to class imbalance?
Before deep learning, I worked with machine learning problems where the data had a large class imbalance (30:1 or worse ratios). At that time, all the classifiers struggled, even after under-sampling the represented classes and creating synthetic examples of the underrepresented classes -- except Random Forest, which was a bit more robust than the others, but still not great.
What are guidelines for class distribution when it comes to deep learning (CNNs, ResNets, transformers, etc)? Must the representation of each class be 1:1? Or maybe it's "good enough" as long as it is under some ratio like 2:1? Or is deep learning completely immune to class imbalance as long as we have enough training data?
Furthermore, as a general guideline, should each class have a certain minimum number of training examples (maybe some multiple of the number of weights of the network)?

I don't think that neural networks are immune to class imbalance, but I'm not really an expert on this topic. One way to deal with class imbalance could be to make the NN uncertainty-aware. Maybe you could look at this paper. Feel free to provide an answer to your own question, once you know the answer, if nobody provides an answer meanwhile.

Thanks! Will give it a read and post an answer later

@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC AUC drops by 5-10%, and accuracy decreases by 20-30%. These effects are worsened on more complex tasks.
There are 3 noteworthy class imbalance approaches to partially alleviate this:

Undersampling: sampling the over-represented class less often
Oversampling: sampling the under-represented class more often
Thresholding: after the neural network learns the weights, during inference, multiply the output class probability by a weight (the class prior, which is different for each class). The weight is the inverse of the class representation of the dataset (i.e. the inverse of $\frac{numInstances(c)}{\displaystyle \sum_i numInstances(i)}$, where $c$ is the current class and $numInstances(i)$ is the number of unique instances of class $i$ in the training set.

The paper concludes with the following best practices:

Regarding the choice of a method to handle CNN training on imbalanced dataset we conclude the following.

The method that in most of the cases outperforms all others with respect to multi-class ROC AUC was oversampling.

For extreme ratio of imbalance and large portion of classes being minority, undersampling performs on a par with oversampling. If training time is an issue, undersampling is a better choice in such a scenario since it dramatically reduces the size of the training set

To achieve the best accuracy, one should apply thresholding to compensate for prior class probabilities. A combination of thresholding with baseline and oversampling is the most preferable, whereas it should not be combined with undersampling.

Oversampling should be applied to the level that completely eliminates the imbalance, whereas the optimal undersampling ratio depends on the extent of imbalance. The higher a fraction of minority classes in the imbalanced training set, the more imbalance ratio should be reduced.

Oversampling does not cause overfitting of convolutional neural networks, as opposed to some classical machine learning models.

The last point is very interesting, because oversampling is known to cause overfitting in classical machine learning models and many have advised against doing it.


text:
What is the gradient of an attention unit?
The paper Attention Is All You Need describes the Transformer architecture, which describes attention as a function of the queries $Q = x W^Q$, keys $K = x W^K$, and values $V = x W^V$:
$\text{Attention(Q, K, V)} = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right) V \\
= \text{softmax}\left( \frac{x W^Q (W^K)^T x}{\sqrt{d_k}} \right) x W^V$
In the Transformer, there are 3 different flavors of attention:

Self-attention in the Encoder, where the queries, keys, and values all come from the input to the Encoder.
Encoder-Decoder attention in the Decoder, where the queries come from the input to the Decoder, and the keys and values come from the output of the Encoder
Masked self-attention in the Decoder, where the queries, keys and values all come from the input to the Decoder, and, for each token, the $\text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right)$ operation is masked out (zero'd out) for all tokens to the right of that token (to prevent look-ahead, which is cheating during training).

What is the gradient (i.e. the partial derivatives of the loss function w.r.t. $x$, $W^Q$, $W^K$, $W^V$, and any bias term(s)) of each of these attention units? I am having a difficult time wrapping my head around derivating a gradient equation because I'm not sure how the softmax function interacts with the partial derivatives, and also, for the Encoder-Decoder attention in the Decoder, I'm not clear how to incorporate the encoder output into the equation.

Let me try to understand what you're really looking for. You're not looking for all the details of the partial derivatives with respect to the parameters, i.e. you're not looking for someone to describe to you how back-prop works, right? If I am right, can you please explain what type of answer you're looking for? Anyway, I suspect that the original authors of the transformer did not have to compute the gradients by hand, but let the library perform automatic differentiation (i.e. back-prop) automatically, as it's usually the case. You just need to make sure your operations are differentiable.

I understand how backprop works in general for deep networks, networks with convolutions, and networks with residuals. I guess I am asking for what the mathematical equation of a gradient is for a specific layer of the neural network, where the layer is an attention unit. Specifically, deriving the partial derivatives of the loss w.r.t. the input $x$, weights $W^Q$, $W^K$, $W^V$, and the bias terms (if the attention unit has any -- I wasn't clear on that from the paper), assuming we have the partial derivatives of the later layers already. This may be solved by software, but I'm still curious!

Have you got any solution for this. If so please let me know. I'm also trying the same

https://math.stackexchange.com/questions/4692899/derivation-of-the-attention-block-of-a-transformer

I have written a blog to answer this question, please see https://say-hello2y.github.io/2022-09-07/attention-gradient
I use Matrix calculus to solve this question, here I put the final result of the gradient of an attention unit.
$$
\frac{\partial f(X)}{\partial W_h^K }=\gamma K^T\mathbb{P}_h\frac{\partial f(X)}{\partial A  }^TQW_h^Q
$$
$$
\frac{\partial f(X)}{\partial W_h^Q }=\gamma Q^T\frac{\partial f(X)}{\partial A  }\mathbb{P}_{h}^{T}KW_h^K
$$
$$
\frac{\partial f(X)}{\partial W^v  }=A^T\frac{\partial f(X)}{\partial X  }(W^O)^T
$$
$$
\frac{\partial f(X)}{\partial W^O }=\mathrm{Concat}(\mathrm{head_1}, ..., \mathrm{head_H})^T\frac{\partial f(X)}{\partial X }
$$
For more detail, please see https://say-hello2y.github.io/2022-09-07/attention-gradient.If you have any questions, feel free to contact me ,my email is longxhe@gmail.com.

You didn't define several symbols.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is the Bellman Equation actually telling?
What does the Bellman equation actually say? And are there many flavours of that?
I get a little confused when I look for the Bellman equation, because I feel like people are telling slightly different things about what it is. And I think the Bellman Equation is just basic philosophy and you can do whatever you want with that.
The interpretations that I have seen so far:
Let's consider this grid world.
+--------------+
| S6 | S7 | S8 |
+----+----+----+
| S3 | S4 | S5 |
+----+----+----+
| S0 | S1 | S2 |
+----+----+----+

Rewards: S1:10; S3:10
Starting Point: S0
Horizon: 2
Actions: Up, Down, Left, Right (If an action is not valid because there is no space, you remain in your position)

The V-Function/Value:
It tells you how good is it to be in a certain state.
With a horizon of 2, one can reach:
S0==>S3 (Up)   (R 5)
S0==>S0 (Down) (R 0)
S0==>S1 (Right)(R10)
S0==>S0 (Left) (R 0)

From that onwards
S0==>S3 (Up)   (R 5)
S0==>S0 (Down) (R 0)
S0==>S1 (Right)(R10)
S0==>S0 (Left) (R 0)

S1==>S4 (Up)   (R 0)
S1==>S1 (Down) (R10)
S1==>S2 (Right)(R 0)
S1==>S0 (Left) (R 0)

S3==>S6 (Up)   (R 0)
S3==>S0 (Down) (R 0)
S3==>S3 (Right)(R 5)
S3==>S2 (Left) (R10)

Considering no discount, this would mean that it is R=45 good to be in S0, because these are the options. Of course, you can't grab every reward, because you have to decide. Do I need to consider the best next state yet, because this would obviously reduce my expected total reward, but as I can only make two steps it would tell me what is really possible. Not what the overall Reward R(s) in that range is.
The Q-Function/Value
This function takes a state and an action, but I am not sure. If that means that I have a reward function that just considers my actions as well to give me a reward. Because in the previous example I just have to land on a state (It doesn't really matter how I get there). But this time I get a reward, when I choose a certain action. R(s,a)
But otherwise I do not rate the best action and select that next state to calculate the next state. I choose every next step and from that I choose the 2nd next.
Optimization V-function or Q-function
This works the same as V-Function or Q-Function, but it just considers the next best award. Some sort of greedy approach:
First step:
S0==>S3 (Up)   (R 5) [x]
S0==>S0 (Down) (R 0) [x]
S0==>S1 (Right)(R10)
S0==>S0 (Left) (R 0) [x]

Second Step:
S1==>S4 (Up)   (R 0) [x]
S1==>S1 (Down) (R10)
S1==>S2 (Right)(R 0) [x]
S1==>S0 (Left) (R 0)

So, this would say that is the best I can do in two steps. I know that there is a problem, because when I just follow a greedy approach I risk that I won't get the best result, if I would have had a reward of 1000 on S2 later.
But still, I just want to know, if I have a correct understanding. I know there might be many flavours and interpretations but at least I want to know that is the correct name of these approaches.

Your main questions seem to be 1. "What is the Bellman equation telling us?" and 2. "Are there different variants of the BE?". These are already 2 distinct questions that should be asked in their own separate post, so this post is already too broad and I could close it as such. In addition to that, you're also talking about value functions and there are some details/calculations that I don't think are really relevant to answer your 2 main questions. So, I suggest that you remove all the details that are not really necessary to answer your 2 main questions because this post is already long.

The answers below seem to address what are the Bellman equations and what they are telling us. They do not seem to address any of the details or calculations that you're performing. If you're interested in specific details, I suggest that you ask another question specifically asking about something that you don't understand about those calculations or value functions. Alternatively, instead of removing all the details, you could leave just a simplified version of your calculations that is relevant to your 2 main questions.

Moreover, you start by asking "What does the Bellman equation actually telling? And are there many flavours of that?", but you're not even specifying which Bellman equation are you referring to. Then you say "I get a little confused when I look for the Bellman equation, because I feel like people are telling slightly different things about what it is", which people? Can you provide a link? That's what a good question should include (links and the context). This post is just too broad and brings up too many topics that are difficult to address in a single answer.

So, as I said, simplify this post as much as possible, but keep the relevant context and information. More importantly, now that there are already 2 answers (both of them seem to be good), try to leave the questions that those answers address. If you have more questions, ask them in another post, as I already said.

@nbro I will do that. Just give me a little time.

For a Markov Decision Process $(\mathcal{S}, \mathcal{A}, P, R)$ (here $P(s, s') = \mathbb{P}(S_{t+1} = s' | S_t = s, A_t = a))$;, let us define the value of being in a certain state. That is,
$$v_\pi(s) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[\sum_{i=0}^\infty \gamma^{i+t}r(s_{t+i}, a_{i+t}) | S_t =s\right].$$
That is, the value of being in state $s$ at time $t$ is equal to the expected value of the discounted sum of future rewards, where the expectation is taken with respect to $\pi(\cdot | s)$ the action policy and the environment dynamics $P$.
We will now define
$$\sum_{i=0}^\infty \gamma^{i+t}r(s_{t+i}, a_{i+t}) = G_t$$
Now, we can rewrite this as
$$\mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[G_t | S_t =s\right] =  \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma G_{t+1} | S_t =s\right].$$
The RHS is now in a nicer format and we can express it as
$$\mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[G_t | S_t =s\right] = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma v_\pi(s') | S_t =s\right],$$
where $s'$ is the state that we transition into at time $t+1$. Note that $s'$ is a random variable and we are taking the expectation over this according to the action policy and the environment dynamics -- this is because the state we transition into depends first on the action we select and then the environment dynamics $P$.
You can see that we now have
$$v_\pi(s) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + \gamma v_\pi(s') | S_t =s\right].$$
This is the Bellman equation (at least a form of it) and it expresses a recursive relationship between the values of states. That is, the value of being in state $s$ is equal to the expected immediate reward from being in this state plus the value of being in the state that we transition into. This relationship is useful in Reinforcement Learning as many algorithms use this equation to form update rules to approximate the value/state-action value function of the MDP, such as the  SARSA algorithm, so it is more than just a philosophy, it is the driving force behind many of the RL algorithms.
Now, when I say at least a form of it, that is because in RL it is also common to see a Bellman equation for the state action value function
$$q_\pi(s, a) = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ G_t | S_t = s, A_t = a\right] = \mathbb{E}_{a_i \sim \pi, s_i \sim P}\left[ r(s_t, a_t) + v_\pi(s') | S_t = s, A_t = a\right];$$
noting that the value function $v$ is the expectation of $q$ over the action space, i.e. we marginalise out the action we condition on -- thus there is the recursive relationship.
As pointed out in the comments for this question, it is also worth noting that Bellman equations originated in Dynamic Programming, which exist to solve planning problems such as the knapsack problem.

This answer provides a nice/concise derivation of how one can get to one Bellman equation, but, especially because the OP was confused about different definitions of the "Bellman equation", I think it is worth emphasing that when people say "Bellman equation" in RL they typically mean any of the recursive equations that involves some value function, not just the state value function, but what they specifically refer to may probably depend on the context. For example, Barto & Sutton sometimes refer to Q as also a Bellman equation.

So, first, it may also be worth saying that the Bellman equations are actually very important in RL, and they are not just "philosophy" (as the OP thinks). Moreover, you may also say that "Bellman equations" typically arise in dynamic programming (basically invented by Richard Bellman, hence the name) and that dynamic programming solutions exist to solve planning problems (i.e. value/policy iteration, which require the "model"), but DP is not just used in the context of MDPs, but used to solve traditional CS problems (the knacksack problem and so on).

Just to add to the previous answer some more background and intuition. The background of Bellman equation comes from optimal control theory of dynamic systems of form (in discrete time case)
\begin{equation}
s_{k+1} = f_d(s_k, a_k) \tag{1}
\end{equation}
where $s_k$ represents state at time $k$ and $a_k$ action at time $k$. The goal is to optimize multistage objective function of form
\begin{equation}
V_N(s_0) = \sum_{k=0}^N J(s_k, a_k)
\end{equation}
while satisfying dynamic constraints $(1)$ where $J(\cdot)$ is a stage cost at time $k$. The product of this optimization are optimal control policies $a_k = \pi_k(s_k)$ which provide optimal value for the multistage objective function. Bellman's principle of optimality states that, for the multistage optimization problems, the objective function value at timestep $k$ should satisfy
\begin{align}
V^*_{k}(s_k) &= \min_{a_k}[J(s_k, a_k) + V^*_{k-1}(s_{k+1})]\\
&= \min_{a_k}[J(s_k, a_k) + V^*_{k-1}(f_d(s_k, a_k))]
\end{align}
This can be of course proven, you can find the proof in any optimal control/dynamic programming book.
This also makes intuitive sense. Consider that you are at timestep $N$ (end of trajectory you want to optimize). You only need to consider 1 action $a_N$. You would now go through all possible states $s_N$ and pick action $a_N$ which minimizes your stage cost $J(s_N, a_N)$ for all those states separately. After you did that, you would go one step backwards in time and find optimal action $a_{N-1}$ for all states $s_{N-1}$. According to the Bellman optimality principle, you only need to consider action $a_{N-1}$ in your optimization, because you already know action $a_N$ (calculated previously) which would minimize $J(s_N, a_N)$ for any possible future state $s_N$. Then you would keep going backwards in time until $k=0$. This is very useful because you don't need to consider all $N$ timesteps at once, you only consider one timestep at a time which prevents combinatorial explosion for large $N$. Bellman optimality principle has been adapted to many different applications, including RL and stochastic systems.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to graphically represent a RNN architecture implemented in Keras?
I'm trying to create a simple blogpost on RNNs, that should give a better insight into how they work in Keras. Let's say:
model = keras.models.Sequential()
model.add(keras.layers.SimpleRNN(5, return_sequences=True, input_shape=[None, 1]))
model.add(keras.layers.SimpleRNN(5, return_sequences=True))
model.add(keras.layers.Dense(1))

I came up with the following visualization (this is only a sketch), which I'm quite unsure about:

The RNN architecture is comprised of 3 layers represented in the picture.
Question: is this correct? Is the input "flowing" thought each layer neuron to neuron or only though the layers, like in the picture below. Is there anything else that is not correct - any other visualizations to look into?

Update: my assumptions are based on my understanding from what I saw in Geron's book. The recurrent neurons are connected, see: https://pasteboard.co/JDXTFVw.png ... he then proceeds to talk about connections between different layers, see: https://pasteboard.co/JDXTXcz.png - did I misunderstand him or is it just a peculiarity in keras framework?

As you can see in fig 15-2, neurons in a layer don't communicate with each other, it is unrolled version in the 2nd part of 15-2, ie. it is the same layer across each timestep, there is no next layer in that diagram. In fig 15-1, a single neuron is shown (unrolled on right). What's happening for a single neuron in 15-1 is happening to every neuron in 15-2, there is no concept of multiple layers introduced in the images you referenced, they are different timesteps for the same neuron/layer.

Yea, I was getting the 2nd part of the argument you presented, now I also understand the first one - the confusion creeped in because the example was using a single neuron RNN as a simplified self-contained network and then added several cells in a single layer RNN. As you said - both pictures show single layer RNN unrolled through time, the difference being the first one has a single neuron and the other has multiple neurons in that layer. Thanks

The first image is correct. The information will flow from left to right in each layer and from top to bottom in between layers.

Thanks for the comment I take it that there should be 5 nodes in the picture, not 4? Also good point on the hidden state - I think I'm missing this part as well - is the hidden state passed to the dense layer and the y value (prediction)?

@SajanGohil yes you are right, I haven't noticed the return sequence=True on the second layer. I will update the answer.

@MindaugasBernatavičius Yes there should had been 5 nodes instead of 4. I have removed the picture since it's not mandatory anymore. Yes the hidden state is passed to the Dense layer. If you select return_sequences=True, each hidden state from the last RNN layer is passed through the dense layer. If you set return_sequences to false, then only the last hidden state from the last RNN layer is sent to the Dense layer.


text:
Understanding LSTM through example
I want to code up one time step in a LSTM. My focus is on understanding the functioning of the forget gate layer, input gate layer, candidate values, present and future cell states.
Lets assume that my hidden state at t-1 and xt are the following. For simplicity, lets assume that the weight matrices are identity matrices, and  all biases are zero.
htminus1 = np.array( [0, 0.5, 0.1, 0.2, 0.6] )
xt = np.array( [-0.1, 0.3, 0.1, -0.25, 0.1] )

I understand that forget state is sigmoid of htminus1 and xt
So, is it?
ft = 1 / ( 1 + np.exp( -( htminus1 + xt ) ) )

>> ft = array([0.47502081, 0.68997448, 0.549834  , 0.4875026 , 0.66818777])

I am referring to this link to implement of one iteration of one block LSTM. The link says that ft should be 0 or 1. Am I missing something here?
How do I get the forget gate layer as per schema given in the below mentioned picture? An example will be illustrative for me.

Along the same lines, how do I get the input gate layer, it and vector of new candidate values, \tilde{C}_t as per the following picture?

Finally, how do I get the new hidden state ht as per the scheme given in the following picture?
A simple, example will be helpful for me in understanding. Thanks in advance.

This is an image to better understand lstm... At $f_t$, we are taking the sigmoid of a weight matrix * the input at the current timestep + another weight matrix * $h_{t-1}$
Code Sample for $f_t$:
import numpy as np
import math

def sigmoid(values):
    sigmoid_applied = []
    for value in values:
        result = 1 / (1 + math.pow(math.e, -value))
        sigmoid_applied.append(result)
    return np.array(sigmoid_applied)

w1 = np.random.uniform(0, 1, size=[hidden_vector_len, input_len])
w2 = np.random.uniform(0, 1, size=[hidden_vector_len, hidden_vector_len])

f_t = sigmoid(np.dot(w1, input) + np.dot(w2, prev_hidden_state)) # Its matrix multiplication and not just simple multiplication

Note - There is also a bias term which I haven't included here for simplicity
If you understood $f_t$, you can do the same for other states also.
If you feel I am wrong anywhere in this post, then please do consider adding a comment

Shouldn't the size of w2 be [hidden_vector_len, hidden_vector_len] ?


text:
what is the correct approach for KNN in item based recommendation system?
if I make an application for movies and each user in the system can rate the movies. And I want to make a recommendation system to recommend movies to active user based on his rating for other movies. using item based collaborative filtering using KNN.
when we find the similarities between the movies and pick the top k items, which approach is correct?
1- calculate the similarities between all movies and then take the top k for every movie the user rated it highly. (the dataset is a matrix represent the rating values for each item from each user)
2- The KNN is applied to all the movies that the user likes, one after the other, and we find the similarity between each movie and the films that the user did not rate, so that for each film we take the top K of similar films (the user not rated yet) for each movie the user rated highly, then show it to the user . (the dataset for each time we apply knn is a matrix contain rating for each item the user rated and all other items that the user not rated yet).


text:
Can a convolutional neural network classify text document images?
I know convolutional neural networks are commonly used for image recognition, but I was wondering if they would be able to distinguish between predominantly text-based documents vs something like objects. For example, if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image? If not, is there a different AI technique better suited that is purely image-based, or would it require OCR and leveraging the text in the invoice?
Update: based on a comment, my ask my not be clear. I'm not trying to see if the CNN can differentiate between a document (mostly text based image) and a photo image. I want to know if based on a gif/jpeg/png of a document (no OCR performed) a CNN would be able to classify the documents, which basically could be used as a means of identifying the vendor.

When you say "be able to distinguish between predominantly text-based documents vs something like objects", are you asking if CNNs would be able to distinguish between 1. images that contain predominantly text and 2. images that contain predominantly objects? Then you say "if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image?", here "first page of invoices matched to a vendor name", I assume that invoices are examples of documents that contain mainly text...

but then you say "to predict the vendor based on an image", here "image" refers to the invoice, right? So, are you asking whether CNNs can actually perform classification of text-based documents?

@nbro Yes, I'm asking if they can perform classification of text-based documents, but without relying on OCR. So basically, classify based on how the document "looks". So, there's the same logo top left, similar position of text, etc. Granted, not two invoices will be the same, date will change, etc. Just wondering if a predominantly text based image provides enough nuance for a CNN to classify or not.

I will clarify the question.

Yes, it is possible, CNNs can also be used for OCR (see the MNIST task and this blog), although it's not the common way for OCR because it is considered a bit overkill and inefficient. Furthermore, OCR is considered a solved problem already and don't need deep learning to do well, unless perhaps under unfavorable conditions like dark lighting, complex background of the document, weird fonts, occlusion of text, etc.
Breaking down your questions:

a CNN can likely classify an image as "this image has a word document in it" vs "this image does not have a word document in it". The word document itself, and the outline of the words, should form distinct enough features for the CNN to discriminate against
a CNN can likely identify the vendor based on the logo of the vendor if you have enough training images of it at various distortions (angles, occlusions, different lighting, etc)
a CNN can likely do OCR well enough to extract the text, and then an entity extraction model can identify the vendor from the words. However, this approach would be uncommon as OCR is already considered a solved problem. But a neural network can potentially help preprocessing difficult OCR images (rotating image, identifying text boundaries, fix lighting, perform super-resolution, etc) to increase OCR accuracy

What I'd like to do is identify the vendor purely based on the image, not using any OCR. So I'd have folders for vendor1, vendor2, vendor3, etc. with the first page of dozens of invoices for each to train the CNN. Just not sure if an image of a document has enough nuances for the CNN to distinguish, especially since these are not different document types, they're all invoices, just different vendors.

It's theoretically possible for CNN to do well, but I think in practice with commodity hardware and the CNN architecture we have today, it will perform poorly. Would be better to use OCR, then use fuzzy keyword matching to see if the receipt mentions a vendor names. You'd only need to build a database of vendor names with this method, which is much easier than designing + training a neural network model.


text:
Why does my neural network to predict $x$ given $\sin(x)$ not generalize?
I made a simple feedforward neural network (FFNN) to predict $x$ from $\sin(x)$. It failed. Does it mean the model has overfitted? Why doesn't it work?

set.seed(1234567890)
Var3 <- runif(500, 0, 20)
mydata3 <- data.frame(Sin=sin(Var3),Var=Var3)
set.seed(1234567890)
winit <- runif(5500, -1, 1)
#hidUnit <- c(9,1)
set.seed(1234567890)
nn3 <-neuralnet(formula = Var~Sin,data = mydata3,
                hidden =c(4,2,1),startweights =winit,
              learningrate = 0.01,act.fct = "tanh")

plot(mydata3, cex=2,main='Predicting x from Sin(x)',
     pch = 21,bg="darkgrey",
     ylab="X",xlab="Sin(X)")
points(mydata3[,1],predict(nn3,mydata3), col="darkred",
       cex=1,pch=21,bg="red")

legend("bottomleft", legend=c("true","predicted"), pch=c(21,21),
       col = c("darkgrey","red"),cex = 0.65,bty = "n")

Im not familiar with r code. you are just feeding the x value into the network and adding tanh function to the hidden layers in the network am i right ? Christopher Bishop's machine learning and pattern recognition textbook has an example of sin(x) function I believe. However, he tries to fit sin(x) with a polynomial function.

it's the other way around! feeding the network sin x and try to predict x.

Any loss function metrics that you can report ? Eg. training loss

The labels are not unique for the input domain [0,20]. Think about sin(x)=0, x could be 0, pi, 2*pi, 3*pi, ..., n*pi, all are correct from a mathematical point of view, but this is not reflected in your MSE loss. At this point your NN has to guess the correct label from your input data. Predicting the mean of your input data is the safest bet for the network.
In essence you're trying to build a arcsin function with a NN. If only consider x values in [-0.5*pi,0.5*pi], the labels are unique and your network should work.


text:
Has there been an instance of an AI agent breaking out of its sandbox?
There have been instances of agents using edge cases like bugs in physics engines, repetitive behavior in games or word repetition in text prediction to cheat their reward function. However, these agents are arguably still contained, as while they explore the extremes of the state space of the simulation they don't expand their action space beyond what is possible in the simulation.
The Pokémon Yellow Total Control Hack shows that in some systems, it is possible to gain full control of a computer by exploiting bugs in the hardware or software (here: memory corruption), enabling the agent to even completely reprogram the system 'from within', just using the normal inputs.
Do you know of similar extreme examples where an AI agent went far beyond what was intended with the simulation environment and expanded its action space?

This kind of scenario is still solidly in the realms of science fiction. The current crop of optimising and statistical pattern AIs are not "minds" in any common sense understanding of the word.

"There have been instances of agents using edge cases like...", can you please provide a reference that supports all your examples?

@nbro: The first link is to a short video which does demonstrate a game physics exploit being used to gain reward for a task. I think someone has collated a few similar examples on a page somewhere, I'll see if I can find it

@nbro: Here's a good list, and it might be the foundation of an answer: https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml (but as per my comment, the idea of an agent somehow "evolving" or transcending its state/action space to attack the hardware or redefine itself is still far out of reach).

@NeilSlater I had come across that list in the past and I had completely forgotten about it when I wrote my comment above. Thanks.

This is actually a very interesting question, but it might be a good idea to clarify what you mean by this "agent went far beyond what was intended with the simulation environment". In my view, the question is a bit vague, and the examples that Neil provide could actually be what you're looking for or not. So, if they are not, maybe you can edit your post to explain why and refer to them.


text:
What are the general inequalities needed for the logic gate perceptrons?
I'm trying to understand how the logic gates (e.g. AND, OR, NOT, NAND) can be built into single-layer perceptrons.
I understand specific examples of weights and thresholds for the gates, but I'm stuck on understanding how to translate these to general inequalities for these gates.
Is my reasoning correct in the table below, or are there cases where these general inequalities do not make sense for this problem? Am I missing other logic gates that can be done in a similar fashion? (e.g., I know XOR cannot)?
In the table below, a perceptron has two input nodes, and one output node. W1 and W2 are the weights (real values) on those input nodes. T is the threshold, above which, the perceptron will fire. I have come up with example values that would work for each logic gate (e.g., for the AND gate, a perceptron with two input weights, W1 = 1 and W2 = 1, and a threshold = 2, will fire, and I'm trying to understand more generally, what is the equation needed for each gate).

Gate
Example W1, W2
Threshold
General inequalities

AND
1,1
2
W1 + W2 >= t, where W1, W2 > 0

OR
1,1
1
W1 > t or W2 > t

NOT
-1
-0.49
W1 > 2(t)

NAND
-2,-2
3
W1 + W2 <= t

This post is not fully clear to me. $W_1$ and $W_2$ are the weights, right? If that's the case, are you assuming binary, integers, or real weights? I suppose that $t$ is the threshold here. The numbers in the second column would be the values of those weights, I suppose. It may be useful to provide more context regarding where you've been studying about perceptrons and how they can learn these logical operations, so that to understand why you're trying to understand what you're trying to understand.

thanks for this. I added more information to the question, I can't add the exact notes I was reading from because of copyright, but the idea is that there is a note that says 'we can use perceptrons to model different logic gates, such as an AND gate. An example of an AND gate set of weights and threshold that would fire for a perceptron is W=1, W2=1 and threshold=2'. But I want to understand further than that, what is the equations that would tell you to let you quickly assess, in any situation, whether the perceptron will fire for the different weights described above.


text:
Are there any disadvantages to using a variable population size in genetic algorithms?
When implementing a genetic algorithm, I understand the basic idea is to have an initial population of a certain size. Then, we pick two individuals from a population, construct two new individuals (using mutation and crossover), repeat this process X number of times and the replace the old population with the new population, based on selecting the fittest.
In this method, the population size remains fixed. In reality in evolution, populations undergo fluctuations in population sizes (e.g. population bottlenecks, and new speciations).
I understand the disadvantages of variable populations sizes from a biological view are, for example, a bottleneck will reduce the population to minimal levels, so not much evolution will occur. Are there disadvantages to using variable population sizes in genetic algorithms, from a programming perspective? I was thinking the numbers per population could follow a distribution of some sort so they don't just randomly fluctuate erratically, but maybe this does not make sense to do.

Population size is a tricky issue even in pure biological models. Biological population sizes obviously vary. The two great protagonists of the argument were Ronald Fisher and Sewell Wright, with argument being between Fisher favouring few large populations against Wright’s many small interconnected populations. There is evidence that evolution occurs more rapidly in Wright’s model but the evidence is inconclusive. The theory concentrates on the probability that a mutation will occur and then become dominant in a population. In a small population a beneficial mutation is more likely to be selected for reproduction, but premature convergence is a serious danger.  While in a larger population a mutant is less likely to be removed from the population during reproduction. I would strongly recommend a read of Games of life by Karl Sigmund.

When you say " There is evidence that evolution occurs more rapidly in Wright’s model but the evidence is inconclusive", maybe you should cite some reliable research paper that provides this evidence. Not sure if this answer is based on the book that you're suggesting.

The result is widely reported, with a suitable reference being The Island Model Genetic Algorithm: On Separability, Population Size and Convergence https://www.researchgate.net/publication/2244494_The_Island_Model_Genetic_Algorithm_On_Separability_Population_Size_and_Convergence

In any case, how does this answers the question "Are there any disadvantages to using a variable population size in genetic algorithms?" Here you're just talking about the trade-off between big and small population sizes and not about variable population sizes. The OP accepted this answer, but it doesn't really address the actual question, to be honest. So, I suggest that you do a little bit of research (in case you don't know the answer to the actual question), then provide the answer to the actual question too.

I can send you my PhD thesis if you want? To answer the question in full would take more time than I have at the moment. But if you want more info on varying population sizes I’d suggest you read Yin and Germans paper A Fast Genetic Algorithm with Sharing Scheme Using Cluster Analysis Methods in Multimodal Function Optimization, which applies cluster analysis to each population to dynamically form breeding populations of similar individuals.

You should edit this answer to include the info that you added in these comments and try to address more the original question by at least linking to the resource that talks about varying population sizes.

using different population sizes at different stages of the optimization process can be beneficial. With large population sizes you can effectively explore the landscape to find proper areas. Large population sizes helps in finding global optima or high fitness local optima. However, using large populations require more fitness evaluations and waste of computational resources. With a small population you can effectively exploit a previously find appropriate area and get high accuracy solutions. For this reason, some works suggest to use a large population at the start of the algorithm and gradually, decrease its size, like [Improving the search performance of SHADE using linear population size reduction]. Also, some dynamic optimization methods, use dynamic population sizes. For instance, they create sub-populations when it is necessary to discover more optima or to cover the landscape, they decrease their sub-populations when they detect a change or new appropriate uncovered area in the landscape.

"some works suggest to use a large population at the start of the algorithm and gradually, decrease its size", can you please cite those works (and provide a link)? Also, please, provide references for the other parts of this answer too, such as for the part "Also, some dynamic optimization methods, use dynamic population sizes" and for the initial claim  "using different population sizes at different stages of the optimization process can be beneficial.".


text:
Is this LSTM layer learning anything?
I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:

I don't understand. Is this layer learning anything? Or no?
Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got:

So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.

What may be more informative in terms of whether it is learning or not, is to track gradients.
Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.
You are developing with tf.keras, in which case you can add to your tensorboard callback:
tf.keras.callbacks.TensorBoard(write_grads=True)
Additional experiments may include:

trying shorter length sequences and compare grad flow
try replacing tanh with alternative activation functions in the LSTM layers of your model tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu()) (see tanh saddle points problem)

(graph borrowed from d2l)

It seems that tf.keras.callbacks.TensorBoard(write_grads=True) is deprecated in tensorflow 2.4 but I found this article on how to do it manually. Also I'll try the leaky relu too, thanks.

Oh interesting article @SepehrGolestanian, wasn't aware. I have been using tbX for pytorch and it seems to still be there. Feel free to upvote/choose as solution if answer was useful.

I’ve tried leaky relu too but nothing changed

You mean nothing changed in the gradients or the weigh/bias distributions?

I don't know anyway to check gradients in tensorflow but the weight distributions didn't change.

It is difficult to know if you cannot. The weight distribution of a neural network that has been trained with regularisation is expected to be distributed normally. See here relevant paper. http://proceedings.mlr.press/v37/blundell15.pdf


text:
Is my pseudocode titled "Monte Carlo Exploring Starts (with model)" correct?
Reinforcement Learning: An Introduction second edition, Richard S. Sutton and Andrew G. Barto:

We made two unlikely assumptions above in order to easily obtain this guarantee of
convergence for the Monte Carlo method. ... For now we focus on the assumption that policy evaluation operates on an infinite
number of episodes. This assumption is relatively easy to remove. In fact, the same issue
arises even in classical DP methods such as iterative policy evaluation, which also converge
only asymptotically to the true value function.

There is a second approach to avoiding the infinite number of episodes nominally
required for policy evaluation, in which we give up trying to complete policy evaluation
before returning to policy improvement. On each evaluation step we move the value
function toward q⇡k, but we do not expect to actually get close except over many steps.
We used this idea when we first introduced the idea of GPI in Section 4.6. One extreme
form of the idea is value iteration, in which only one iteration of iterative policy evaluation
is performed between each step of policy improvement. The in-place version of value
iteration is even more extreme; there we alternate between improvement and evaluation
steps for single states.

The original pseudocode:

Monte Carlo ES (Exploring Starts), for estimating $\pi \approx \pi_{*}$
Initialize:
$\quad$ $\pi(s) \in \mathcal{A}(s)$ (arbitrarily), for all $s \in \mathcal{S}$
$\quad$ $Q(s, a) \in \mathbb{R}$ (arbitrarily), for all $s \in \mathcal{S}, a \in \mathcal{A}(s)$
$\quad$ $Returns(s, a) \leftarrow$ empty list, for all $s \in \mathcal{S}, a \in \mathcal{A}(s)$
Loop forever (for each episode):
$\quad$ Choose $S_{0} \in \mathcal{S}, A_{0} \in \mathcal{A}\left(S_{0}\right)$ randomly such that all pairs have probability $\geq 0$
$\quad$ Generate an episode from $S_{0}, A_{0},$ following $\pi: S_{0}, A_{0}, R_{1}, \ldots, S_{T-1}, A_{T-1}, R_{T}$
$\quad$ $G \leftarrow 0$
$\quad$ Loop for each step of episode, $t=T-1, T-2, \ldots, 0$
$\quad\quad$ $G \leftarrow \gamma G+R_{t+1}$
$\quad\quad$ Unless the pair $S_{t}, A_{t}$ appears in $S_{0}, A_{0}, S_{1}, A_{1} \ldots, S_{t-1}, A_{t-1}:$
$\quad\quad\quad$ Append $G$ to $Returns\left(S_{t}, A_{t}\right)$
$\quad\quad\quad$ $Q\left(S_{t}, A_{t}\right) \leftarrow \text{average}\left(Returns\left(S_{t}, A_{t}\right)\right)$
$\quad\quad\quad$ $\pi\left(S_{t}\right) \leftarrow \arg \max _{a} Q\left(S_{t}, a\right)$

I want to make the same algorithm but with a model. The book states:

With a model, state values alone are sufficient to determine a policy; one simply looks ahead one step and chooses whichever action leads to the best combination of reward and next state, as we did in the chapter on DP.

So based on the 1st quote I must use "stars exploration" and "one evaluation — one improvement" ideas (as well as in model-free version) to make the algorithm converge.
My version of the pseudocode:

Monte Carlo ES (Exploring Starts), for estimating $\pi \approx \pi_{*}$ (with model)
Initialize:
$\quad$ $\pi(s) \in \mathcal{A}(s)$ (arbitrarily), for all $s \in \mathcal{S}$
$\quad$ $V(s) \in \mathbb{R}$ (arbitrarily), for all $s \in \mathcal{S}$
$\quad$ $Returns(s) \leftarrow$ empty list, for all $s \in \mathcal{S}$
Loop forever (for each episode):
$\quad$ Choose $S_{0} \in \mathcal{S}, A_{0} \in \mathcal{A}\left(S_{0}\right)$ randomly such that all pairs have probability $\geq 0$
$\quad$ Generate an episode from $S_{0}, A_{0},$ following $\pi: S_{0}, A_{0}, R_{1}, \ldots, S_{T-1}, A_{T-1}, R_{T}$
$\quad$ $G \leftarrow 0$
$\quad$ Loop for each step of episode, $t=T-1, T-2, \ldots, 1$:
$\quad\quad$ $G \leftarrow \gamma G+R_{t+1}$
$\quad\quad$ Unless $S_{t}$ appears in $S_{0}, S_{1}, \ldots, S_{t-1}:$
$\quad\quad\quad$ Append $G$ to $Returns \left(S_{t}\right)$
$\quad\quad\quad$ $V\left(S_{t}\right)\leftarrow\text{average}\left(Returns\left(S_{t}\right)\right)$
$\quad\quad\quad$ $\pi\left(S_{t-1}\right) \leftarrow \operatorname{argmax}_{a} \sum_{s^{\prime}, r} p\left(s^{\prime}, r \mid S_{t-1}, a\right)\left[\gamma V\left(s^{\prime}\right)+r\right]$

— Here I update the policy in $S_{t-1}$ because the step before we update $V(S_{t})$ and changes to $V(S_{t})$ don't affect $\pi (S_{t})$, but affect $ \pi (S_{t-1})$, as $S_{t}$ is in $S'$ for $S_{t-1}$.
Pseudocode as images:

Hello. Welcome to AI SE! Take a look at our on-topic page to know more about our site and scope. Meanwhile, you should probably fix the language mistakes (e.g. in the title). I would also like to note that you can use latex/mathjax on this site, so I suggest that you edit your post to use it and fix these mistakes.

I've changed the title a bit, but not sure I have time to rewrite the images of pseudocode in latex. It certainly will not make the question more clear.

No worries if pressed for time, but part of the utility of rendering the formulae in latex/mathjax is that it facilitates getting an answer.  (Person answering can cut and paste the original formulas to save time.  Not doing that places the burden on them, which can be a disincentive for answering.)

I was not asking you to convert the pseudocode to latex/mathjax, but just the formulas at the end of your post, but someone already proposed an edit that fixes that. You don't need to convert the screenshots of the pseudocode to latex, so you can revert your last edit. If you really want to do that, maybe have a look at this: https://ai.meta.stackexchange.com/a/1680/2444.

authors: Дмитрий Винник


text:
How many tasks are needed for meta-learning?
This is an empirical question, essentially how many tasks do you need data for, to make a useful meta learning model (e.g. using MAML)? I'm looking for ranges based on personal experience or if anyone has done research on the topic and you know of references for the estimates that would be helpful as well.
For context I'm trying to work with about 5-7 tasks. I saw a person implement meta-learning with about this many in the paper Multi-MAML. But I've since seen example code in the learn2learn library which uses thousands of tasks...
P.S. I'm not sure if different parameterizations of a single task definition are still 'one task' (e.g. y=a*cos(x), where 'a' varies). Could that account for the discrepancy?


text:
How to use Actor-Critic RL with a categorical, state-dependent action space?
I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each represented by a unique text string. The strings don't necessarily follow any natural language rules, though they should contain information about the action.
How does someone use the actor-critic algorithm here?
It is obvious for Q-learning. You just use the $Q(s,a)$ function on each state-action pair. But the actor-network must choose an action directly. And implementations I've seen do this use a categorical output (of all possible actions, which is intractable), or a multi-variate normal distribution of some continuous representation of actions.
I'm not really sure how this applies in this context. And I've considered building a 'fake' actor network out of a Q-network module that just applies the Q-function to each possible action & takes the best action, but I'm not sure if there are some theoretical problems with this which is why the authors of actor-critic avoided the use of a Q-function?


text:
What determines when Dropout, BatchNorm & other Regularization will be effective?
I just had a very strange experience where I was training an 8 layer deep & pretty wide (max: 512 neurons) neural network for a regression task. I had assumed since it was big enough that it would benefit from some combination of dropout, batch norm, and/or skip layers. And I expected all 3 would be best. I turns out however that each one of them made the model perform much worse, together or nearly in any combination (even at 10% dropout rate).
This made me wonder when you can expect these methods to help the model vs hurt? I thought they were pretty universal before this (if the model is big enough)... It might be related to this data's strange distribution, I think it is exponential (except it can dip negative sometimes). But scaled to stay within 0-1.
P.S. Similar experience with log transforming the data!
I've attach a histogram of the predicted variable for reference in my example (log scale using only positive values, which are the majority).

From prior experience, there really isn't a robust method of knowing when such methods will be effective or not. In my opinion, methods like dropout or batch norm aren't "standard tricks" like many believe. They should be applied based on strictly experimental results rather than vague theories behind them.
Even the theory behind them aren't exactly robust. For example, the reason behind batch norm's success is still controversial (https://arxiv.org/abs/2102.06171).

Also, I have seen many cases in regression tasks where batch norm and dropout are hurtful. See (https://towardsdatascience.com/pitfalls-with-dropout-and-batchnorm-in-regression-problems-39e02ce08e4d) for more detail.

These are hyper-parameters of the model, and you need to use popular methods like Bayesian Optimization to find the best value for them.


text:
How to deal with Q-learning having low variance in predicted Q-values?
I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning.
I've noticed that, given a particular state $s$, the neural network will predict nearly identical $q(s, a_i)$ for all actions $a_1, \dots, a_n$. As the neural network gets trained, this situation gets worse.
I think it is predicting the mean Q value of the state. Perhaps, the small amount of data that comprises the action is being drowned out by state input?
I've considered using softmax to predict the best action, but it seems like we have nearly an unlimited number of possible actions to take, and I don't want to hard-code them.
EDIT, More Detail: The state is represented by all the text detected by an OCR of a GUI program + an embedding of screenshot image of the current GUI state. While in comparison the action is just 1-2 words (the text or tooltip of the GUI element we are considering clicking on), and I'm concerned that this imbalance of representation of state vs action in the input could be causing the network to accidentally (nearly) ignore the action input.

Hi. Can you please elaborate on 'I've noticed that given a particular state it will give nearly constant estimates of action Q-values across any number of actions' part and also what do you mean by the Q-values getting worse?

I mean given some state S, the network will predict nearly identical Q values for all possible actions. And that the problem gets worse as training progresses.

I'm facing similar issue while trying to implement linear FA with DQN. It is because my state-action representation as vectors is bad

@MonkeyDLuffy : how so?

I started with very bad representation of states as vectors and play around with it. While doing so I observed the Q values are not changing for all state, action pairs. So I asked for help here

Given the comments above, maybe you can edit your post clarify how your state and actions are represented and what they are in your context. You should also explain what you mean by "the state contains a lot of data" while the action doesn't. Moreover, what do you mean by "dual Q-learning"? Do you mean "Double Q-learning"?

Ok I updated it


text:
Why is it difficult to train large RL networks?
First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true?
Intuitively it seems very strange & to be perhaps a fundamental weakness of RL that it
very ineffective at learning large networks. Supervised learning usually has no problem learning features from unstructured data via potentially quite large networks (e.g. upwards of 8+ layers). But in the case of RL it is a very big problem it seems (that is learning the extra embedding layers in the front of the network 'head').
P.S. in case you don't believe me here is a plot of hyper-parameter search data I collected on Acrobot-v1, CartPole, MountainCar-v0, and Pendulum-v0.
Also it is not hard to find evidence for this online... Just look at stable-baselines3 it has default num_layers=2...

I know that it might be true, but who said that "training large networks in RL is usually (more) unstable"? It might be a good idea to quote someone or something that claimed this, or maybe tell us that you've observed this empirically.

Sometimes the initial layers of the model will approximate the environment better. So adding layers means adding more noise and hence the learning becomes unstable. Every neural network has certain optimal number of layers and depth depending upon the use case. Too many layers in a neural network leads to bad learning and it is not limited only to RL. One of the areas I see this more frequently is in image classification where in some cases, few layers perform better than large number of layers

@MonkeyDLuffy I appreciate your answer but this is definitely something special about RL. The 'standard' number of layers is 2! Compared to 8 for even a pretty standard feedforward network, something along these lines is common in computer vision/NLP too.
In RL that leads to learning collapse.


text:
Why do skip layer connections require the same layer sizes?
I know how skip connections work: you add the activations of the previous layer to the activations of a successive layer to stabilize information/gradient flow.
My question is, why doesn't it just get implemented in the seemingly more sensible way of concatenating some previous layer's activations onto a later layer's activations?
Most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.
What is the reasoning behind the choice to do this rather than simply using concatenation?

It's a good question, but why does addition have a "questionable benefit"? If you can explain why you think that, it may help with an answer. As it stands, it is not useful in the question, because it seems to just be an opinion, so cannot be answered.

Because most regularization methods are implemented somewhat transparently (to avoid possible negative consequences, e.g. BatchNorm having learnable parameters to disable it). While this method instead interferes with regular functioning of the network rather than simply making itself available in case it is useful.

Thanks for the response. It would be helpful to [edit] that into the question.

Is your question why adding the output of layers vs concatenating or why the same shape? Either way, the spatial dimensions (in CNN) must have the same shape. As far as I know, skip connections utilize concatenation, whereas ResNet addition (long vs short skip connections)

It seems to me that there are 2 distinct questions in this post (the one in the title and the other in the body of the post). Please, edit your post to leave just one question.

@nbro well one property is a consequence of the other. My point is that I don't want someone to answer explaining that the reason is because of addition. I want someone to explain the root cause which requires explaining the addition which as far as I know is an arbitrary design choice.

One can concatenate with the previous layer outputs as well, and this approach in pursued in DenseNets. A nice illustration, that compares difference between ResNets and DenseNets is presented below:

As pointed in the other answer it will lead to an increase of computation cost, with the same number of channels (given all other properties of architecture are the same).
Suppose, you had ResNet with the fixed channel size $N$. Standard convolution has computational and storage cost proportional to the product of input and output channels or $O(N^2)$ in the present case.
If you concatenated features from the previous layer, after each layer number of channels would be doubled. Therefore, the computational cost would grow $4$ times for each new layer, and the total cost grows exponentially with depth in this approach.
However, you can make each of the convolution to shrink the number of channels $2$ times and concatenate only half of the channels from the previous layer. In this way, total computational cost and storage cost is the same in every layer.

I don't see how it would grow exponentially... You're correct that it could grow if you used the output the concat operation directly in the next skip layer connection. But the idea is that you would instead use the output of the next layer as skip connection input. That being said I see that this growing structure is what Dense net is trying to implement, that is interesting...

But even if you allowed it to grow the number of parameters would grow linearly, and then plugging into O(N^2) would give you quadratic growth right?

@profPlum it depends on how you craft the layers. Typically, one would double the number of feature maps each next layer, which leads to exponential growth.

There is no obvious reason why concatenating layers will "stabilize" the training of the network. In fact, you are adding more information, which might improve accuracy, but perhaps at certain computational disadvantages.
ResNets prevent the vanishing gradient problem by "passing information" from previous layers onto the next. Page 6 of these slides shows calculations that show this to be the case in a specific degenerate edge cases; but as for how it consistently provides such an advantage on more general basis is from what I understand still unclear.

tl;dr: read page 6 of those slides

Of course, it does create more parameters to train, but that seems like a small sacrifice to me.

It is not a "small sacrifice". For the very deep networks that skip connections are applied to, to get the same benefits when concatenating, you would end up witha significant multiplier on the number of parameters.
To get the same passthrough effect on gradient signals (and allow later layers to learn modifications to the identity function), each layer's output would need to be copied to all the following layers. This scales poorly.
Let's take an example of a 10-layer fully-connected network, with 100 neurons per layer in the hidden layers where we want to apply skip connections. In the simple version of this network (ignoring bias to keep the maths simpler), there are 100x100=10,000 parameters for each added layer, making 90,000 parameters overall. If you use addition-based skip connections, the total number of parameters remains the same, at 90,000. If you use concatenation, the layers connect as 100x100, (100+100)x100, (100+100+100)x100 etc, so you end up with 450,000 parameters, five times as many. This is not a "small sacrifice", this is a scaling problem.
The technique of concatenating layers into later parts of the network is known and used. As is adding "early output" or head layers to generate gradient signals at different points in the network. These are valid approaches, and can help with vanishing gradient problems in a similar way. However, the additive skip connections in a residual network scale much better into very deep networks.
Concatenated copies of layers, or additional network heads with the same target functions and loss, are still used, but more sparingly.

This avoids the strange choice of addition, which has a questionable benefit

It may avoid a specific addition mechanism that you seem concerned about. However there are still very similar additions occurring when the concatenated layer feeds forward to the next layer. You could set those weights in a specific way (a copy of the weights of the new layer that it is concatenating with for just two combined layers) when the layer sizes are the same, and the concatenation and addition approaches would very similar.


text:
Best way to measure regression accuracy?
I'm asking because classification problems have very concrete metrics like accuracy that are totally transparent to understand.
Whereas regression models seem to have a very large number of possible evaluation strategies and to me at least it is not clear which (if any) of them is as reliable/interpretable as accuracy is in classification problems.
Possible Candidates:

Regular loss (e.g MAE): MAE is potentially quite interpretable, but again interpretation depends upon distribution statistics which vary across regression problems.
MAPE/Relative Loss: This is interesting and is potentially decently similar to accuracy. Yet it has obvious draw backs, like the true value being extremely small causing explosion of loss values & there being no incorporation of overall distribution statistics for the output values.
Chi-squared test: I like the idea of this but I have not seen it used at all for NN regression for some reason. I'm not sure why and I'm curious if people think it would be a good idea to use it for that.
(adjusted) R^2 coefficient: Another statistic that seems great in theory, but again I see almost never being used for NNs and I'm not sure why. This has the great advantage of being a 'bounded'/'normalized' metric like accuracy and in theory is should be just as interpretable. Why is it not used for NNs?

I've gotten no answer on this, but after some reflection I've come to accept R^2 as regression's analogue to accuracy. I have no idea why it is not used by more people in deep learning, but I recently started using it and it's exactly what I was hoping for! It scales from 0 to 1 and tells you in no uncertain terms how useful the model is and when it becomes unrealistic to improve the model further (e.g. at R^2=0.99). Additionally, R^2=0 when the model starts guessing only the mean of the data, which means it immediately reveals an important and recurring issue in regression which can otherwise go undetected for a while.


text:
How do modern Transformer AI researchers compete with LLMs like ChatGPT?
I'm considering getting into NLP research (that is AI research into Transformers specifically) but my friend made this point which got me concerned, "he said NLP research is dead b/c of ChatGPT". We all know ChatGPT is enormous and highly versatile. Is it realistic for NLP researchers to make meaningful contributions without working at: google, amazon, ivy leagues, facebook, OpenAI/Microsoft etc...
P.S. For example recently saw the Vega2 paper 2nd place on SuperGlue leader board used 320 A100s for > 30 days. Looking at rental prices online that should cost about $300,000!

While LLMs like ChatGPT have indeed reshaped NLP, they have not killed NLP research. Instead, the field has shifted in terms of focus and the right niche opportunities available for researchers, even those outside the biggest organizations.
LLMs like ChatGPT are powerful but often brittle when faced with out-of-distribution (OOD) inputs or nuanced domain-specific higher-level rule-based traditional symbolic AI and NLP tasks. For instance, many languages and dialects are underrepresented or near-extinct and NLP research for these is critical and often does not require massive resources. Real world NLP applications often require domain-specific fine-tuning. For example, legal, medical, and financial NLP involve challenges that general-purpose models like ChatGPT cannot solve out of the box.
Multi-Modal NLP to combine text with other modalities (e.g., vision, audio) is another evolving domain where new contributions are welcome. Similarly combining NLP with fields like social sciences, mathematics, cognitive science, or healthcare opens new avenues that are less reliant on current ChatGPT like LLMs.
Finally addressing imminent and difficult NLP issues related to explainability and interpretability, fairness, bias, societal risks and ethical concerns is also a burgeoning area of study.

Thanks but I should've mentioned I'm explicitly interested in transformers.
Like how are modern transformer AI researchers working around ChatGPT?

It is difficult. At my previous company I (NLP Engineer) was replaced by an LLM, and looking for NLP jobs, everything was geared towards experience with LLMs. "Traditional" NLP methods are not really en vogue enough at present, it seems.
However, my own view is that this is a temporary bubble. There is a lot of hype, and over-promises about the capabilities of LLMs. At some point, I expect this to calm down, and there to be a falling back on tried and tested methods which aren't as "sexy", but do actually work.
I do realise this is perhaps not a very popular view, but LLMs are basically just smoke and mirrors, albeit very sophisticated.
So, for the near future it might be difficult to find a job in the NLP area with more traditional skills, but my prediction would be that this will change in the longer term; and even now there are companies who don't want to use LLMs for various reasons (compliance, data protection, etc). Academia might be a better bet for now than industry, as it seems less susceptible to the hype.

I don't believe that we will be falling back to "traditional NLP" (e.g. Bag of words) if that's what you mean, Transformers are here to stay. But perhaps with improvements in efficiency & pre-training will make things more accessible.

Bag of words is not traditional NLP. I was thinking of syntactic parsing etc.


text:
Achieving low train error for exponential response?
I'm trying to fit an ML model with perfect information to predict an exponentially distributed response without getting exponentially distributed error...
Also, this situation is special for a few of reasons:

I've tried fitting $M(x)\approx log(y)$, unfortunately it actually degraded performance (in the original domain).
I'm positive that the response variable is 100% determined by the inputs, since we are trying to mimic physical simulation code which takes the same inputs.
Despite point 2, the best error I can get is still on the order of MAE$\approx$ 1.2e+9 (the response variable itself has mean=1.334e+09 & SD=1.35676394551e+11).

Is it possible to somehow get low train error (i.e. in absolute, rather than relative terms) given perfect information?


text:
Is ReduceLROnPlateau() Schedule too aggressive by default in pytorch?
Hi I'm asking about the learning rate schedule ReduceLROnPlateau(). I've noticed that it decreases the learning rate by a factor of 10 each time and by default the patience is 10. Given phenomena like double descent its default parameters are very concerning.
Not to mention that in practice I find it to be very common that learning might stall for 10 epochs (even if not truly stuck). And for example this LR schedule (with patience=6) gives me learning_rate=2e-9 after just 100 epochs!! Presumably it would further decrease it if not for the epsilon parameter, which isn't intuitive as you would think it should decrease gradually...
Are they basing these defaults off of some paper? Can anyone speak to their experience using this schedule & the performance of its default settings? And/or share relevant papers.

I'll start by saying that optimization is as much art as it is science.
ReduceLROnPlateau is my favourite scheduler and I've found it performs quite well out of the box. Of course it's good to play with it to achieve optimal results but for what you experience there might be other reasons.
Depending on the size of your data, 100 epochs might be more than enough so this might not be a problem to begin with.
In case this is indeed a problem, since you don't give much details of your training, I can only provide some generic suggestions.
Check if your model learns without a scheduler. Find some good hyperparameters for:

The starting learning rate. It might be too small to begin with, or even too big (this could mess the network initialization).

The batch size cause it could greatly affect training dynamics.

Any other parameter you think it's important.

After you have these, then add the scheduler. If you still have issues try one of the following:

Check if the metric that you're monitoring is the right one, i.e. a loss on a validation set and that it's values make sense.

Check the iterations per epoch. A very small number of iterations per epoch could give noisy validation metrics.

I also like the idea of it, but this is the 2nd time I've found it's defaults problematic because in practice I usually want to train my models for thousands of epochs.

It's hard to help more without knowning the specifics. Consider sharing a bit more information about your model, data and training.

Thanks I think I discovered the problem to be that epochs aren't a proper "unit" because the number of gradient steps that they represent can vary by orders of magnitude depending on the size of the dataset. And these defaults are probably designed for enormous image datasets with many gradient steps.

Good to know you solved it. Yeah these methods target the stochastic gradient descent regime. If you're doing only a small amount of iterations per epoch then measuring the validation error in every epoch can be too frequent, your model doesn't have time to become better (hence noisy validation metrics). It's probably not very efficient either.

Ya your comment about 100 epochs "being more than enough", was enough for me to figure out the rest. Thanks


text:
KL loss not changing in a Bayesian Neural Network?
I've been trying to train a Bayesian Neural Network and I noticed that the KL loss (which enforces the prior) isn't changing over time. And it occurred to me that while in standard Bayesian inference the prior acts like an artificial dataset (e.g. beta(N,N) adds 2N more observations). In Bayesian Neural networks like Bayese-by-Backprop, the prior is just a regularizing term in the loss function (the KL loss term).

Which as I understand it means the KL loss term might never increase (depending on its weight), and thus uncertainty might never decrease, which contradicts the standard Bayesian inference behavior where the prior is guaranteed to lose it's importance as you get more data... Further more if you play with loss term weights then there is a question as to what weights are valid & how to prevent collapse to a fully deterministic network...
Please tell me I'm wrong, and explain how (hopefully) this is fully comparable to regular Bayesian inference as well as why KL-loss might not change during training?

This looks like more underfitting than other issue.
But you are overlooking that Bayes by Backprop is an approximation to Bayesian principles, because a true Bayesian NN is not tractable, you are using an approximation, and it behaves approximately, so not all principles you mentioned actually apply.

You're right actually the loss plots at the time had a bug in it. So I've removed them to focus on the theory component.

@profPlum If this is the right answer, you could accept it.

Your observation is correct. In BNNs trained with methods like Bayesian by backprop (BBB), the KL loss serves as a regularizer that encourages the posterior distribution of weights to approximate the prior distribution, therefore it's possible for the KL loss to remain relatively constant during training, especially if the model has not yet learned to deviate significantly from the prior meaning it's actually learning well w.r.t the prior as expected or the weights are initialized close to the prior. In BNNs and ML in general the prior remains fixed throughout training to be qualified as a regularizer for reducing test error, while in standard Bayesian inference the prior is continuously updated based on newly observed data without generalizable learning or prediction consideration. Though bearing in mind such functional difference, in principle BNNs are designed with the same aim as Bayesian inference to decrease prior uncertainty with more training data.
BNNs training sometimes can readily lead to undesirable underfitting behavior, such as "posterior collapse" to a deterministic network caused by the complex interactions between the architecture of the neural network, the choice of prior, the amount of training data and the KL loss weight. For instance, your prior is put too much weight due to your confidence but the data is actually in total conflict with your prior, which is possible to fit your specific case. The loss of uncertainty propagation and update of learnable parameters defeats the purpose of employing Bayesian method which is crucial for robust decision-making.
Balancing KL loss weight hyperparameter is often an empirical process and it depends on factors like the model capacity and the amount of available data. The more effective capacity or lesser training data, the larger the hyperparameter in general. Also techniques such as the use of a fixed schedule for annealing the KL weight could mitigate the "posterior collapse".

Thanks. Part of the reason, I asked is because ordinarily you would expect that uncertainty would decrease with more data (or in this case more training) in a Bayesian setting. Is it still reasonable to expect that with BNNs?

In principle BNNs are designed with the same aim as Bayesian inference to decrease prior uncertainty with more training data, see my updated answer.


text:
Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset
I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they construct the discretized codebook latent space), so I went with a simple convolutional autoencoder with a kl-regularizer to map to an approximately gaussian latent space. I've run this model a few times and verified that it does reconstruct the original image inputs fairly well.
I run into issues when I try to use this model with my LDM implementation. I first made sure standard diffusion directly in image space works, which it does. I then tried latent diffusion with my trained autoencoder and I can't get the loss to drop below a certain threshold (~1.0), and the image outputs are pretty much gaussian noise still.
As stated in the paper, I sample latent vectors from the encoder part of the autoencoder and scale them with stats collected from the first batch of data like so:
                    batch = autoencoder.encode(batch).sample().detach()
                    # rescale the embeddings to be unit variance
                    if epoch == 0 and step == 0:
                        print("Calculating scale factor...")
                        std = batch.flatten().std()
                        scale_factor = 1. / std
                        cfg.scale_factor = scale_factor.item()
                    batch *= scale_factor

And then pretty much everything else (applying noise, calculating loss, etc) is the same as standard diffusion. Am I missing something, or is the latent space of my simple conv-autoencoder hard to learn for some reason? I would think that, since an autoencoder is just a deterministic mapping from image space to a lower dimensional one, that the LDM should be able to learn the latter just fine, or is there something inherent in the original paper's transformer architectures for the encoder / codebook latent space that is important to learning for the LDM?

Rather than writing "Question on Autoencoders Used in Latent Diffusion Models", can you please just put your specific question in the title?

Two things come to my mind: 1. LDM uses a quite deep convolutional auto encoder, this may not required for Fashion-MNIST, but have you tried changing the depth and see how your result changes? 2. LDM uses the attention mechanism to condition the diffusion. How are you currently conditioning your diffusion process?

I noticed the same issue as well. I was working in conditioned generation of spectrogram, in which it works well with spectrogram. But after pipe-in a pre-trained VAE into the latent space and training on the latent space, the generation output become blank and the loss doesn't visibly decrease.


text:
Stack of Planes as the Action Space Representation for AlphaZero (Chess)
I have a question regarding the action space of the policy network used in AlphaZero.
From the paper:

We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8 × 8 positions identifies the square from which to “pick up” a piece. The first 56 planes encode possible ‘queen moves’ for any piece: a number of squares [1..7] in which the piece will be moved, along one of eight relative compass directions {N,NE,E,SE,S,SW,W,NW}......The policy in Go is represented identically to AlphaGo Zero (29), using a flat distribution over 19 × 19 + 1 moves representing possible stone placements and the pass move. We also tried using a flat distribution over moves for chess and shogi; the final result was almost identical although training was slightly slower.

I don't understand why a stack of planes is used for the action space here. I'm also not entirely sure I understand how this representation is used. My guess is that for Chess, the 8x8 plane represents the board, and each square has a probability assigned to it of picking up a piece on that square (let's assume that all illegal moves haven't been masked yet, so all squares have probability mass on them). From there, we choose from possible 'Queen' type moves or 'Knight' type moves, which total to 73 different types of moves. Is this interpretation correct? How would one go from this representation to sampling a legal move (i.e. how is this used to parametrize a distribution I can actually sample moves from?)
During MCTS when expanding a leaf node, we get $p_a$, the probability of taking some action $a$ from the policy head, so I would also need to be able to go from this 'planes' representation to the probability of taking a specific action.
The paper also mentions trying out 'flat distributions', which I'm not entirely sure what this means either.

Can you please put your main specific question in the title?


text:
In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?
In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture
$$
\text { Attention }(Q, K, V)=\operatorname{softmax}\left(\frac{Q K^{T}}{\sqrt{d_{k}}}\right) V
$$
Everything is fine up to here.
Then they introduce the multi-head attention, which is described by the following equation.
$$
\begin{aligned}
\text { MultiHead }(Q, K, V) &=\text { Concat}\left(\text {head}_{1}, \ldots, \text {head}_{\mathrm{h}}\right) W^{O} \\
\text { where head}_{\mathrm{i}} &=\text {Attention}\left(Q W_{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right)
\end{aligned}
$$
Once the multi-head attention is motivated at the end of page 4, they state that for a single head (the $i$th head), the query $Q$ and key $K$ inputs are first linearly projected by $W_i^Q$ and $W_i^K$, then dot product is calculated, let's say $Q_i^p =  Q W_i^Q$ and $K_i^p = K W_i^K$.
Therefore, the dot product of the projected query and key becomes the following from simple linear algebra.
$$Q_i^p {K_i^p}^\intercal = Q W_i^Q {W_i^K}^T K^T =  Q W_i K^T,$$
where
$$W_i = W_i^Q {W_i^K}^T$$
Here, $W$ is the outer product of query projection by the key projection matrix. However, it is a matrix with shape $d_{model} \times d_{model}$. Why did the authors not define only a $W_i$ instead of $W_i^Q$ and $W_i^K$ pair which have $2 \times d_{model} \times d_{k}$ elements? In deep learning applications, I think it would be very inefficient.
Is there something that I am missing, like these 2 matrices $W_i^Q$ and $W_i^K$ should be separate because of this and that?

I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation.  The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture.  $W$ has dimension $d_{model} \times d_{model}$, which means that we are adding $d_{model}^2$ parameters to our architecture.  $W^Q$ and $W^K$ each have dimension $d_{model} \times d_k$, and $d_k=\frac{d_{model}}{h}$. If we use these two matrices, we only add $2\frac{d_{model}^2}{h}$ parameters to our architecture, even though their multiplication (with the transpose) allows us to have the correct dimensions for matrix multiplication with $Q$ and $K$.
We do use $h$ attention heads, which then brings our number of parameters back up, but the multiple heads let the model attend to different pieces of information in our data.

I understand now, actually the resulting $W$ matrix is rank deficient when defined by inner product of $W^Q$ and $W^K$. This makes parameters fewer :) Thanks.

I just read a paper (https://arxiv.org/pdf/1906.07510.pdf) in which the authors use multi-head attention from Vaswani et al, but have $W^Q$ and $W^K$ be of dimension $d_{model} \times d_{model}$.  I don't know if there is a reason for this, or if it's a typo.  I plan to email them, and if there is a reason for it, I'll post it here.

@BioBroo Might be a dumb question, but do these projections render the embedding (linear) layer redundant?

In practice, matrices $W^Q, W^K, W^V$ (each of size $d_{model}$ x $d_{model}$) are completely removed instead, and Transformer implementations just learn a single set of matrices $\{ W_i^{Q*}, W_i^{K*}, W_i^{V*} \}$ (each of size $d_{model}$ x $\frac{d_{model}}{h}$) for each head, where
$W_i^{Q*} = W^Q W_i^Q \\
W_i^{K*} = W^K W_i^K \\
W_i^{V*} = W^V W_i^V $
so that:
$Q_i(x) = x W_i^{Q*} = x W^Q W_i^{Q*} = Q W_i^Q\\
K_i(x) = x W_i^{K*} = x W^K W_i^{K*} = K W_i^K\\
V_i(x) = x W_i^{V*} = x W^V W_i^{V*} = V W_i^V\\
head_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\sqrt{d_k}} \right) V_i(x)$.
I can confirm this with the original Transformer implementation in Tensor2Tensor, and also the BERT code that uses the encoding part of the Transformer.

I missed the part where $d_k=d_{model} / h$ . Thank you.

I just stumbled upon this Question after not understanding the original paper. $$
\text { where head }{\mathrm{i}}=\text { Attention }\left(Q W{i}^{Q}, K W_{i}^{K}, V W_{i}^{V}\right)
$$I would really like to know if this redundant now and it can be$$ \text { head }{\mathrm{i}}=\text { Attention }\left(Q{i}, K_{i}, V_{i}\right)$$ where $$Q_{i}, K_{i}, V_{i} $$are simply generated from the input to the layer and the matrices $$
\left{W_{i}^{Q }, W_{i}^{K }, W_{i}^{V }\right}\in \mathbb{R} ^{D\times \dfrac{D}{h}}
$$ respectively


text:
How to deal with KerasRL DDPG algorithm getting stuck in a local optima?
I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the exact same DDPG to solve Pendulum-v0 and it works, but my environment is a more complex with a continuous space/action space.
How do you deal with local optima problem in reinforcement learning? is it just an exploitation issue?
More details:
My state space is not pixels, it is numerical, in fact it's a metro line simulator and the state space is the velocity, the position of each train on the line and the number of passengers at each station. I need to control the different trains so I am not trying to control only one train but all the operational trains and each one can have different actions such as speed or not, stay longer on the next station or not etc.
1/ I am using the same ANN architecture for the actor and critic: 3 FC layers with (512, 256, 256) hidden units.
2/ Adam optimizer for both the actor and critic with a small lr=1e-7 and clipnorm=1.
3/ nb_steps_warmup_critic=1000, nb_steps_warmup_actor=1000
4/ SequentialMemory(limit=1000000, window_length=1)
5/ The environement is a simulator of a metro line with a continuous state and action space

Hi @BAKYAC. Given that there's no logical bug in the algorithm/program, it could come down to fine-tuning the hyperparameters (learning rate, ann architecture, etc.). As I understand, finding a perfect combination of those can be quite challenging. There are some vague general guidelines, but still, the hyperparameters are usually specific for each particular environment. If you elaborate in your question on what kind of architecture you have, and what kind of problem you're trying to solve, you might get some input from other users (though this SE is somewhat in dormant state).

Thank you for your suggestions! I edited my question and yes I do think it's a hyperparameters fine-tuning problem but as the training takes too long I am not able to test out different parameters !

You could try using Google Colab – they let you use their GPU and TPU for free, though there's a limit. That really helped me to speed up model training and tuning

I have an NVIDIA Quadro T1000 GPU so I don't think I need google colab as they provide a tesla k80, plus the training crash after some time and mine can take days to finish..

Got it. Is your environment visual or just numeric? Also, what are the nb_steps_warmup_critic and nb_steps_warmup_actor parameters?

I see you're clipping the gradients, you could also look into adding regularization to your layers (that supposed to help the model to generalize and not to over-fit)

My environement is visual with a continuous state space, I think that DDPG is so simple and can't solve it I will try TRPO or can you suggest any other algorithm?

If you're working with pixels, you might need to add some convolutional layers, as they work better for visual feature recognition. Can't suggest any continuous algorithms unfortunately -- haven't worked with them before. This guide says that SAC and TD3 are current state-of-art algorithms: https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html

Oh no I am not working with pixels. I edited my question.. thanks anyway !

and good luck with your project :)

Thanks! will update this thread if I find something interesting..