text:
Is this the right approach to preprocessing data for artificial neural-networks?
I recently participated in a competitive "hackathon" with the problem being binary classification of overall satisfaction for travelers. The dataset mostly consisted of survey questions and answers: ex. "Seat_Comfort" : Extremely Unacceptable, Unacceptable, Needs improvement, acceptable, Good, extremely good
Other numerical info such as travel distance and sex/age was also included. Some Null data was throughout.
My question is what is the "right" way to preprocess this data?
The ordinal "survey" data I encoded as -3, -2, -1, 1, 2 ,3. Other categorical variables I hot-encoded. I standardized but DID NOT Normalize the data (trial and error did not show significant accuracy gains after normalization). For the most part I filled Nulls with averages, but did not really remove outliers first.
I spent so much time on trial and error in preprocessing the data and trying new approaches that I barely had time to hypertune the actual ANN parameters. I decided to come here and ask the "right" way to do it, if there is such a thing or standard.
For starters was ordinal encoding with negative numbers the way to do it? or should I have stuck to 1,2,3,4,5,6.
travel distance mean was like 2000, should that have been Normalized considering the relatively high values? Should I have removed outliers as a matter of course?
Some of the features were highly correlated with each other and I removed them (0.6 - 0.9) correlation. Is there a "best case" threshold for removing correlated features?
At one point I filled missing "SEX" values with UNK, as opposed to Male/Female in the data
If it matters at all, my ANN was a 3 hidden layer with dropouts and 1 TANH and 2 SELU activation. Sigmoid activation for the output obviously. I implemented a bagging, ensemble method to good results in the competition, but was still left with these burning questions.
To summarize:
Whats the right way to ordinal encode data?
Whats the threshold for dropping outliers?
Always Normalize? Always Standardize?
How to fill missing categorical data with similar probabilities (SEX for instance)
Is mean the most effective way to fill null numerical data?
Is there a threshold for dropping highly correlated features to improve performance?

You're asking too many questions in a single post. Only one main specific question per post.

Here is what I'd do:
Whats the right way to ordinal encode data?
If there's a natural ordering to the data, which is your case, then it's fine to map the data to a non-decreasing sequence of numbers. It's not very important where you start the sequence, because you should always normalize your input data.
If you don't normalize the data, then it's going to make learning much more difficult: the weights will have to have wildly different magnitudes to compensate for the difference in the magnitude of your inputs, and the variance of the gradient updates is also going to be very large, which is detrimental to learning.
What's the threshold for dropping outliers?
It's actually quite important to understand whether these outliers are erroneous data or not. If not, you want to keep them and might actually need to re-balance your dataset, or change your loss function to overweight them, so that your model correctly classifies them.
Imagine a situation where you want to know whether someone is a basketball player or not: your dataset will be mostly made of average heights, with a few extreme outliers which you actually want to classify as basketball players!
Always Normalize? Always Standardize?
As I explained above, always standardize
How to fill missing categorical data with similar probabilities (SEX for instance)
How to deal with missing data also depends on why the data is missing. In your case, I could imagine that whether or not the "sex" variable is reported by the user is highly correlated with the outcome you're trying to predict.
One could imagine that people not reporting their "sex" are more privacy conscious and therefore have higher standards when judging a service, which would entail a higher probability of being dissatisfied.
In that case, I would have a special value for missing data, and the neural network will use that information for its predictions.
Is mean the most effective way to fill null numerical data?
Same answer as above, you should consider whether or not the data is reported  is important to the classification task.
Is there a threshold for dropping highly correlated features to improve performance?
In general, if you suspect you have highly correlated features, you can do something like PCA first on the input data and keep only the K top components. However, K is now a hyper parameter of your algorithm, which you need to tune.
In general though, neural networks are quite good at discovering correlation in the input data, so I would not try to decorrelate the input features as a preprocessing step.
Hope this helps!

Thanks for the detailed reply! I felt the same way about the UNK vs Male/Female give the model a new variable to learn off of. I generally kept outliers if they were reasonable (a few age 80 folks for instance), but was really just guessing.


text:
Why does the SVM perform poorly on test data that has a different class distribution than the training data?
Do you know why the SVM performs poorly on test data that has a different class distribution than the training data? The training data has around 15 classes, and the additional testing data has around 6 classes (a subset of 15 classes). I found that the accuracy of new testing data is around 3% (all predicted labels belong to the same class).
How can I deal with this problem?

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Pattern recognition for live stream Time serie
I would like to submit you a problem with which I struggle.
Suppose I have this kind of record over time in a dataframe:

fig.1
If we zoom in a bit we see such shape:

fig.2
We see that the general pattern is a increase with a pick (very high or very thin sometime even flat) follow by an almost flat part with vibration then a decrease, the we go back to zero (almost) for a time (like in the middle of fig.1) or we start an other cycle
Some have very high peak, some are more flat, some have a more longuer part before to decrease.
I have 4 classes :

increase time - 1
working time  - 2
decrease time - 3
rest time (no activity) - 0

Now assume in my dataframe I have columns that tell to what class belong each point in time.

I would like to build a model that can recognize those 4 class when it see it on stream data . Imagine that our stream data is fig.1 and that we read N points (on a sliding windows) over time. What model could allow me to classify correctly each point or subpart data point in this window according to a certain pattern (hope I'm clear)
Regarding the fact that in reallity I could be in rest time for a very long time or in working time a very long time also.
It may also depend a lot of the sliding window, we for exemple see the beginning of the increase time on the first window then it end on the next window.
I first try to use LSTM or 1D-CNN, problem I have is that it tend to see this general pattern even when it's not present.
--- UPDATE : Chillston
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=777, shuffle=False)

# input N x T x D
X_train.shape, y_train.shape, X_test.shape, y_test.shape,

((17562, 1000, 1), (17562, 1000, 4), (4391, 1000, 1), (4391, 1000, 4))

I then pass it to 1d-CNN model, I have try many Architecture even resnet-cnn version, here's it's just more classical one.
def build_res1dcnn(n_classes):

    input_shape = (X_train.shape[1], 1)

    inputs = Input(shape = input_shape, name = 'input')

    # Stage 1
    x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu',
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)

    x = Conv1D(128, kernel_size= 5, strides = 2, padding = 'same', activation = 'relu',
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)

    x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu',
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)

    x = GlobalMaxPooling1D()(x)

    # Here I want T x n_class for a T X 1 input sequence
    outputs = []

    # https://stackoverflow.com/questions/51397484/appending-tensors-in-keras
    Ty = X_train.shape[1]
    for i in range(Ty):
        out = Dense(n_classes, activation = "softmax")(x)
        outputs.append(out)

    output = Concatenate()(outputs)
    output = Reshape([Ty, n_classes])(output)

    model = Model(inputs = inputs, outputs=output)

    model.compile(optimizer = Adam(learning_rate = 0.1),
                  loss = 'categorical_crossentropy',
                  metrics = ['categorical_accuracy'])

    #model.summary()

    return model

As your data is coming in as a stream, I would try a model that, given some number of historic samples (in your case you already have a sliding window $w_{t-n:t}$), classifies only the last sample of that window. That way the model can base the prediction on the $n$ previous samples. And from the looks a 1D-CNN should work fine with that data. Maybe you can provide a bit more info on what you have tried and what exactly didn't work?

@Chillston Hello, sure. I have update my post. First I start by cutting 1000 points in the fig.1 with a step of 500 points (before I was cutting sequence in fig.1 so to have many sequence like those on fig.2) - maybe what cause the model to see this general patttern everywhere). I also get the label associated with those point in "y" variable, then I one-hot encode "y". after that I send it to 1d-cnn that predict (N x T X n_class). I guess the real problem is the model architecture+hyperparam combine with windows size from train and test.

There are a number of possibilities you can consider:
simple classification
Following what you have suggested as your problem definition, you can train a model for those labels, given a hisotry of some window size (your hyper-parameter). In that case, common timeseries models can be utilized (LSTM, TCN, MCNN ). I prefer the MCNN method for it is a nice model especially when fixing a proper window-size is a challenge. It probably outperforms LSTM, so chances are you'll have better results. Moreover, LSTM must be improved with Attention mechanism, as you said the main challenge of your problem is long-term dependencies.
Kernel-methods classification
Since you already know that meaningful patterns exist in your data, you can base your classification on similarity measures. Note that this is still a classification model like the one above, but now instead of raw timepoints, you provide a similarity measure for signals (or shapelets so to speak) such as Dynamic Time Warping (DTW) and do classification using kernel methods. This still requires the window-size as one of the hyper-parameters)
Switching models and timeseries segmentation
The way you have associated those patterns to a label, I think you can assume that your problem is an example of switching state space models, where you have $k$ states, each defined by a state-space model, and you switch among those states. you can assume the switching mechanism depends on the data, or the duration of the current state, which makes it interesting for your problem, since you have the challenge of long-term dependencies. switching model can learn the expected dwelling time on states, thus it will form a prior belief, not to switch too often, if it repeatedly sees rare switches. Nice thing about this model is that you can do timeseries segmentation, which marks for you your "class" regions. the learning is a bit tricky though. for that purpose, you can checkout this paper.
conclusion
In timeseries problems, the problem definition is often very important and deciding. Sometimes you aim for an impossible problem to solve, whereas with a change of perspective, you can solve it. As an example, imagine a simple linear trend with an additive noise component. if you define your goal as "predicting the next timepoint" you might not achieve satisfying "regression" results because you're basically trying to learn an independent noise. Whereas you can instead be interested in "if the timeseries will go down or up in the next timepoint" then the expected value your label is simply the slope of the trend, which is way more doable than regression. My point is, maybe you can get better result if you think of your problem as another form of research question such as segmentation, etc.

Hello, thanks. It's been a little while since I was thinking about DTW or Shapelets. If I remember correctly You compute distance matrix between points of the serie your're studying and one or more series of references in a database for example (I my case it would be the subpart that compose my curve like in fig.2). Then you send the result of the calculation to a model (svm, ...) ? I had also investigated image-transformation to turn time serie into an image - very interesting but until now it gave me only average results.

yes, a kernel method accepts the pair-wise distance matrix. for a more basic method, you can also do KNN for example, if your sample size allows for that.
your "average" results might be due to lack of information in shapelets. maybe you can do some summary statistics to see how diverse your signals are, or calculate TFIDF on your shapelets. I'd start with the most basic models to check for some trivial results

Hi, I have indeed started by extracted on shapelet statistical and temporal features (haven't try spectral features yet) such as Mode, Skewness, signal_distance, and so on. Then I used features selection to keep only the most relevant one. I then trained a SVM model  on it and achieved a pretty good précsion on test set. But once I try on a stream version with different windows size I end up with the same class prediction (class 2). I'm reading for the switching model its look very interesting.

Since your data has lots of repeated patterns, and thus correlations, I suggest you try using Singular spectrum analysis. The core idea is that you re-structure your timeseries of $N$ points into a matrix of size $N - L + 1 \times L$. The first row has (one-indexed) samples from $1$ to $L$ from the series, the second has samples $2$ to $L+1$ etc. See Wikipedia for an example matrix.
Then you apply PCA (or SVD) to this, projecting your samples to a lower dimension. Maybe you'll get neat clusters even in 2D, and can apply for example the simple K-nearest neighbors algorithm. The benefit of running SSA first, is to visualize your data, and you can run kNN in a much lower dimensional space.
And since I'm a big fan of linear methods, an other option is to apply Linear discriminant analysis to the "time-shifted" matrix. It will learn to project your time-series samples into a lower dimensional space, with maximal margin between classes.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to explain near zero gradients on first epochs?
As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero and as we train the net the gradients will arrive to values nearest to zero.
But it is not the case as you can see for example here (This image show gradients distribution on each epoch):

https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA
and here (This image show gradients for 5 layers after the first batch):

https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html
I've seen the same behavior in other simple nets.
Can someone explain this?

I'm not sure how meaningful gradient distributions are. Try observing L2 norm of gradients. That should get smaller as your model converges.

Thanks for your response. I've tried but the distribution is basically the same (centered on zero). Why do you think that L2 is better?

L2 norm of the gradient kinda tells you, on aggregate, whether your gradient gets smaller or not. Individual dimensions may vary. L2 norm of a vector is just a scalar value, you can just observe how this changes over the training,

Nope.  Delta rule: taylor series with truncation relates mean, variance, and gradient.
Variance is proportional to the square of the local slope in the region. If your minimum is at the bottom of a wide-area (think grand canyon) then you can have low variance.  If the minimum is at the bottom of a well, then the local variance can be high.
The goal of optimizers (like those with momentum and adaptivity) is to test the local area and maybe bump the learner out of a local but not global best to something better nearby.  That means you aren't exactly testing the point as much as you are testing the space around the point.


text:
How to manage impossible actions?
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I can't use a function that, given a state, returns all the possible actions. In order to solve it I am using a dummy state which is terminal and a bad reward.
When the agent tries to take impossible actions, what  is the difference between using a dummy state which is terminal and remaining in the same state (+bad reward) until the end of the episode? Other possible solutions?
Specifically, how can I either avoid defining an impossible action or alternatively define it as an impossible action?

What do you mean by "Because of the solver"? Do you mean "Because of Q-learning"? Also, which function are you referring to when you say "a function that, given a state, returns all the possible actions"? A policy? Also, are you using the a "dummy state and a bad reward" for what?

I am using Julia language and its q learning solver doesn’t work with state dependent actions. The function was made by me

By "state dependent actions", do you mean that your Julia implementation doesn't accept an action that depend on the state (whatever that means - maybe you mean that the function you're calling doesn't accept a state as input in order to product an action)? I don't understand your question at all. Edit your post to try to describe your problem more in detail and clearly. Right now, it seems that you have a problem related to impossible/illegal actions, but, apart from that, it's not fully clear what your problem is (at least to me).

Done, I have changed the description

You could code your agent's policy to never select impossible actions.
Your other question implies that you are writing your own behaviour policy function (e.g. you asked about implementing a softmax policy). The behaviour policy function must take the current state as input, and output an action choice.
For your environment to work, some code somewhere presumably knows which actions are "impossible".
Rule the impossible actions out in the policy by calling the environment code that knows about them, and assigning probability zero to them, then you don't need to care that the rest of the framework is allocating places in the Q table for them.
An alternative to querying the environment directly from inside the agent would be to pre-populate the Q table with large negative values for all impossible combinations, and set probability to zero in your policy for any Q value this low or lower. You should be able to choose a really large negative value for this purpose (perhaps even -Inf if supported) to avoid any interference from the rest of the environment's reward structure. The advantage of this approach is that an eventual greedy agent would not need to query the environment code, it could use the Q table as-is.


text:
Is there a term for unquantifiably uncertain prior knowledge?
I'm working on a clustering algorithm which assigns each data point an index encoding its cluster. Index permutation is irrelevant to the correctness of the result. The algorithm is self-learning, in that it doesn't require labelled data to achieve its task.
The learning process can be sped up by initially providing labelled data, where each sample is assigned to a specific cluster via its index. Of course, prior to learning, we can only assume the existence of some specific clusters which may actually differ in position, shape and number. Thus, the labels are allowed to be wrong. Even if all of them are wrong, the algorithm will eventually still converge to a correct solution, but it will require a greater number of samples until it does. As long as the majority of labels are correct, there will be a measurable speedup. So the working assumption is that not all, but a sufficient portion of the labelled samples initially fed into the learner, are labelled correctly. But really, there is no guarantee. It's a mere heuristic. Neither the possibility that all labels are correct, nor that all labels are incorrect, can be excluded.
Now, I am unsure about how to describe this concept. Initially, I thought: The speedup is due to 'prior knowledge'. But I find that this term doesn't capture the uncertainty aspect of the concept very well. Personally, I'd prefer something like 'potentially flawed assumptions', but that seems a bit too vague and unwieldy. Is there an established term for this kind of concept in machine learning? If not, what would be an appropriate term which is compatible with existing terminology?

It seems that the problem you're describing is that the labels could be incorrect. You can just call it noisy labels (e.g. this paper uses this term to refer to labels that have been flipped with a small probability). If you are performing supervised learning with noisy data, you can call it weakly supervised learning (specifically, inaccurate supervision).
Note also that, when analysing ML algorithms in learning theory or statistical learning, we also talk about an irreducible error/noise, i.e. the error that you cannot remove because it is caused by unknown or uncontrolled factors (e.g. how the data was sampled or collected).
Finally, if you have trained a model with inaccurate data, you could also say that you have introduced a bias, but biases are not necessarily bad, you can bias a model in the good direction.

I personally feel like noise doesn't quite hit the nail on its head, as faults in the initial labels would not be caused by interfering background processes, as is usually the case in measurement. It's rather that the correct labels cannot be known beforehand due to the very nature of the problem. They are merely educated guesses which tend to speed up the search for an actual solution. But the term 'irreducible error' seems accurate, even though the error isn't really quantifiable in any meaningful way.

The irreducible error is more related to the generalisation ability of the model. If you're more interested in the problem of inaccurate labelling, then some term like "inaccurate/approximate labelling" might convey what you want to say. I am not sure if these are common. I know that weekly supervised learning can be divided into subcategories, one of them is when you train the model with "approximate labels" (which is your case).


text:
Encoding Actions with Parameters in Neural Network Output
I have a task which I would like to teach an AI to perform. The input to the task will a screenshot of the screen and the output at any given time step is one of the following actions:
    DoNothing
    Click(x, y)
    Drag(from_x, from_y, to_x, to_y)
    KeyPress(Key)
    LongKeyPress(Key, duration)

where:
    Key ∈ { 1 , 2 , 3 , 4 , Space , Up , Down , Left , Right }

I think that deciding which action to take could be done with a softmax output layer, but I don't know how I would also encode the parameters of each action. For example, the Click action also 2 co-ordinate values but the Drag action needs 4 co-ordinates.
My best idea at the moment is that I could use one neural network to decide on which action type to perform and for each action, train another network to decide the parameters. So for example I have the main network which decides to take a Click action, and then a secondary network which will choose the x and y co-ordinates (given the same input).
The problem with this is that I would end up training 6 networks (decide the action, Click co-ordinates, Drag co-ordinates, KeyPress key, LongKeyPress key, LongKeyPress duration). The drawback for this is that I would need a very large dataset with a lot of samples for each type.
I considered how it might be possible to encode all the values into one output, but I wouldn't know how to deal with missing values (for samples where the user clicked, what value should i use for the Key output of KeyPress).
I have tried to find research papers on the topic, but I must be missing the right keywords because I can't find anything about strategies for this problem. Any ideas or pointers to relevant material would be great!

I think your problem would fit to reinforcement learning, but you also mentioned a dataset - so the main question is: Given a screenshot, is there one and only one correct action to perform? If yes, I think you are on the right track with classification, If No you should explore the realm of reinforcement learning (RL).
Also, if this is RL, the number of possible actions is enormous (especially the continuous actions click + drag & drop) which makes the task really hard and you might want to simplify the actions a bit.

Unfortunately, I cannot explore the possibility of RL because I do not have a test environment in which the agent can 'practice'. Besides this, I will mention that I am not interested in finding the most efficient solution, but instead the most human-like solution. That is why I would like the agent to learn from a dataset of actions made by a human. I know that I might also be able to use a GAN architecture for this but that is something I have no done before. Also I am aware that the action space is huge, but I cannot see a feasible way to reduce it without also reducing expressiveness.


text:
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max policies instead?

In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the most likely to visit parts of the trajectory (state, action, reward sequences) that you want to explore for improvements.
The main problem with using a softmax policy in Q learning* is that you have no independent set of preferences, just value estimates. So the agent's exploration performance would become dependent on the scaling of the reward signals. A simple method to adjust for that is to use a temperature hyperparameter $T$, that divides the action value estimates used in the softmax. A high temperature results in near random behaviour, and a low one will almost always select actions with the highest action value. You can start with a high temperature and slowly decay it, similar to decaying $\epsilon$ for $\epsilon$-greedy.
Will this help for you case? Maybe . . . although using an approach which filtered out the impossible actions would be cleaner and definitely faster (in terms of number of samples required).
There are other approaches you could use, such as Thompson sampling, Gibbs sampling, upper confidence bound (UCB). Some of these don't translate easily to deep RL, so you don't see them mentioned as often, but they can work nicely in tabular approaches.

* Outside of Q-learning, in policy gradient and actor-critic methods a softmax policy can be used more directly, and might be a good choice when action space is large.


text:
Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?
I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem that takes a lot of time for computation, so I would like to know if there are more efficient methods for it.
Why are tabular methods are not useful in large state spaces? Maybe too many possible combinations? Could Softmax Action Selection be better than epsilon greedy?

Please, don't change the meaning of your question, once it's answered. "softmax action selection" and "value function approximation" are the same thing, so it seems that you changed your question. If you have another similar question, ask it in a new post!

Your question contains the answer. Use value function approximation. Tabular methods must compute a value for each state. That becomes unfeasible with large state spaces. Function approximators can genererlize, and perform well even without ever having seen every state.


text:
How to decode P bits that represent a random weight generator?
So I've been tasked by my neural network professor at university to replicate the following research: Intelligent Breast Cancer Diagnosis Using Hybrid GA-ANN.
Each chromosome represents a possible net, more specifically, a possible MLP network. They've used a binary convention, and have used $P = 15$ bits for the random initial weight generator, $Q=2$ bits for the number of nodes and $R = 9$ bits for feature selection.

P bits random initial weight generator allows 2P different
combinations of the initial weight. The numbers of hidden
nodes (i.e. represented by Q bits) permit the GA to explore
up to a maximum of 2Q hidden nodes’ size. For the
representation of the feature subset, the value of R is set to
the number of full feature size. Value ‘1’ or ‘0’ indicates if
the feature at that particular location is selected or otherwise.

Decode each chromosome
in the population to obtain the selected feature subset, hidden
node size and random generator.

I don't understand why they say with $P$ bits, there's $2*P$ combinations, wouldn't it be $2^P$?
Also, I can't grasp how they decoded the $15 P$ bits into the net for the weight generation. I've searched everywhere but I can't find anything specific. What I thought to do was to transform the $15$ bits into a decimal number and use it as a specific seed for rand and randn function in Matlab, through which I make random initial weights.

The paper's pdf doesn't seem to be freely available. So, it might be a good idea to quote the relevant section.

I did, hoping now it's more clear what my question is.

You can always try to email the authors. They are usually happy to have someone interested in their work.

I don't understand why they say with $P$ bits, there's $2*P$ combinations, wouldn't it be $2^P$?

I think you are correct here, and would suspect a typesetting issue or typo. I also think that 2Q should be $2^Q$ in the same paragraph.

Also, I can't grasp how they decoded the 15 bits into the net for the weight generation.

Without anything in the paper or associated materials, I don't think it is possible to be sure.

What I thought to do was to transform the 15 bits into a decimal number and use it as a specific seed for rand and randn function in Matlab, through which I make random initial weights.

This could work, although it will make the GA search over those 15 bits dependent on all the bits at once, due to the nature of PRNG seeds. Any crossover or mutation would destroy any coherence between generations. Perhaps that is not important, although it then does seem strange to dedicate the majority of the genome to it.
The title and description does imply that the GA is mainly being used to set start conditions for training the network, so values of P might not be very important, in which case it would mainly be used to avoid risk of poor initialisation missing the best architecture.
However, P could also be an encoded  multiplier for initial randomly sampled weights, in which case the search should find stable values for the most significant digits of P. That would be similar to how Q is used (as an encoded number).

Hello, thank you for your answer. I'd like to elaborate more on the solution you proposed: when you say encoded multiplier, what exactly are you suggesting? Because I already transformed the 15 bits into the actual decimal number. What should I multiply that for? Shouldn't all initial weights for both input-hidden and hidden-output layer be different?

@JOSEPHCAROÈ I mean use it to multiply all randomly generated weights. You might generate weights by sampling $\mathcal{N}(0,1)$ for each one, then multiply by P/MAX_P

Okay so supposedly, with P = 15, MAX_P = (2^15 - 1) and P = the decimal number of the 15 P bits.


text:
Given A and B, C are independent of each other. Given A, B and C, D and E are independent of each other. What is the minimal number of parameters?
Assuming all variables $A, B, C, D,$ and $E$ are random binary variables. I come up with Bayes net: $D \rightarrow B \rightarrow A \leftarrow C \leftarrow E$ which has the minimal number of parameters of 10, I think. However, the given choices are 8, 15, 16, 21, 23, and 32. I don't know what I did wrong??

I cannot understand this question. I think it is missing context, to clarify what the parameters are being used for. Could you please link or quote more from the original source?

see the image above

Is this a homework problem? Is it taken from a book? If yes, which book? Can you provide more context, like, if this is a course, what course you're taking? Is this a course on Bayesian networks, statistics or something else?

it is a past paper from a course on ai

I don't really know what a Bayes Net is, or how to apply Bayes rule in this scenario, but I think the answer is (d) 21 = 1 (for A) + 4 (for B, C depending on A = 2 x 2) + 16 (for D, E, depending on A, B, C = 2 x 8). Hopefully someone can put that into terms and approach from your coursework, and I'm interested to know if I'm right. If you address nbro's questions it may help someone answer using the correct terminology.

Thanks to @Niel Slater's comment, I realized I had the wrong graph structure. The prompt is asking to first determine the graph structure given a set of conditional independencies, then second how many parameters are needed to express the joint probability distribution over five boolean variables?
1a. Given the outcome of A, the variables B and C become independent of each other.
Clearly the prompt above indicates a fork structure: $B\leftarrow A \rightarrow C$
1b. Also given A, B, and C, the variables D, and E become independent of each other.
The prompt above is slightly more difficult because I am not certain if this is saying $D\bot E | A,B,C$ or the three separate conditional independencies:

$D\bot E | A$
$D\bot E | B$
$D\bot E | C$

Maybe the two are the same, I am unsure, but I think the resulting graph structure assuming the latter results in the below conditional probability tables.

There are 18 total parameters and 9 non-redundant parameters. Neither are choices. There might still be a mistake in my work, or the prompt is mistaken, or both. I will leave the original work below where I started from the wrong graph structure. I suspect I am wrong with 1b but I couldn't think of a graph structure after playing around within dagitty satisfying $D\bot E | A,B,C$ where all $A,B, C$ have to be observed at once.
Below starts with the incorrect graph structure
I don't see the correct answer among the choices from what I understand. From my interpretation of the prompt, the question asks how many parameters are needed to express the joint probability distribution over five boolean variables given the graph structure of $D\rightarrow B\rightarrow A \leftarrow C \leftarrow E$ ? If my understanding is correct, the answer should be 20, where you could use the formula $n2^k$ where $n$ is the number of boolean variables and $k$ is the maximum number of parents a node has for the given network. I can also just work out each conditional probability table for this small of a network, as shown in the picture below.

If you count up the number of rows in each table, you will get 20 parameters. Some are redundant, shown in grey, so if we only include the non-redundant parameters, we get ten parameters as the OP suggested. Without a graph structure, the number of parameters to express the full joint distribution is $2^n - 1$ resulting in 31 parameters for 32 different instantiations. There are 31 parameters since each variable only has two dimensions, with five variables ($n=5$), leaving one instantiation as redundant. If the answer is 21, I am unsure where the extra one parameter is from.
Reference

See page 515 of section 14.2 The Semantics of Bayesian Networks, sub section Compactness and node ordering of Artificial Intelligence: A Modern Approach Third Edition


text:
Can single agent trained by RL handle total different tasks?
In the classic "human level control" paper, it writes:

We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters.

It seems that different networks(agents) were trained on different tasks. Is that right?
Can single agent trained by RL handle total different tasks? e.g., can we train a network to get reasonable results on different Atari games?

multi task RL and Continual RL are two very broad areas of research

Yes, early Deep-Q papers set out to prove a single architecture can reasonably approximate the policy for a single Atari game but not all Atari games at the same time.
In the simplest version of your question you seem to ask if a single model can generalize to a few similar tasks, not just one.  This is still within the range of vanilla DQN as possible especially if features of several games in the distribution have common elements (ie, paddle ball games, side scrolling shooter, etc).  Model sizes inevitably must increase and dissimilarity of tasks will make progress slow.
As the tasks become very dissimilar we wade into the waters of Meta-RL.  You might check out "A Survey of Meta-Reinforcement Learning" by Beck et. al 2023 for a very recent discussion of the state of the field.
If the task(s) are to be fully learned and then more tasks introduced later the field becomes continual, lifelong or non-stationary RL.


text:
What is the rigorous proof behind the causality simplification step of off-policy policy gradient
In lec5 of Sergey Levine's CS258 on RL, I can understand the derivation of the causality  simplification of on-polciy version below (page 17).

But what is the detial of the derivation of the last line in the PPT below (page 25)?

It seems that the cumulative reward term and the importance ratio term cannot be decompled and handled one by one.

casuality =/= causality

@Alberto Thanks for your kind reminder. Have edited.

While this may be apparent to many, I'll provide an answer anyway.
In a nutshell, 2 steps:

Bring the distribution from $\theta$ to $\theta '$ to cancel the importance ratio term. Use causality to simplify the cumulative reward term to reward-to-go. Details can be found here, which utilized Expected Grad-Log-Prob Lemma.
Bring the distribution back to $\theta$. Use distribution law and the definition of marginal distribution to get the final formula.

proof:
\begin{align}
&\mathop{\mathbb{E}}_{\tau\sim \theta}\left[ \left( \prod_{t=1}^{T} \frac{\pi_{\theta '}(a_t|s_t)}{\pi_{\theta }(a_t|s_t)} \right)\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\
=&\mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\
=& \mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\sum_{t=1}^{T}\left(\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \sum_{t'=t}^{T}r(s_{t'},a_{t'})\right)\right] \tag{causility} \\
=& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta '} \left[\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{distribution law}\\
=& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{T} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right]\\
=& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{marginal distribution}\\
=& \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\sum_{t'=t}^{T} \prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} r(s_{t'},a_{t'}) \right]\\
=& \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\prod_{{t''}=1}^{t} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})}\sum_{t'=t}^{T}  r(s_{t'},a_{t'})\prod_{{t''}=t}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \right] \\
\end{align}


text:
Why my ppo with critic learn nothing while no critic does learns?
update
After some attempt, I find the learning rate is too small for the version with critic if just adopting the lr(0.0005) of the one without critic. I alter the lr to 0.005 (light red curve) and 0.02(green curve), the return curve goes up but still slow. It is very strange that such big lr is needed! It is way beyond empirical lr range for rl. And why critic need bigger lr than actor?

Original question
I have implement ppo on my target guard environment. At first I just set critic output to zero and train actor network and get reasonable policy improvement. When I create a critic network and use advantage as the gradient weight, it learns nothing. At first I thought my critic code is messed up but I noticed the explained variance(EV) of my ciritc network is improved, wich means it learns something. Although the EV is not good but at least higher than 0 so it should be better than just gives zeros as the value. And my code is from cleanrl, the critic part is not modified. What might be the reason?
Environment description:
General: training a global coordinator to assign 5 defenders to attackers to defend 5 attackers from target area(a circle). one-vs-one interception strategy is predefined.
Obs: (5*5, 25) shape, every defender-attaker pair obs consists of [defender_id, attacker_id, aliveness, relative_position, reltive_angle, ...]
action: global assignment: [target_id_for_defender_0, target_id_for_defender_1, ...]
Critic network architecture:
It is verbose to explain it here. Simply put, use self attention to calculate shapley value of every defender-attacker pair and sum up them as the value. More in this paper
↓(Blue curve is without critic, red one is with critic.)

This is likely to be related to the nature of the environment and how you are representing current state. Could you please add some details using [edit]

It may also be related to the critic network architecture, so please share that and the policy network for comparison

@NeilSlater Thanks. I have update my question.

That critic network seems quite specialised, so it may require someone who knows more about it and your multi-agent use-case, before they can explain your observations about learning rate. Good that you have a work-around and functional Actor-Critic setup now. Although it still seems you get little to no benefit from implementing the critic . . .


text:
What should the discount factor for the non-slippery version of the FrozenLake environment be?
I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much worse for the 8x8 grid with success around 50%. I thought in the non slippery case, it should definitely give me 100 percent success, but it turned out to be 0 percent.
After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the agent going around in circles. Reducing the discount factor from 1 to anything below 1 gave me 100 percent success. But going to anything below 0.9 gave me worse results in the slippery case.
Can someone give me some intuition as to why this happened and how to interpret and choose the discount factor for any use case?
Algorithm used (where $\theta = 0.0000001$):

Could you please refernce the exact algorithm you used - e.g. tabular Q-learning - plus any other hyperparameters that you set? The learning rate especially is going to interact with the discount rate to determine stability.

Hello @NeilSlater, I have updated the question with the algorithm used and I only set one additional parameter theta = 0.0000001

Value iteration requires a model of the environment - i.e. the state transition probabilities. Gym doesn't give you those via the normal API - how are you getting them?

I get them from the API itself. For this particular environment the API gives the state transition probabilities

Do you mean the info block returned after taking the action in the .step method? How are you iterating the states?

No I am not using the step method. It is an attribute of the environment object itself. Calling env.P[s][a] returns the Probability of transitioning to state s_dash from s after taking action a along with the return.
The states are just indexed from 0 to 15 so I am iterating over them using a simple for loop.

After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the agent going around in circles. Reducing the discount factor from 1 to anything below 1 gave me 100 percent success.

This is expected (and optimal, as defined) behaviour with a discount factor of 1 in the deterministic case. With a reward of 1 at the end, no discounting and and no negative rewards for taking its time, the agent has infinite time to complete the task. So whatever it does, provided it does not step into a hole at any point, is optimal.
The caveat is that when following the policy as a test, you need to use random choice to resolve ties (this is assumed in the theory behind value iteration, that the agent can reach the high rewards). Most simple argmax functions will not do that, so you will need to add some kind of shuffle or other random selection over the maximum results to ensure you are really following the policy that value iteration discovered the state value function for.
The last sentence in the pseudocode is ignoring the possibility that you have this kind of situation, so it is not strictly true in all cases. That is, you cannot reliably build a working deterministic policy directly from the optimal value function when $\gamma=1$ for environments like Frozen Lake.

But going to anything below 0.9 gave me worse results in the slippery case.

I am less sure about what is happening in this case. Given that you successfully found an optimal value function in the deterministic environment, then this may actually be optimal too. Whether or not it is may depend on the map - an optimal agent with low discount factor may prefer not to "run a gauntlet" early with risk of failure even if the overall probability of success is better. That may cause it to prefer a safer start that leads to a tougher end, where it is forced to take more risks overall than would be optimal for a goal of completing the most episodes.

how to interpret and choose the discount factor for any use case?

The discount factor is not a free choice for the agent, and it is not a hyperparameter for the learning method in value iteration. Instead, it is part of the definion for what is and isn't optimal. So you should select it based on the problem definition.
In an episodic environment where the agent's goal is to reach a certain state or states as fast as possible, there are two "natural" formulations:

Zero reward for most transitions, "goal" rewards for reaching certain terminal states, and a discount factor below $1$. I think your choice of $0.9$ would be fine, and it is commonly used in the literature for this kind of toy environment.

Negative reward for most transitions, "goal" rewards for reaching certain terminal states. If there is a single "target" state, such as exiting a maze, the "goal" reward can be zero. Usually no discounting, but you could add discounting if it represents something else about the problem.

In more complex systems, the discount factor may also become a hyperparameter of the agent. For instance, it does so when using function approximation with temporal difference learning, because it can dampen positive feedback from bootstrapping. In those cases, where "naturally" there would be no discounting, then you may want to use the highest value that prevents the estimator diverging e.g. $\gamma = 0.99$.


text:
Why is exploitation necessary during training?
I have read many blog articles making all kinds of broad analogies to explain the exploration/exploitation trade-off. However, I still can't fully grasp it. On an extremely abstract level, I understand why you would want to "try new things to gain information", but then I don't understand why you would want to "exploit" in training. It seems as though it would be better to keep trying as many things as possible to gain the most information.
What is the value of exploitation during training? Intuitively, I would think you would only want to explore during "training" and only exploit in "testing".

An algorithm that chooses to always explore during training is unlikely to find an optimal policy because it will be employing a more random search as opposed to a directed search. During training, the neural network aims to determine the relation between states or state-action pairs and the reward signal through past experience. If the agent is always exploring during training, it will never use the experience gained from past episodes to influence its training policy and therefore will search more randomly.
Exploitation during training allows the neural net to use its past experience to guide its future actions and avoid random search. There can be many states in which there is an obvious optimal action. After some training, the neural network may be able to quickly learn these states and corresponding optimal actions. By primarily exploiting at those states, the agent will not be wasting training time by exploring suboptimal actions at those states, allowing the agent to focus its exploration on other more uncertain, unexplored, or complex parts of the state space.
For a practical example, consider the original Super Mario Bros game on NES. Let the reward be the number of pixels traveled to the right before Mario loses a life. If Mario is exploring the whole time, it is unlikely that he makes it very far to the right or over many obstacles, let alone to the flagpole. Since it is more rewarding in general to go to the right, Mario's exploitation action at most states is to run to the right. In this manner, Mario will usually run to the right until he reaches an obstacle (e.g. a pipe, pit, staircase, enemy). At that point, Mario may need to explore to overcome the obstacle, but Mario needed to exploit to be able to reach that obstacle in the first place.

Exploitation is important during training to help the network encounter and learn to handle situations that don't occur until the network has successfully navigated other situations.
For example, consider the Atari game Breakout (a common RL benchmark). In this game, the player must move a paddle on the bottom of the screen to bounce a falling ball. The ball accelerates as the game continues, and the network can only get training data with a fast moving ball after successfully exploiting its knowledge of how to play when the ball is moving slowly.
A purely random strategy is possible (and often used for very early training) but  only generates data useful for learning the beginning of the game.

Imagine trying to navigate a maze from the outside. Let's say you lose if you get to a dead end, and win if you get to the middle. After some experience by random trials, we know where some dead ends are.
In the future, we should exploit our knowledge and not turn directly into a dead end, as this is simply inefficient. If we do this, we shall find the middle quicker :)

This leads to interesting quesitons, such as
'How does the algorithm know if the dead end is always a dead end'.
You might be able to see we might want to tune our randomness rate near a dead end. We might sometimes want to try going down that path, just to be sure, but most of the time lets not bother and go somewhere else. This gives some intuition behind POLICY methods in Reinforcement learning (also see multi-armed bandits if interested) :)

There is an additional factor to consider about exploration/exploitation trade-off, that sometimes applies in addition to the reason in the accepted answer and most other answers here.
Sometimes an agent is required to both act and train itself in a "real" system, or at least one where the rewards are more than just collected training data from a simulation, but also represent actual profits and losses realised by the agent.
This is a common feature of adaptive content display in advertising, which typically uses the simpler k-armed bandit or contextual bandit model - still related to RL, and importantly still affected by the exploration/exploitation trade-off. It is very hard for a machine to model how humans will respond to an advert, so the only reliable measurements are made in production. Each click-through is then real money to someone, so it is important to adapt quickly to incoming data - but due to the variance in results it is also important to still keep testing the non-optimal choice and improve on any early estimates.
In such a scenario, you have to accept some non-optimal returns as the cost of finding the best ones through trial and error. However, it is important to balance this with gaining as much reward as possible whilst training.
So it can be a more important consideration, to obtain best cumulative reward during an ongoing training process, than even finding the optimal policy. That means taking care to balance exploitation and exploration, often strongly favouring exploitation after a relatively short high exploration phase.

If you explore too much, you waste your time (among other resources.) You will probably exhaust your resources before you learn anything meaningful.
Let's say your goal is to learn as much about Star Wars as possible within a library. If you fully explore, you just pick books at random.
Exploitation might look something like "pick most of your books within the Sci-Fi section", or "choose books with lightsabers on the cover" or "books with 'Star Wars' in the title" because that's where you have found relevant information in the past.

In supervised ML there is no exploration and exploitation.
In reinforcement learning, the agent in each step has many choices.
So the agent can exploit, meaning gaining the highest reward known to him from the next move. Or explore, trying to get a better long-term benefit by trying a different move.


text:
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to say, the real reward function is partially dependent on previously accumulated reward.
Is there any way to implement this kind of dynamic successfully?
I have tried to, but for some reason, the agent completely stops learning when I do this. I can implement a linear/additive reward function and the agent does learn good behaviors, but I feel that it's important for the agent to "understand" the true reward dynamic.
Essentially, here is the reward function I have:
reward = points_gained_this_step

But here what I need:
reward = points_gained_this_step*(total_score_so_far)
total_score_so_far = total_score_so_far + reward

Has anyone ever worked with something like this? Any ideas/insight for how to implement such a reward? I might be wrong, but it seems to me like part of the problem is exploding/vanishing gradients?
EDIT: I have already added "total_score_so_far" to my observation space.

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent is going to need to know this.
You may still have some problems after doing this. The final return that the agent needs to predict is likely to have the following traits:

A value that could range by orders of magnitude, making it hard to scale loss functions. You may be able to base the loss function for value predictions on mean relative error to help with this.

Large variance, making it difficult to learn expected returns, especially from the impact of early decisions. If your immediate rewards have any random element, this could be a major problem.

If value-based methods such as DQN are struggling in your case, and the optimal policy function is straightfoward, then you may want to try policy gradient methods, maybe even basic REINFORCE, to avoid needing to predict returns whilst dealing with such a high variance. You will still need to take care with scaling, as policy gradient methods scale update steps based on the return.

Thank you very much for your response. I forgot to mention, I have actually already added the accumulated reward to my observation space. While I'm not sure, I doubt the optimal policy is straightforward (it's complicated enough that I myself was unable to learn the optimal policy by manually playing with the data). I agree with every point you make, particularly the two bullet points. I had an idea to make my episodes shorter (less timesteps per episode), so that 1) there isn't enough time for rewards to get huge/tiny and 2) the "impact of early decisions" is reduced. What do you think?

@VladimirBelik That's a good idea IMO. If you can simplify the environment and find an approach that works there, you may be able to scale it back up to the full environment by degrees based on what you discover. I cannot say what is likely to work though, as I have not really studied an environment with reward behaviour that you describe in the question


text:
Why are agents trained in episodes, even in non-episodic tasks?
Let's consider some non-episodic problem. Maybe a game which can go on forever.
My question is: Why are agents still trained in episodes?
My understanding is that the agent's neural network is updated in batches depending on the batch size (so every x timesteps, the neural network will be updated). Therefore.. nothing special happens at the end of an episode, right? The agent does not "review" its performance and update itself again or anything like that.
Nevertheless, I find that shorter episode lengths can be helpful in my problem, but for the above reasons, I don't understand why 100 short-length episodes are better than 1 huge long episode, or why that would make any difference at all in a non-episodic task.
Any insight would be greatly appreciated!
EDIT: I suppose I'm trying to understand whether anything "special" happens in the network at the end of an episode. If not, let's imagine a situation where I have a time series environment, and I train the agent in one single episode from t=0 to t=100 . Now, let's say I train 10 agents, each starting a different multiple of 10 (t=0, t=10, t=20 etc.) and each new agent's start environment is exactly the same as the end environment of the previous agent (startenv for agent at t=20 is equal to endenv for agent that trained t=10 to t=19, preserving continuity). Would these two situations result in the exact same final model/agent? That is, would the 10th agent of the divided data group be the exact same as the agent at the end of the t=0 to t=100 run? Maybe this is impossible due to the way the gamma discount parameter will interact with the episode length. What if we set an extreme gamma parameter, so that the differences are negligible? I just want to understand if the agent in any real way considers "episodes" in its learning/training.

I tried to give some of that information here: https://ai.stackexchange.com/questions/35793/for-continuing-tasks-is-the-choice-of-episode-length-completely-arbitrary/35796#35796 - it suggests there is a balance to be struck due to possibly special start conditions or isolated loops. Could you suggest what additional information might help you.

@NeilSlater I edited the question. Does that make it more distinct and clear? I agree that the answer you gave definitely touches on this, but I think I'm trying to really understand if there's something special about an episode ending, or if it's purely a hyperparam (as you mentioned) which determines the states an agent can reach.


text:
How to teach DRL agent to avoid erratic, high frequency behavior (when costs for taking actions are already included)?
I have a situation where an agent can take actions to enter a state where rewards can be obtained (or costs can be incurred), but the actions themselves have a small cost. In my reward function, I give the negative rewards immediately when the agent takes the action.
As a result, if the agent takes these actions extremely frequently, the small, repeated costs of the actions outweigh the rewards gained during the very short periods between actions, and the agent does poorly. For some reason, my agent doesn't seem to be able to understand this. It acts erratically, taking actions at extremely high frequency.
I have tried giving it a sense of "frequency" by having a state variable that increments by 0.02 for every time step that the agent is in the state where it can earn rewards, therefore giving it a direct measure of "how long I've been in this state". This did not remove the erratic behavior.
Is there some other way I can give it a sense of "frequency"? Why is the agent unable to grasp that it shouldn't keep "flip-flopping" and incurring these costs?
EDIT: From @NeilSlater's comment, adding more info.

Yes, the agent can just "wait" with no associated cost. There are two possible states (one in which it just sits there and nothing happens, and the other in which it can earn rewards or incur costs, depending on various factors). There are 2 possible actions - the agent can either 'stay in current state' (meaning no costs/rewards in the "inactive" state, and costs/rewards in 'active' state), or it can move between states, which incurs a small cost.
My environment is not naturally episodic. I chose a somewhat arbitrary episode length to have some end.
I'm using a discount factor (gamma) on the order of 0.9 to 0.975 . Changing it within that range doesn't appear to make a huge difference (though I can investigate this more)
I'm not very familiar with the technical definitions of MDP vs. POMDP, but given that there is a huge stochasticity in my observation space, I think POMDP might be correct. Nonetheless, the costs associated with moving between states are completely consistent, so I don't know how to understand this behavior.

Could you clarify: Is one of the agent's action choices "no action" or "wait" with no associated cost? Also worth stating if your environment is episodic vs continuing, and what discount factor you are using (or are you using average reward). My biggest suspicion though is that you have some important hidden state - but I cannot tell anything about whether your state model is correct or if you might have a POMDP from your description.

@NeilSlater I'm sorry for not being clear originally. Thank you for the questions, I added all answers in my edit.


text:
How should you reshape data before feeding it to LSTM layers?
I was curious if anyone had any advice on how to reshape data for a recurrent neural network. What I've been doing is array.reshape(len(X_train), # of points in time, # of features)
And then in the array, the X axis represents the features, and for each item, there are entries corresponding to the number of points in time (and a column with id so I know what the original id is). So if the time series has 12 points, each sample will have 12 entries in a row in the array.
Am I reshaping correctly, and if not, how can I reshape correctly?
Edit
The array initially looks like this:

This is a pandas array which I convert to numpy by using values.astype(float). The article, and timestep column are just for reference. I am using the command array.reshape(-1, 12, 17)
The -1 is the length of the array, 12 is the timesteps (or number of entries in y axis for each article), and the 17 is the number of features at each time step. I am then feeding that data to a recurrent neural network.

An LSTM takes in a sequence of items. In Tensorflow, this takes the form of a tensor with the shape $[ batch, sequence, dim ]$, where:

$batch$ is the size of the batch
$sequence$ is the sequence length
$dim$ is the size oft he embedding dimension.

For example, I could have a tensor in the shape $[ 64, 100, 25 ]$, and $64$ would be the batch size, $100$ would be the sequence length, and $25$ would be the size of the embedding dimension.
To be more specific, on the sequence dimension here each element is an item in the sequence.
Given the specifics of your data, each time step would be a single item in this sequence, and the features would be the embedding space, giving you a tensor in the form $[ timesteps, features ]$ for each input sample to your model, which would then be stacked as $[ batch_size, timesteps, features ]$ by e.g. Tensorflow in your data pipeline.

In regards to the data, I have data for 17 different features of data at 12 different time points for lots of different articles (about 70 thousand). What I'm curious about is how to setup the array and then reshape it. The way I currently have it looks like this:

I can't figure out how to format it: but I will upload to original post.

Updated answer based on your additional information.


text:
Does pairing children with their parents cause any harm (in a genetic program)?
If you pair parents with their children (with a cross-over) does this prevent making individuals which are more fit or does this cause other side effects which are harmful to the genetic process?
I can provide any specifics about my concrete program but don't know what is relevant (and cannot write the whole setup here).
EDIT: As I think I figured out the answer myself I'll add the information that I think is relevant,
if I'm wrong, and you'd need more information, don't hesitate to ask (I check stack exchange quite often and am always pleased to communicate).
The relevant information:
2 parents create 2 children with a random 10-point cross-over.
The 10 points get chosen out of 700 lines of dna.
(See my answer for more information.)

I assume that, in your case, children are derived from parents by applying a mutation to the parents. Right?

You're asking or raising at least 2-3 distinct questions or problems, respectively, here. First, you seem to be asking if pairing a child with a parent can prevent diversity. Second, the question in the title. Third, whether combining the child and the parent can lead to fitter individuals. I assume that the question in the title is your main question and the others are just follow-up or sub questions, but I recommend that you edit your post to either leave only one question or clarify which of the questions is your main one. This would help answerers to focus on one question.

By the way, if you're looking for research on any topic, you can use the tag [tag:reference-request]. This will clarify that you're not looking for a debate.

Please, also change the title accordingly.

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Children pairing with their parents has a chance of harming the genetic process:
If a child pairs with one of it's parents it's possible to have identical fit individuals who pair with each other (and produce identical individuals, so stalling the genetic process).
This chance is still existing, but diminishes drastically if children are not allowed to be paired with their parents (but still with their grandparents).
In that case a grandchild has dna of 4 different individuals, so it's still possible to have a cross-over which is identical to itself, but chances are very slim and are very unlikely to harm the genetic process.


text:
What is the need for _agency_ in AI?
Why seek to develop artificially intelligent agents? Are there certain advantages and/or needs provided by such supposed intelligent agents that are preferred to simply using intelligent tools that are devoid of agency (e.g. language models)? If so, what are these needs and advantages that intelligent agents could serve?
Basically, why would you want agency in an AI Model in the first place?
Here, I define agency as the ability to autonomously perceive and interact with a given environment. Anything capable of agency is then an agent. Furthermore, I define

autonomous perception: the ability to perceive a given environment without the need of an external agent
interaction: the ability to change the state of the environment

Please move your follow-up questions to actual separate follow-up questions, to keep the focus on your main question. You can link back to this one in the body text if you feel it is important.

I removed my follow up questions elsewhere for focus, and now my question is closed for lack of focus - How can I make it more focused?

The close/reopen is a bit laggy because it relies on people voting. I have voted to reopen.

When dealing with AI on a mathematical level we do not use soft terms, we define them rigorously. I recommend researching mathematical definitions of agents. Everything can be mathematically defined as an agent within some environment (eg a neural network has a subset of a computer's hardware/software as an environment). Though, not everything is an interesting agent. I also argue that a LM is not devoid of agency by definition: "action or intervention, especially such as to produce a particular effect." Agent: "a person or thing that takes an active role or produces a specified effect."

Many interesting applications of AI/ML do involve agents. And not all AI/ML is about agents. If you want an AI/ML model to control a robot or a video game character then naturally you are trying to train an agent.

@thesofakillers not "somehow" but by definition of the term "agent." Furthermore, it is not the AI model that is the agent it is the real hardware and software implementation of that model that becomes the agent. Because then you have "a person or thing that takes an active role or produces a specified effect." I think the issue here is that you have an implied level of "complexity" or some other constraints that constitute your own personal definition of "agent" from a more theoretical definition of "agent." Checkout: https://en.wikipedia.org/wiki/Intelligent_agent

Might be something interesting to discuss in the chat: https://chat.stackexchange.com/rooms/43371/the-singularity

Interesting question(s). I understand an 'agent' as an entity that perceives its environment, performs appropriate actions, and can to a certain degree adapt its behavior. If such an agent is also endowed with an understanding of the consequences of its actions the agent also has agency. Of course, the complexity of agents varies widely with respect to the complexity of their forward dynamics model, the amount of domains an agent operates in, and the degrees of freedom an agent posses in its actions. Both the idea of what an agent is and what agency constitutes should best be seen on a spectrum instead of as binary concepts (my dog likely has more agency than most RL models, but both show signs of agency).
Video games provide a minimal (controllable) set up to test a range of capabilities. They provide a safe and reliable testing bed that one can quickly and cheaply iterate on.
RL is a subdomain of AI, which as a field is interested in intelligent agents. Many domains, such as cognitive neuroscience, computational neuroscience, behavioral economics, and others are also interested in the study and modeling of intelligent agents. Building and applying those is however particularly relevant in RL and robotics.

I guess the form of agency I’m referring to is “performs appropriate actions”. Why pursue an “agent” that takes actions instead of a tool that e.g. describes actions to take? Why are we interested in agents basically? What advantages do they have over e.g. a transformer?

I'm not sure that I can follow. In the most general sense, you could consider every output the model produces as an action (e.g. a language model producing a text output). Such a language model could be based on a transformer architecture but still fall within the definition of being an agent if it its influences upon the environment are interactive. The term 'agent' can thus be used as an umbrella term for a broad range of models with different architectures.

I disagree. A vanilla LM is barely interacting with its environment - you could argue at most that it interacts with the environment only indirectly by influencing the actions of actual agents (humans) with its output. For me, an agent is something that directly interacts with its environment (e.g. manipulate it, guide its own perception of it, etc.). LM’s are just tools at the mercy of agents

The colloquial definition of an agent (in an RL setting) is: "The learner and
decision-maker is called the agent. The thing it interacts with, comprising
everything outside the agent, is called the environment. These interact continually, the agent selecting actions and the environment responding to those
actions and presenting new situations to the agent"(Sutton & Barto). Under this definition, one could consider an LM-based chatbot as an agent. If you have a different definition of an agent, you will of course arrive at a different conclusion.

But an LM-based chatbot is not an LM, it’s an agent that uses an LM as a tool. The definition you quote aligns with what I am saying and I don’t see a vanilla LM aligning with it without performing some minor mental gymnastics.

An LM that is prompted to predict the next token and does so is performing an action in reaction to an input, so one could consider it as an agent. But tbh this is on the 'minimal' end of the spectrum (it's like considering cells as an agent because they have sensitive and locomotive capacities).


text:
Are platformer games, with the camera centred on the character, examples of egocentric vision?
An example may be CoinRun, where the character sprite is always centred in the camera view, while the environment moves as a result of player input.
To me, this sounds like egocentric vision, although the wikipedia page seems to rely heavily on the notion of a wearable camera in the definition, and the treatment seems to ignore 2D environments, directly jumping to the 3D case.
If this (CoinRun, other similar platformers, etc.) is indeed not an example of an egocentric vision problem, what is it an example of? Is there a formal way of expressing the difference between platformers where the camera is centred about the character vs platformers where the camera is centred about the environment?
I am particularly interested in this to help me with formalizing differences in environments in an RL context, where environment state is perceived visually, for example through a CNN.

At the time, this probably referred to what we call an POV or first person view. It was applied to head mounted cameras. This term was developed a long time ago to describe something that didn't have a term for it by researchers slowly getting into virtual experiences.

naturally approximates the visual field of the camera wearer

If a platformer was 3d and you controlled their field of view. Then it's egocentric.  A platformer like coin run would NOT approximate the runner's field of vision. There is a 3rd person camera following the runner. It is not egocentric.

Thanks. Any clue for the second part of my question? Is there a formal way of expressing the difference between platformers where the camera is centred about the character vs platformers where the camera is centred about the environment?

I'm Not entirely sure.
"chase camera" or "follow camera" provides an unambiguous term to describe a third-person camera that follows the player's movements. These terms are widely used in the gaming industry and are well understood by researchers in the field of HCI.
I can now see how the term ego-centric might apply here.
Perhaps you can consider using the term ego-centric view with modifiers to the phrase to specify how it's egocentric. first person ego-centric view vs third person ego-centric view

thanks. If you integrate this comment into your answer, I'll accept your answer.


text:
Why does CLIP use a decoder-only transformer for encoding text?
In CLIP [1], the authors train a model to learn multi-modal (text, vision) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders.
For the text encoder, the authors choose to use a variant of GPT2 which is a decoder-only transformer, taking the activations of the highest layer of the transformer at the [EOS] token the feature representation of the text (emphasis mine):

The text encoder is a Transformer (Vaswani et al., 2017) with the architecture modifications described in Radford et al. (2019). As a base size we use a 63M-parameter 12- layer 512-wide model with 8 attention heads. The trans- former operates on a lower-cased byte pair encoding (BPE) representation of the text with a 49,152 vocab size (Sennrich et al., 2015). For computational efficiency, the max sequence length was capped at 76. The text sequence is bracketed with [SOS] and [EOS] tokens and the activations of the highest layer of the transformer at the [EOS] token are treated as the feature representation of the text which is layer normalized and then linearly projected into the multi-modal embedding space.

I found this pretty weird considering that they could have used an encoder (a-la BERT) which to me seem more fitted to act as encoders than decoders. Perhaps they wanted to enable generative text capabilities, but they could've achieved that with an encoder-decoder architecture (a-la T5) too.
I was expecting ablations on the text-encoder architecture, motivating their choices, but found none. Any clue why they made these choices?
References:
[1] A. Radford et al., ‘Learning Transferable Visual Models From Natural Language Supervision’, in Proceedings of the 38th International Conference on Machine Learning, Jul. 2021, pp. 8748–8763. Accessed: Feb. 07, 2023. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html

I believe because Decoder-only basically cuts down the model size in half, and has also shown empirically to be better.
In the original Transformer paper, the evaluation task was about Machine Translation, which at that time, encoder-decoder architecture was very successful.
This paper is probably the first to propose the decoder-only Transformer, in which they observe the following improvements:

It removes the encoder, which means half the parameters and hyperparameters.
It helps them with long input sentences
The inputs of the Encoder and Decoder are the same, so basically it is probably redundant

Later paper also finds out that decoder-only works better than encoder-decoder part. One thing to note about this is Encoder is Bi-directional while Decoder is Uni-directional. This nature fits with GPT-2, which is an autoregressive language model.
Related:

This paper may be related. It studies different decoder-encoder variants and finds that decoder-only performs better.
A StackExchange question regarding the decoder in BERT.

I think one potential explanation for their text-encoding method choice is that they probably realised that an image will be typically captioned with more than one word (which on its own can be multiple tokens), so they need sentence-embeddings of the text, not token embeddings which is what you get with BERT. Basically, image::sentence.
You notice this when using the CLIP encoder in code - it gives you sentence embeddings, not token embeddings.
Because they use a decoder-only transformer, which is autoregressive, they can take the feature representations at the EOS token as sentence representations of the sentence that preceded the EOS token.
Had they used an encoder, they would've had think about some aggregation method to go from token embeddings to sentence embeddings (Although, to be fair, this may have been as simple as using the CLS token representation).


text:
Why are AI Safety discussions almost always from the perspective of reinforcement learning?
I have been reading some articles on AI safety and they almost always speak of AI Safety from the reinforcement learning (RL) perspective, i.e. where we have some artificially intelligent agent acting in an environment so to maximise some reward.
Is there a reason why the focus is very often from this RL POV?
Thanks.
Examples:

2021 AI Alignment Literature Review and Charity Comparison
Unsolved Problems in ML Safety
Paperclip Maximizer
AI Alignment Research Overview by Jacob Steinhardt

I think what you are really asking is why don't we talk about alignment from the perspective of general deep learning?
In fact, we do talk about alignment for general AI system that's being used in production. For example, making large language model less toxic is an active
area of research.
The reason you see a lot of references to RL in the context alignment is that one method that has been actively researched to make our AI systems aligned with our preference is through training them via RL. The reward function is, not surprisingly, good alignment with human preferences.


text:
Deep Learning for occlusion recognition is 2D or 3D space
Given a dataset of spatial 2D or 3D object map with their bounding box annotations,

How feasible would it be to train a deep learning model to recognize (classify) "occluded" objects from a 2D or 3D "source" coordinate?
What model/learning architecture would best work for such a task?

Example: in this top-view 2D image, two objects are classified "occluded". Similarly this could be the case in 3D space.

What you are looking for are adversial neural networks. GANs or other adversial neural networks are used to classify or detect objects that are partially occluded. They can perform very well given enough training data and tend be quite robust because they train for/in a 'hostile' environment where occlusion and other effects undermine the quality of the data.


text:
How can I produce crossbred images out of two datasets?
I'm very new to AI and deep learning. So my question is going to be very basic.
I'm trying to understand which approach I would need to use to cross-breed set of images. Let's say I'm having dataset of cats and dataset of human faces and as output I want to get images that would look like cat-humans.
I tried to use DCGAN network, this in particular: https://github.com/pytorch/examples/tree/main/dcgan. But as output I'm getting an image that is baed only on one of the datasets (either human or cat, not anything that would look like cross-breed).
I suppose, I completely misunderstand the concept of DCGAN, or perhaps I just need to tweak the network to achieve desired results?
Anyway, I'm kinda lost here and just asking for directions. Perhaps there's better network architecture for this purpose or some specific set up needed.


text:
A Feedforward Neural Network (FNN) implemented with RMSProp optimization is exhibiting a tendency to overclassify instances into one particular class
I'm coding an FNN in Rust using the nalgebra crate. I coded the backpropagation based on this article from Brilliant (the link directly highlights the formulas' section I).
The issue
My network tends to overclassify one class. See logs of some iterations:
Pred: [0.4957141833444244, 0.5042858166555757] Exp: [1.0, 0.0]
 Pred: [0.1159099292782023, 0.8840900707217978] Exp: [0.0, 1.0]
 Pred: [0.49812391110550713, 0.5018760888944928] Exp: [0.0, 1.0]
 Pred: [0.785823559592914, 0.21417644040708617] Exp: [1.0, 0.0]
(0) Loss: 1.7553771084566945  accuracy: 75%
 Pred: [0.921104553204082, 0.07889544679591799] Exp: [1.0, 0.0]
 Pred: [0.43669346100767914, 0.5633065389923209] Exp: [0.0, 1.0]
 Pred: [0.8390020708741205, 0.16099792912587946] Exp: [0.0, 1.0]
 Pred: [0.9358457093202687, 0.0641542906797313] Exp: [1.0, 0.0]
(1) Loss: 2.5487814857735516  accuracy: 75%
 Pred: [0.9938891101436632, 0.0061108898563368074] Exp: [1.0, 0.0]
 Pred: [0.8117580010095468, 0.18824199899045324] Exp: [0.0, 1.0]
 Pred: [0.9637103688436267, 0.03628963115637338] Exp: [0.0, 1.0]
 Pred: [0.9836993037515837, 0.01630069624841633] Exp: [1.0, 0.0]
(2) Loss: 5.008814788847665  accuracy: 50%
 Pred: [0.9995224418411457, 0.0004775581588542201] Exp: [1.0, 0.0]
 Pred: [0.9640882324687131, 0.03591176753128681] Exp: [0.0, 1.0]
 Pred: [0.9940401356203591, 0.005959864379640862] Exp: [0.0, 1.0]
 Pred: [0.9967080323177023, 0.0032919676822977125] Exp: [1.0, 0.0]
(3) Loss: 8.453172874424054  accuracy: 50%
 Pred: [0.9999695301335619, 3.0469866438138036e-5] Exp: [1.0, 0.0]
 Pred: [0.9955011694391566, 0.004498830560843456] Exp: [0.0, 1.0]
 Pred: [0.9993378187078414, 0.0006621812921585512] Exp: [0.0, 1.0]
 Pred: [0.9995329830642827, 0.00046701693571723986] Exp: [1.0, 0.0]
(4) Loss: 12.724406571994546  accuracy: 50%
 Pred: [0.9999986403638063, 1.3596361937582465e-6] Exp: [1.0, 0.0]
 Pred: [0.9996152538707804, 0.0003847461292195687] Exp: [0.0, 1.0]
 Pred: [0.9999496312441125, 5.036875588738215e-5] Exp: [0.0, 1.0]
 Pred: [0.9999544427172501, 4.5557282750020274e-5] Exp: [1.0, 0.0]
(5) Loss: 17.759113261749448  accuracy: 50%
 Pred: [0.9999999567632185, 4.323678145397258e-8] Exp: [1.0, 0.0]
 Pred: [0.9999768122330864, 2.3187766913582424e-5] Exp: [0.0, 1.0]
 Pred: [0.9999973081629956, 2.6918370044323987e-6] Exp: [0.0, 1.0]
 Pred: [0.9999969459673318, 3.054032668285032e-6] Exp: [1.0, 0.0]
(6) Loss: 23.497175499984603  accuracy: 50%

This also happens when I have more classes, using other datasets (like the MNIST hand-digits).
My implementation
Based on the Brilliant article I coded a matrix based implementation of the Backpropagation, i.e. all calculations are made matrix-wise. I do not see any problems with my implementation. I also coded the RMSProp which seems to be fine since when I remove it and only use SGD to train the networks the same issue happens.
If you want to take a look at the whole project, I created a branch, which is a copy of the current main branch, that I'll never delete for the sake of this question.
What I tried

Checked every activation and loss function I have coded (see my functions module code). I'm using sigmoid for the hidden layer and softmax for the output layer. But even if I use relu for the hidden layer the issue is the same.
Applied the math by hand, it seems to be check (I'm not an expert at math at all).
Remove the RMSProp, just used SGD, same issue the network doesn't learn.
Use the Relu activation function for the hidden layer.

Question

In theory, what could cause this?

Hi @Evry and welcome to AI Stack Exchange! Thank you so much for detailing what you have tried. This is an excellent question, and we hope to see more of your questions on this site soon!

How are you initializing the weights of each layer? It turns out that weight initialization plays a surprisingly large part in the network's ability to learn, but these issues are abstracted away in libraries torch. I wonder if that could be an issue here. See this paper

This was the issue. I found out 14 hours ago when heading home. Got myself thinking about the weights initialization. Some time along the way I changed the weight initialization for some technique I do not recall the name of. It probably had some bug or wasn't appropriate. Just initialized between -0.5 and 0.5. But why does the initialization have so much impact on learning?

Initializing the weights between -0.5 and 0.5 works fine

Poor initialization can cause exploding or vanishing gradients - it's possible in your case that your network simply didn't learn anything because your gradient norms were too small. The reason I'm guessing that is because your network predicted only one class, which is typically what happens with randomly initialized networks

I see. Good to know that. Thanks for your time.

Of course - honestly I am not an expert on the dynamics of model initialization. But a good paper is the Kaiming one I linked (that was huge for getting models to learn consistently) as well as Xavier init.

That is a good recommendation, I'll take a deeper look at the paper, thanks

So, after some time I changed the weights initialization for the one you suggested. You were right, it was the initialization. Thank you very much


text:
Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)
I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next state? e.g. next state is independent to action $$P(s_{t+1} | s_{t}, a_{t}) = P(s_{t+1} | s_{t})$$
I think it is doable as it won't change the derivation of policy gradient, e.g.
$$
\begin{aligned}
\nabla \log \mathrm{p}_{\theta}(\tau) &=\nabla\left(\log \mathrm{p}\left(\mathrm{s}_{1}\right)+\sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)+\sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}\left(\mathrm{s}_{\mathrm{t}+1} \mid \mathrm{s}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right)\right) \\
&=\nabla \log \mathrm{p}\left(\mathrm{s}_{1}\right)+\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)+\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}\left(\mathrm{s}_{\mathrm{t}+1} \mid \mathrm{s}_{\mathrm{t}}, \mathrm{a}_{\mathrm{t}}\right) \\
&=\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right) \\
&=\sum_{\mathrm{t}=1}^{\mathrm{T}} \nabla \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)
\end{aligned}
$$
$$
\begin{aligned}
\nabla \log \mathrm{p}_{\theta}(\tau) &=\nabla\left(\log \mathrm{p}\left(\mathrm{s}_{1}\right)+\sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)+\sum_{\mathrm{t}=1}^{\mathrm{T}} \log \color{red}{\mathrm{p}\left(\mathrm{s}_{\mathrm{t}+1} \mid \mathrm{s}_{\mathrm{t}}\right) }  \right) \\
&=\nabla \log \mathrm{p}\left(\mathrm{s}_{1}\right)+\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)+\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \color{red}{\mathrm{p}\left(\mathrm{s}_{\mathrm{t}+1} \mid \mathrm{s}_{\mathrm{t}}\right) }\\
&=\nabla \sum_{\mathrm{t}=1}^{\mathrm{T}} \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right) \\
&=\sum_{\mathrm{t}=1}^{\mathrm{T}} \nabla \log \mathrm{p}_{\theta}\left(\mathrm{a}_{\mathrm{t}} \mid \mathrm{s}_{\mathrm{t}}\right)
\end{aligned}
$$
Also, I am curious about the difference between the RL setting where the next state is independent to action $$P(s_{t+1} | s_{t}, a_{t}) = P(s_{t+1} | s_{t})$$ and the multi-armed bandits setting. If the problem is the next state variable is independent of actions, what would be the correct framework to start with?


text:
What is the formula for $VC(H)$ in Computational Learning Theory?
Courtesy link: How would you intuitively but rigorously explain what the VC dimension is?

Trying to explain the idea of VC to some of my colleagues I've discovered quite an intuitive way of laying out the basic idea. Without going through lots of math and notation as I've done in my other answer.
Imagine a following game between two players $\alpha$ and $\beta$ :

First, player $\alpha$ plots $d=4$ points on a piece of paper. She may place the points however she likes.
Next, player $\beta$ marks several of the drawn points.
Finally, player $\alpha$ should draw a circle such that all the marked points are inside the circle, and all the unmarked points - outside. (Points on the boundary considered "inside".)

The player $\alpha$ wins if she can draw such a circle at step #3. The player $\beta$ wins if making such circle is impossible.
If you try to analyze this game then you'll notice that the player $\beta$ has a winning strategy. For any $d=4$ points on a plane, there is always a subset such that player $\alpha$ is unable to draw a required circle. ( I don't want to go into the detailed proof of the strategy - it is straightforward, but cumbersome - I've sketched it my other answer). If we now change the number of points to $d=3$ then the game suddenly becomes winnable by player $\alpha$ - for three points that are not on the same line any subset can be separated by a circle.
The largest number $d$ at which the game is winnable by player $\alpha$ is called the VC dimension of our classification set. So, in the case of 2D discs (insides of a circle) the VC dimension is 3. If one changes the rules to use rectangles instead of circles, then the maximum number of points winnable by $\alpha$ would be 4 - thus, the VC dimension of rectangular classification sets is 4.
Restoring our the mathematical notation, we denote $\mathcal{X} = \mathbb{R}^2$ our two-dimensional plane. $C$ is the subset of cardinality $|C| = d$ that the player $\alpha$ selects. And $\mathcal{H}$ is a class (a "set-of-sets") of the subsets of $\mathcal{X}$ that one should use as a classification boundary.  > Formally, the statement that the game above is winnable by $\alpha$  can be written as:
$$ \exists\;C\subset\mathcal{X}.|C|=d \Rightarrow\forall\; A\subset C\; \exists\; B\in\mathcal{H}\;. A = B\,\cap\,A  $$
The maximal $d$ at which this statement is true would be the VC dimension. (I actually worked backwards from noticing the alternating quantifiers $\exists\,\forall\,\exists$ in the VC definition - which is typical in game playing, so I worked back from the definition to make the game above.)

This chapter discusses frameworks for analyzing learning algorithms, such as the Probably Approximately Correct (PAC) and Vapnik–Chervonenkis (VC) theories, with relevance to deep learning.
How can I look at the VC-Dimension like a measure of capacity?

How is this question different from the linked question (which I had asked a few years ago)?

You can look at the VC-Dimension like a measure of capacity (like richness, expressive power, flexibility, complexity) of the hypotesis space $H$ of a model: with it, we can extend the PAC Analysis to problems having infinite hypotesis space.
Intuitively, you can think about $VC(H)$ as the cardinality of the largest set of points (your instance space $X$) the algorithm itself can shatter. This special ability the algorithm has is strictly connected to the concept of inductive bias embedded in $H$. That's the neat part: why?
Imagine having a bias-free learner, i.e. a learner that makes no real prior assumptions about the target concept: obviously there won't be any rationale behind its classification of new, unseen instances.
Injecting the inductive bias, we are effectively converting our learner into a deductive system, one thatjumps to logical conclusions entailed by the premises of our training examples from the assumed hypotesis in H. Now the learner can define a new policy for generalising beyond observed data in a non-procedural way starting from what he learnt about the training examples.
Now: an unbiased $H$ is one that shatter the instance space $X$, but we want a biased $H$, so we are almost certainly losing the power to shatter the whole instance space $X$.
But what about a $Q \subset X$? Moreover, given that the learner can shatter this subset $Q$ of $X$, what does it can tell us about $H$? Surely we can say that the expressiviness of $H$ is directly proportional to the size of $Q$, i.e. how large $Q$ is. How do we measure this expressiviness?
With VC-Dimension. That's it: $VC(H)$ is, for $H$ defined over $X$, the size of the largest finite subset $Q \subset X$  that can be shattered by $H$.
Two details:

If the size of $Q$ is arbitrarily large and still $H$ can shatter it, then $VC(H) = \infty$
For any finite $H$, $VC(H) \leq \log_2(|H|)$, clearly because $H$ needs $2^n$ different hypoteses to shatter $n$ instances if $VC(H) = n$, but it's not an upper bound: matter of fact, you can have $|H| \geq 2^n$, that's why $log_2(|H|) \geq VC(H) = n$.