text:
How can I improve this toy Graph Neural Network Generative Language model
Background
I'm an undergraduate student with research interests in a field of physics that has significant overlap with graph theory, and a functioning knowledge of how simple neural nets work and how to build them with TensorFlow and Keras. As many people are, I'm fascinated by the recent advancements in transformer-based language models, and I've spent the last several weeks reading up on them in an attempt to construct my own simple "mini GPT". In doing so I encountered Graph Neural Networks, and decided I'd try instead to construct a generative language model out of these, inspired by the fact that graphs are (in a very hand-wavy sense) perhaps inherently more amenable to encoding relationships like "attention", etc. I'm aware that the task I'm trying to accomplish could probably be much more easily achieved using alternative architectures. This is mostly just a fun "what if" project.

Code
I still haven't quite wrapped my head around all the details of how and what a Graph Neural Net learns, nor what the different types of GNN layers do. Nevertheless, I've constructed a simple GNN using the Spektral library, which takes an input string, and predicts the rest of the string word-by-word by predicting the most probable next token. Here's what I have so far:
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Embedding
from tensorflow.keras.preprocessing.sequence import pad_sequences

from spektral.layers import GCNConv
from spektral.utils import normalized_adjacency

import numpy as np

import random
import os

def get_training_data(training_data_dir):
        filenames = []

        for filename in os.listdir(training_data_dir):
                filenames.append(os.path.join(training_data_dir, filename))

        random.shuffle(filenames)

        lines = []

        for filename in filenames:
                with open(filename, "r") as file:
                        for line in file:
                                lines.append(line.strip())

        return lines

# Import data
training_corpus = get_training_data("./training_data")
tokens = [line.split() for line in training_corpus]
vocab = set(token for line in tokens for token in line)

vocab_size = len(vocab)

print("Vocabulary size: ", vocab_size, " words")

# Tokenize
word_to_idx = {word: idx for idx, word in enumerate(vocab)}
idx_to_word = {idx: word for word, idx in word_to_idx.items()}

# Pad and truncate sequences to a fixed length
train_data = [[word_to_idx[token] for token in line] for line in tokens]
train_data_padded = pad_sequences(train_data, maxlen=vocab_size, padding='pre', truncating='pre')

# Shift train_data_padded to create train_labels
train_labels = np.roll(train_data_padded, -1, axis=1)
train_labels[train_labels >= vocab_size] = 0

train_data_padded = np.array(train_data_padded)
train_labels = np.array(train_labels)

# Construct token-to-token similarity matrix
similarity_matrix = np.zeros((vocab_size, vocab_size))

for sentence in tokens:
    for i, token1 in enumerate(sentence):
        for j, token2 in enumerate(sentence):
            if i != j:
                num_words_between = pow(abs(j - i), 2)
                similarity_matrix[word_to_idx[token1], word_to_idx[token2]] += num_words_between

adjacency_matrix = normalized_adjacency(similarity_matrix)

# Construct model
input_layer = Input(shape=(None,))
embedding = Embedding(input_dim=vocab_size, output_dim=vocab_size)(input_layer)
gcn_layer = GCNConv(vocab_size)([embedding, adjacency_matrix])

output_layer = Dense(vocab_size, activation='softmax')(gcn_layer)

model = tf.keras.Model(inputs=input_layer, outputs=output_layer)
model.compile(optimizer='adam', loss='categorical_crossentropy')

# Training loop
num_epochs = 100

model.fit(train_data_padded, tf.keras.utils.to_categorical(train_labels, num_classes=vocab_size), epochs=num_epochs)

padding_token = "<PAD>"

def respond_to_text(initial_string):
  initial_tokens = initial_string.split()

  for i in range(len(initial_tokens)):
    if initial_tokens[i] not in word_to_idx:
        initial_tokens[i] = "<PAD>"

  while len(initial_tokens) < vocab_size:
      initial_tokens.insert(0, padding_token)

  generated_tokens = [word_to_idx[token] for token in initial_tokens]
  max_generation_length = 40

  # Number of tokens from the initial string that have been used
  initial_tokens_used = len(initial_tokens)

  for _ in range(max_generation_length):
      current_seq = np.array([generated_tokens[-vocab_size:]])  # Always use the last vocab_size tokens
      next_token_probs = model.predict(current_seq)[0][-1]
      next_token = np.random.choice(np.arange(vocab_size), p=next_token_probs)
      generated_tokens.append(next_token)

      # If there are more initial tokens to use, do that
      if initial_tokens_used < len(initial_tokens):
          generated_tokens[-vocab_size:] = [word_to_idx[token] for token in initial_tokens[initial_tokens_used:]]
          initial_tokens_used = len(initial_tokens)

  # Generate text
  generated_text = [idx_to_word[idx] for idx in generated_tokens]

  # Remove trailing "<PAD>" tokens
  generated_text = [token for token in generated_text if token != padding_token]

  # Join the tokens into text
  generated_text = " ".join(generated_text)
  print("Generated Text:", generated_text)

while True:
  input_string = input()
  respond_to_text(input_string)

It's relatively simple, thus far. I train the network on strings of text from a subset of the WikiQA corpus, which I pre-process by removing all punctuation and capitalization. I define the elements of the adjacency matrix to be the squared distance between tokens (and 0 between a token and itself). I'm using GCNConv(), admittedly without knowing the intimate details of how it is different from other options provided by Spektral. Since the number of tokens in the input string must be identical to the vocabulary size, I pre-pend it with "<PAD>" tokens, and only pass in the last vocab_size tokens each time I generate a new token. I deal with unknown tokens by replacing them with "<PAD>".
If I understand correctly, the GNN learns the "strengths" of connections between nodes (words), i.e. the edge weights and, as I imagine it, the adjacency matrix encodes a sort of very weak form of "attention".

Question

What and how, precisely, is this learning? I know that in a simple feed-forward neural network, a set of weights "between" perceptrons is learned. Do I understand correctly that what is being learned here are edge weights? How do node features and the adjacency matrix factor into this, and what does the model "do" with some input text? How are the next token probabilities calculated? I only understand this on a very superficial level, sufficient so as to produce this seemingly somewhat functional script. I can see that, when I plot the graph associated with the GCNConv layer using networkx after fitting, words that are similar (e.g. "boat", "ocean", "water") tend to cluster.
How can I improve the model?* I've spent a fair amount of time reading GNN research papers, however they're written largely in what appears to be very subfield-specific jargon that is unfamiliar to me as someone very familiar with graph theory and somewhat familiar with machine learning. I'd like to begin by taking smaller steps, hopefully starting with some suggestions provided by the community here. I have no strict, "objective" criteria in mind, outside of producing more realistic, human-like text.

Here's some example input and output:

Input: "the traffic" -> Output: "the traffic was invaluable sense day playing urban evening together made past weekends board plot off color cookies calm concert flowers express eye-opening learn garden outside satisfying laughter movie waves how's sunrise of try traffic scratch day captivating hobby live blanket delicious"
Input: "i enjoy" -> Output: "i enjoy watching magical journaling awe-inspiring buds tail feeling entertainment resist homemade ones flavors soothing well-being laughter life culture cleanup picnics beauty accomplishment nature mood ocean up satisfying magical contagious joy admiring feeling live marathons beauty views things expression hikes next happiness vacations"

$*$ Outside of the obvious, e.g. more training data, adjusting hyperparameters, etc.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

First of all, I would like to encourage you to keep trying new things, it sounds super fun! There are a few things I would like you to clarify about Graph Neural Networks (GNNs) and Graph Convolutional Neural Networks (GCNs), and then I will answer your questions as best I can. Also, as a sidenote, there are too many questions here, so don't expect me to answer all of them. Given that, I will try to answer the main ones.
Adjacency Matrix and Attention Mechanism
I will address the notion that "the adjacency matrix encodes a sort of very weak form of attention". This notion is kind of correct. In fact, it can be shown that the attention mechanism of the transformer is just an instance of the generalization of the message-passing function of the GNN, where the underlying graph is a complete graph, or in other words, its adjacency matrix is a matrix of all ones. If you continue your research on GNNs, you will see another famous architecture called Graph Attention Neural Networks (GAT). This is just an adaptation of the attention mechanism of the transformer for more simple static graph data. If you want to understand how the transformer is a type of GNN, I can explain it to you later.
First Question: What Exactly is Being Learned?
What is learned in a graph neural network depends on the type of neural network that you are implementing. Generally, there are two types of features that you can input into a GNN (for the forward pass). These are the feature data and the edge data of each node. In the case of the GCNConv layer, this only implements the learning of the feature data (which are the embeddings of each token in your case). So, in your code, the GCNConv is learning the weights that are multiplied with the embeddings. In your code, there are no edge weights that are being trained or even initialized because there is no edge data in the first place.
GCN uses the following formula for calculating the message passing:
$
\mathbf{h}_i^{(l+1)} = \sigma \left( \sum_{j \in \mathcal{N}(i) \cup \{i\}} \mathbf{A}_{ij} \mathbf{h}_j^{(l)} \mathbf{W}^{(l)} \right)
\
$

$h_i(l)$ is the feature vector of node i at layer $l$.

$W(l)$ is the weight matrix for layer $l$.

$N(i)$ is the set of neighbors of node $i$.

$A_{ij}$ is the element at the $(i,j)$ position of the adjacency matrix
$A$. This could be either 1 or 0, indicating whether there is an edge between nodes $i$ and $j$, or it could be a weight if the graph is weighted.

σ is an activation function, such as ReLU or Sigmoid.

I would recommend watching this video, which in my opinion explains GCNs very well if you have any more doubts.
Second Question: How Can the Model be Improved?
I can understand how you are feeling about GNNs. I did research on them, and they can be very niche with some terminology. I would recommend that you understand the basics of the GNN architecture very well and then follow up with other types of GNNs. Graph theory is definitely useful, but in my opinion, it only helps you to understand the basics of graphs, and the other things are much more specific to the field of GNNs.
As for the question, sadly, deep learning is not a very interpretable field. This means that sometimes we don't know why some architectures seem to work better than others. For example, the transformer is currently the best model for modeling language. We think this is because of the attention mechanism, which is able to take into account all tokens in context at the same time. However, it is also a very scalable model that does not suffer from the gradient vanishing effect, unlike recurrent neural networks.
Recently, there has been a new type of RNN called RWKV that solves the initial gradient vanishing effect of RNNs and does not use the attention mechanism. We have RAVEN, which is an example of how we can build models that are just as performant as transformers with RWKV.
What I'm trying to say is that it is very difficult to say what things you can use to improve the accuracy of your model, besides the basics of training with more data and tuning hyperparameters. If I had to try it, I would probably make my GNN resemble the transformer architecture as much as possible, adding attention, residual layers, and so on. However, this is something that you will have to explore and experiment with.
Final Thoughts
I don't intend to discourage you; quite the opposite. However, I should note that GCNs may not be best suited for this task. You might also explore standard NLP techniques for boosting accuracy, such as improved tokenization and data quality. For more ambitious improvements, you could look into Reinforcement Learning through Human Feedback (RLHF) techniques.


text:
How to transform a loss function into a score function?
Loss_Function/Maximize_Function/Score_Function, CustomLoss, pytorch. Using Custom Loss for Maximizing Score in PyTorch
I'm using a PyTorch model with an LSTM input layer, a linear hidden layer, and 3 neurons in the output layer with a softmax activation function.
Instead of using a loss function (such as nn.MSELoss()) and passing the model's prediction and its target (loss = criterion(predictions, target)), I would like to use a custom scoring function that I've created. This scoring function is represented as score = get_score(predictions), which implies a concept similar to maximizing (predictions, score).
However, since I don't possess a target for each input value of the model, the model would be evaluated based on its score (which is a single number). It appears that considering a single value as a basis for assessing a set of predictions across the entire dataset might not be valid.
Is something like this possible? The times I've attempted it, I often encounter an error during the 'loss'.backward() phase, as it fails to trace the necessary gradients for backpropagation.
I didn't put the code because it's a very big code, but I'll make a simple example.
class Model_123(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(10, 3)

    def forward(self, x):
        out = self.linear1(x)
        out = F.softmax(out, dim=1)
        return out

dataset = torch.randn(10, 10)
model = Model_123()

custom_loss_score = CustomLoss_Score()  # This is what I want to figure out
for epoch in range(epochs):

    model.train()
    optimizer.zero_grad()
    predictions = model(dataset)
    score = random.uniform(0, 100)

    # Random score just for demonstration, but it would be calculated based on the model's actions.
    # Using torch.argmax on the predictions made by the model for that input value.
    # This gives me a success rate, where correctly predicting action 0 or action 1 gets +1,
    # while action 2 means doing nothing. In a situation where the model correctly predicts only one action,
    # between actions 0 and 1, and everything else is action 2, leading to a 100% success rate.
    # This scenario is not good as there's no criteria; it just guessed one action and got 100% score.
    # It would be better if the model correctly predicted 80% of actions with a 70% operation rate,
    # which would be a better outcome. So, based on the success rate, I create another function
    # to calculate the score, a relation between the success rate and the number of operations performed.
    # This is a much larger code that I haven't included here.

    loss_score = custom_loss_score(predictions, score)  # Maximize the score
    loss_score.backward()
    optimizer.step()

This scoring mechanism, represented as score, would ideally be maximized by adjusting the model's parameters. However, I'm unsure about how to define the CustomLoss_Score() function that would achieve this.
The score is generated based on the model's actions, and it's calculated using a relationship between the accuracy rate and the number of operations performed. The intention is to maximize this score rather than minimizing a traditional loss function.
The reason I'm not using Reinforcement Learning is that in this case, I would have to loop through all states (or input data), which takes more time. On the other hand, with supervised learning, I can approach it in a vectorized manner. I'm following this strategy because I'm also utilizing training based on genetic algorithms, which involve multiple individuals (sets of model weights) making predictions on the data. If I were to use Reinforcement Learning, each individual would have to loop through the states, and this would be quite time-consuming, especially with a population size of, for example, 300 individuals.
Hence, I'm employing a training approach based on gradient descent, along with training using genetic algorithms and artificial selection.
When the model seems to be stuck in a particular place, I take the weights from that model and generate a population of individuals. Genetic algorithms can guide the weights to places where gradient descent might struggle, and vice versa. When one approach isn't yielding improvements, I switch to the other approach.

Is there a reason that you cannot score individual predictions? There might be a connection to Reinforcement Learning here, which could give you the tools to solve that.

As @NeilSlater pointed out, just check any PyTorch implementation of policy gradient  for example https://github.com/Finspire13/pytorch-policy-gradient-example/blob/eded52d0d4f56b2d6f72394c083dbd70029294fb/pg.py#L130-L131C32

See this answer: https://ai.stackexchange.com/questions/41264/how-to-do-backpropagation-with-argmax/41312#41312; it's a similar setup.

Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.

We weren't suggesting to use RL, we were saying that you can look at the code and take inspiration from it as RL can be considered a Score optimization more than a Loss since you don't have a target

I suspect you could be reinventing the wheel a little bit here, because some of the description of what this task is for appears to match to a Reinforcement Learning (RL) scenario. You may find if you research approaches using neural networks to solve RL problems, that you get a lot of insight into things that could help with your problem.
However, there is also a general answer using the approach from RL Policy Gradients, assuming your NN output layer is a multiclass classifier for which action to take.

You need to store inputs to the NN used to generate actions (let's call them $S_t$), until you have the associated score function results. Let's call the score function value $G$, and it will be the same for consecutive samples of $S_t$ - each group that ends with a known score is an "episode".
Along with those inputs, you need to store the actual action taken when that input was supplied, let's call that $A_t$. This action can be different to the argmax choice from the classifier. You should be sampling actions according to the probabilities from the classifier (a different sampling approach is possible in general, but won't work well with the rest of the instructions here, because we rely on seeing the different scores in correct ratios).
Once you have some results with score functions - ideally from several runs to reduce correlation between inputs - then you can use them to construct a minibatch for supervised learning. Retrospectively set a bunch of $G_t$ to match the $S_t$ and $A_t$ values.
Calculate the "ground truth" for each input $S_t$ as the action that was taken $A_t$. It doesn't matter whether the action was the best or correct one to take, it matters that it was the one that was actually taken, and as above, ideally sampled according to your classifier probabilities.
Calculate the parameter gradient for each input, output combination as normal for your neural network for supervised learning and multiclass cross-entropy loss . . . but
Before you add to the total gradient for the minibatch, multiply the weights gradient by a normalised $G_t$. You can change the normalisation parameters over time - typically you subtract some average $\bar{G}$ seen so far (e.g. running average of results from last 100 episodes) and maybe multiply by some value to put into an easy to handle range, so $\nabla_{\theta, t} \leftarrow \beta(G_t - \bar{G})\nabla_{\theta, t}$. This is the "trick" from policy gradients that means the neural network will learn to avoid low total scores and find high total scores on average across many updates. The offset is not required, but makes the learning process faster and more stable. The multiplication is not required either, provided you are happy to try different learning rates instead. So the absolute bare bones version is $\nabla_{\theta, t} \leftarrow G_t\nabla_{\theta, t}$
Complete the minibatch by taking a positive gradient step on the mean gradient (perhaps multiplied by some learning rate, or modified with an optimiser like Adam)

This is pretty much REINFORCE with baseline, a policy gradient method from RL, as applied to discrete action space and a softmax action classifier. It can in theory be applied in scenarios which are not technically setup as RL problems, as long as you can follow the procedure, and it reliably means something to generate a "good" sequence of output choices based on available inputs, by some measure (your score in this case).

Thank you all for your time, and for your responses. I'll look at the links left, plus I've added more information about why I'm not using reinforcement learning.

@IAQuestions You should be able to vectorise the process described above. The difficult part is vectorising your environment and environmental responses, and typically that's impossible for MDPs. But if you have a way to vectorise assessing agent behaviour in your problem, all of steps 1-7 above are vectorisable. Steps 3-7 are vectorisable regardless and convert your scores to useful gradient.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
RL agent focusses too much on early rewards, even with no discounting
How can I guide my RL agent to solve tasks in the correct order?
I'm trying to train an agent using reinforcement learning, similar to MuZero. The goal is to solve 4 tasks, A/B/C/D. Each task involves two actions, X1/X2. Initially, only action A1 is possible. Performing it enables A2 and B1; B1 enables B1 and C1; C1 enables C2 and D1; D1 enables D2. There are also some other actions that aren't listed here - they all prevent the successful termination of at least one of the tasks.
Crucially, performing any action X2 before D1 is done will permanently disable D1. So the only winning policy is A1>B1>C1>D1>{A2, B2, C2, D2 in any order}
To guide the training, I've defined small rewards for solving each task, and a large one for solving all four. I've set the discounting factor to 1.0, so there should be no downside to not immediately doing some X2 step.
Unfortunately, the agent never learns to do all A1-D1 steps first. It manages to do A1 and B1 first, sometimes even C1, but never all 4. It then completes some task A-C and thus fails to solve task D.
I noticed that my policy network learns to assign a prior probability of ~1 to A1>A2, strongly biasing the MCTS to explore this path. I assume the reason is that by random exploration the agent will often do A1>A2>... and get a reward. But A1>B1>C1>D1>A2 is way less likely. So the network is well-trained in predicting the reward for A1>A2, but less well-trained for the longer game sequences. If it cannot predict the reward there, the MCTS is biased to do A1>A2. This causes the policy network to also learn that this action should have a high prior probability, further biasing the MCTS, creating a feedback loop.
How can this be prevented? Would it be enough to increase the temperature? If so, should I only increase the temperature for sampling from the distribution computed by the MCTS, or should I also increase the $c_1$ in the UCB formula? That should increase the exploring within MCTS, and reduce the focus on the rewards.
I guess I could also remove the intermediate rewards, but then my signal will be extremely sparse.

I know that I could enforce the order of tasks using the environment, but I want the make the agent learn this by itself.

This seems like a simple toy task. You say that "There are also some other actions that aren't listed here" - what is the ratio of number of possible actions to the 8 you describe in more detail?

Another important thing to clarify - what is your state representation?

@NeilSlater There are 26 actions. The given 8 actions are the "good" ones, all other 18 actions will prevent one of the tasks from succeeding. Not all actions are available in every state, I'm already masking them. But I made sure that the correct sequence of actions is possible

OK, cool. So what is the state representation that the agent uses to calculate the policy? i.e. the input into your policy method

@NeilSlater It's a very simplified version of the AlpheDev setup: The state representation is a list of all previous actions (what values were moved from which memory locations to which other locations) and a snapshot of the current memory. The representation network has a transformer architecture to process the sequence of instructions, the resulting representation is combined with a MLP-processed embedding of the memory values

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why does training a NN using MCTS work even if the number of simulations isn't much larger than the number of actions?
tl;dr If the visit rates of children generated by MCTS is biased because not enough samples were taken, why doesn't the network learn random behavior?
My understanding of combining MCTS and NNs (e.g. for MuZero) is that we start with an untrained network, and then play games. For each new game, we pick the next move by doing a MCTS:

Starting from the current position, we (randomly) traverse the current search tree
Once we reach a leaf node the current NN is evaluated to get a value and policy
The node is expanded. The policy is used to initialize the likelihood of picking the respective possible next moves
The value is propagated upwards, so paths that lead to large values are visited more often during this MCTS
Restart from step 1, for a total of N simulations

Once this is done, we sample the next move by looking at the children of the root node. The more often it has been visited, the more likely the respective action is to be chosen.
We also save the current game state and the distribution of visits to our replay buffer.
During training, a random game state is picked from the replay buffer. The networks policy head is trained to return the empirical distribution generated by the MCTS: Children that were visited more often should have a higher prior.
But I'm not sure why this algorithm succeeds, given the apparently low number of MCTS iterations compared to the action space.
According to their pseudo code and paper, Go has 362 possible actions, and they run 800 simulations (MCTS iterations) per generated move. So for the initial network which was untrained, we would expect that every child has approximately the same prior (they even refer to a UniformNetwork in their code). Therefore, during the initial games, the maximum depth reached will be about 3. This is not deep enough to discern good moves from bad ones, so the value head will not receive useful training signals.
Say, due to randomness, action 1 is chosen twice, but action 2 is not chosen at all. Then, the network will learn this distribution, even though there's no reason behind it.
Once it has learned this, future plays will rarely visit action 2, further enforcing this.
The only explanation I could see is that the number of games played before any training starts is huge - some of them will favor action 1, some will favor action 2. On average, this may cancel out. Is this correct? Or is there another mechanism at work?
If the reason this works is the averaging, is there some literature on how many games are necessary to benefit from this effect?

Your understanding of AlphaZero-style MCTS seems correct, except that tree traversal does not happen randomly! It uses the PUCT formula to pick the child node. Similarly the policy is not simply the probability of picking a certain child, it's just a term in the PUCT formula. In fact the tree search is fully deterministic, except for the additional Dirichlet noise injected at the root node. Your summary of the training process is correct.
My answer here covers the "true signals" the (Alpha/Mu)Zero training process gets, and how training can initially get started even with a random or uniform network. This should already provide some intuition for why all of this works!
Some additional responses to your questions, not in order:

Say, due to randomness, action 1 is chosen twice, but action 2 is not chosen at all. Then, the network will learn this distribution, even though there's no reason behind it.

Like I said, this randomness can only come from the Dirichlet noise. But it will tend to average out over many positions. The NN will not really learn specific details caused by noise, it will only learn the average that remains, which is a lot better.

The only explanation I could see is that the number of games played before any training starts is huge - some of them will favor action 1, some will favor action 2. On average, this may cancel out. Is this correct? Or is there another mechanism at work?

You partially got it right (things average out) but this does not require that many games are played before training starts, this averaging out can also happen over time while the network keeps training. More importantly, there are always the true signals I linked to earlier to provide useful learning information that's able to overcome the noise.

Therefore, during the initial games, the maximum depth reached will be about 3. This is not deep enough to discern good moves from bad ones, so the value head will not receive useful training signals.

Yes, very few nodes are searched during selfplay (typically 200-2000). Two things to keep in mind though:

The NN learns to approximate the result of tree search. This is very important and the core idea behind AlphaZero! This means that each NN evaluation can almost be seen as its own mini tree search, greatly extending the "real" depth of the search that happens.
Though the policy term in PUCT, the tree search is very selective, meaning it explores some branches deeply and other only very shallowly. Even with limited visits, the tree search commonly reaches depths of 20-30 for some moves.


text:
What strategy does ChatGPT use to manage its context in very lengthy conversations?
I'm asking specifically about ChatGPT4, but the question could apply to either that or 3.5.
When you use the ChatGPT API, it's of course up to you to manage conversation history and include that in successive API calls within available context length in whatever manner you choose.
In the case of the web interface, they've obviously implemented some system to manage conversation history in context. It clearly doesn't "remember" the entire thing once the conversation gets very long, because it doesn't have infinite context length. So, what strategy does it use to send conversation history to the model once it's exceeded its context length? Does it truncate all content prior to the max context length? Does it summarize earlier parts of conversations to more efficiently fit them within the context? Does it do some dynamic strategy combining many inputs?
Or is this just another case where we just don't know, and OpenAI is being tight-lipped about what it's actually doing?

well, either it uses some "last N tokens", or it might just ask itself to also summarize the history, and the next time to summarize the summarization+the new question, and so on

@Alberto is there a source for this? Transformers can be finetuned to handle longer contexts (e.g., see https://lmsys.org/blog/2023-06-29-longchat/), so compressing the input tokens doesn't seem absolutely necessary to me.

@AlexanderWan it's not a matter of architecture, it's a matter of efficiency (price-wise)... no, "it might just ask itself" it was just a personal thought, but you can see something like this implemented in RMT


text:
How to add engineered features to an image segmentation model
I have built a U-net model for image segmentation of 3-channel remote sensing images. I have a total of four classes; two of these classes look very similar and are hard to distinguish in the images without extra context. In this case, extra context would be the distance to something (i.e., distinguishing very similar looking buildings based on their proximity to a main road).
To add more context I've created an extra channel of values representing the distance to the nearest road, to try to improve segmentation results, but haven't got the results I expected. What I'm currently doing is using the RGB channels as normal in the U-net, then concatenating the 4th 'distance' channel before the final convolution layer, like so:
def conv_block(input, num_filters):
  x = Conv2D(num_filters, 3, dilation_rate = 3, padding="same")(input)
  x = BatchNormalization()(x)
  x = Activation("relu")(x)
  return x

def decoder(inputs, skip_features, num_filters):
  up = Conv2DTranspose(num_filters, (2,2), strides=2, padding='same')(inputs)
  x = conv_block(up, num_filters)
  return x

  inputs = Input((size,size,4))

  x = Rescaling(1/255.0)(inputs)

  sliced_input_1 = x[:, :, :, :3]

  '''pretrained encoder'''
  xception = Xception(include_top = False, weights = 'imagenet', input_tensor = sliced_input_1)
  xception.trainable = False
  s1 = xception.get_layer("input_1").output ##512

  s2 = xception.get_layer("block1_conv1_act").output
  s2 = ZeroPadding2D(( (1, 0), (1, 0) ))(s2)  ##256

  s3 = xception.get_layer("block3_sepconv2_bn").output
  s3 = ZeroPadding2D(( (1, 0), (1, 0) ))(s3)  ##128

  s4 = xception.get_layer("block4_sepconv2_bn").output ##64
  '''bridge'''
  b1 = xception.get_layer("block13_sepconv2_bn").output ##32

  '''decoder'''
  d1 = decoder(b1, s4, int(size/1))
  d2 = decoder(d1, s3, int(size/2))
  d3 = decoder(d2, s2, int(size/4))
  d4 = decoder(d3, s1, int(size/8))
  '''outputs'''

  sliced_input2 = tf.expand_dims(x[:, :, :, 3], axis=-1)

  outputs = Concatenate()([sliced_input2, d4])

  outputs = Conv2D(4, (1,1), padding='same', activation='softmax')(outputs)

  model = Model(inputs, outputs)
  model.summary()

The shape of the tensor before the final convolution is (512,512,65), which is the previous 64 filters from the RGB U-net and a concatenated 'distance' layer, but should the number of filters from the U-net be reduced even more prior to concatenation? Since it seems like the 'distance' layer is still being ignored.

Adding distance information before the final 1x1 convolutional layer implies that that distance information can be used to linearly separate the two similar classes, given the similar visual features.
If that is the case, then your implementation should work: from my understanding of your task, given an image with class A and class B, it could be that the visual features (d4) for the class A image and class B are similar, but the distance values are different enough to classify them.
You can check to see if this is the case by finding failure modes when the model isn't given the distance information and is struggling to distinguish between the two classes, then seeing if the distance values are different enough in those cases to distinguish the images linearly.
If the relationship between distance and the classes is more complex than that, then you can try adding more layers after your concatenation or adding the distance information earlier in the model


text:
What kind of observation state would you give for that environment?
I'm making a new environment where I have two sphere (one above the other) in a 2D plan.

I would like some advice on what observation state I should give to my RL.
Today I have given the following:

Position of A (X)
Angular velocity of A
Position of B (X and Y)
Angular velocity of B
Angle between A and B (relative to A)

This gives in total 6 different state. The goal is to keep the sphere B as long as possible above A.
I have thought a lot about the observation state and I don't know if the position of B should be relative to the position of A ? or should it be relative to the world ?
Should I remove or add other state ? WDYT ?
Edit
I forgot to mentioned crucial information on my environment:
Both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment.

Provided the discs do not have any properties that vary over their surfaces, and you are not trying to generalise to different sizes/weights of discs for a single agent, then your state variables seem like a complete description that you could use to predict the dynamics of the combined system and consequences of simple actions (presumably your actions will be torque on either A or B in order to maintain balance). However, I think it can be simplified without losing any information for a value or policy function.
Assuming the discs always remain in contact, I would recommend you don't track B's position in X,Y - its angular position with respect to A's centre, and A's X position define everything you need to know.
Asuming you have a simple, perfect friction version with no slipping, but constraints on position of A (similar to cartpole) then I think you have four variables:

Angular velocity of A ($\omega_a$ rotation speed)
Angular velocity of B ($\omega_b$ rotation speed)
Angular position of B's contact with A ($\theta_c$ rotation offset, measured from centre of A)*
Angular position of A with respect to a start position ($\theta_a$ rotation offset) OR position of A's centre with respect to a start position ($x_a$ distance offset). These are in a simple ratio $x_a = r_a\theta_a$ where $r_a$ is A's radius, so you can use either

You can drop this to 3 variables if there is no constraint on the position of disc A, because its position is not required to predict the dynamics for balancing B on top of A.
The physics details you added to the question don't appear to change any of the above. If you want to train an agent so that it can generalise to different values in the same trained model, then you would add the params you want to vary to the state. Plus you would need to train with different values so the approximator can learn how they impact action choices.

There is another possibility, that I have assumed you are not considering. If A and B's surfaces are allowed to slip - i.e. there is not perfect friction between them, then you potentially have another free variable in that B's motion around the surface of A is not simply determined by combination of A and B's angular velocity. Whether or not there is another variable needed will depend on how you calculate this slippage. Simple friction models will not need another variable, but more complex ones could require state to track whether slip is occurring.
* I put a $c$ suffix on this to differentiate it from the natural reading of $\theta_b$ which would be B's rotation offset around its own centre.

Hi Neil! Thanks for your answer (again)! Reading your answer, I found that I missed to include crucial information about the environment and I will edit my question to include them. In fact, both disc have different weight and size (as on the picture) and physics is also enabled. "A" has a weight of 1.5 Kg and "B" 0.5Kg. There is a linear damping on both disc of 0.01. It's a constant that can be adjusted in my environment. I guess all the information I just gave modify a lot the observable state... ? Also why "A" position is necessary ? What if -inf < X < inf ?

If there are no constraints on A position, then it is not necessary state for the RL agent to predict value, so you could drop it. I don't think any of the environment parameters that you commented impact the RL state unless you want to train the agent so that it can operate in variations of the environment without re-training. If you train and operate the agent with the same parameters, then those params should not change anything. I don't see any physics that would add more degrees of freedom in the state, but that is possible, and I mentioned friction and slipping models as possibility.

Ok perfect, thanks for all the info. It really helped :) It's a shame that we cannot DM. Your knowledge on RL would be much appreciated on my project! Thanks again

@CyDevos I'm always happy to answer questions on the site, but I cannot realistically make time to DM and consult on projects, or commit to support a project over time. Here, I think some physics knowledge combined with RL is required to spot the differences between your simulator state and RL state, but lots of experts on this site should have both.

Yeah, I know, thanks for your time and devotion!

Hi Neil, I have one more question. I have put a reward of +1 when the angle is > -15 degree and < 15 degree. This reward only apply if both disc are touching. Do you think it makes sense ?

@CyDevos Sure that makes sense for the balancing task, effectively the return is how long the agent can maintain balance. By the way, I assumed in my answer that an episode would end if the discs stopped touching - similar to cartpole, the game would give up if you've obviously lost control. That's a guess on my part - if I'm wrong about that, the answer could also be wrong

thanks! The game end if “B” fall on the ground and a bad reward is given

as promise, just to let you know that I have successfully make my ball A keep the ball B on top of itself!


text:
Is my PPO agent learning? or is it just exploring?
I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src.
My doubts are mainly due to the convergence of entropy and critic network loss.
Over a long training (1M steps), the loss of the critical network grows over time, parallel to the loss of entropy, that it grows too. The growth in entropy is indicative of the agent taking increasingly random actions.
Despite this, the reward definitely improves as episodes pass.
It seems that as the critic's loss and entropy increase, the reward improves accordingly. It is as if as the randomness of the actions taken increases, the reward the agent gets also improves.
This obviously does not make much sense and I would like to understand the reasons for this situation.

The thing that worries me is that it all seems strongly correlated:

entropy grows, critic's loss grows, reward grows
entropy decreases, critic's loss decreases, reward decreases

My fear is that if entropy and loss converge, reward would collapse, at least that is the feeling I get from seeing the training trend.
mi sembra che non stia capendo che sta ottendno una buona policy
it seems to me that he is not understanding that he is getting a good policy.

I know little to nothing about your problem, but from your code it seems that you use NN to predict the parameters of a Dirichlet distribution... however, i'm not really sure about which entropy you are considering... is it the entropy with respect the sampled discrete distribution from the Dirichlet, or the entropy of the Dirichlet itself
The problem is that it's very important that initially the agent uses a uniform distribution (discrete one), so it would be better to have a "biased" Dirichlet, so that uniform distribution are more likely than skewed ones... however, this implies that the entropy of the Dirichlet is very low, and might happens that with time it increases...
Now given this, if you instead start with a uniform Dirichlet, at the beginning of the training you might sample "non-sense" skewed distribution, just because your agent is not calibrated (so the high prob of an action is not due to learning, but to ill-calibration), so entropy is low (for a wrong reason) and then the agent might learn that he prefers more uniform distribution, increasing the entropy
So my takeaway would be to bias the initialization of the NN so that outputs $\alpha$ that prefers uniform distributions than skewed ones

Thanks for your reply.
I think I implemented what you proposed, without solving the problem. I initialized the actor network weights in the following way:
def init_weights(self):
for m in self.modules():
if isinstance(m, nn.Linear):
torch.nn.init.normal_(m.weight, mean=0, std=0.01)
m.bias.data.fill_(0.01)
Was that what you meant?

Here the results of the training. the critic's loss remains a little lower but the problem is not solved.
https://imgur.com/wWdmBMU

@GPra I don't really have much experience with dirichlet distributions, but it seems that if you want to bias your network to prefer uniform distributions, you have to use high $\alpha$ https://en.wikipedia.org/wiki/File:Dirichlet.pdf

okay, I think I understand better now, basically I have to initialize the network with a bias value and the higher the bias value, the higher the alpha value and therefore the more uniform the dirichlet will be.
Because of the exp activation α=exp(bias)? Is that correct?

@GPra "more uniform the dirichlet will be" not really, if you have high $\alpha$, the dirichlet will be like a "gaussian centered in the middle of the simplex", so the samples will be more uniform distributions... question, why are you using dirichlet instead of a categorical distribution?

I am working in a continuous action space, whose sum of the 3 actions must be 1. I chose Dirichlet because it seemed to me to be the distribution that best fit to represent my problem and that also gave the best results in the initial experiments, with a version of the environment that was more simplified than the current one and that worked perfectly

continuous action space, whose sum of the 3 actions must be 1 isn't this the definition of a categorical distribution? I just saying because it's much simpler to calibrate, and since Dir dist are used as categorical prior, I thought you were in the setting of discrete actions

https://www.sciencedirect.com/science/article/pii/S0951832022001831

@GPra personally I find that they are cherry picking the examples... indeed, if you use a softmax activation over samples from Gaussian, you can have the variance they are saying... but the point is that you can just sample from the softmax, not having such problem (as done in the rest of the literature - that single paper)...

Perhaps I have not explained myself or I am missing some fundamental point. In my problem, at each step, the agent receives a variable number of requests it has to decide how many to process locally, how many to forward and how many to reject, and not whether it should process them locally, forward or reject. For example, an action could be [0.7, 0.2, 0.1], which means 70 locally, 20 forward and 10 reject. it seems to me a continuous space, not discrete. Excuse me if I am not clear or if I don't understand something you are telling me.

@GPra when we say continuous/discrete, we refer to the support, not the values... every common distribution is continuous in their values, but some have a discrete support (for example in your case, you have a discrete number of channels)

At a glance my guess is that what you are observing should be nothing to worry about.
For the critic loss what you have to consider is that as the size of the values grows i.e. the agent gets better the scale of errors will grow too because there are simply more ways to be wrong about predicting a value between 0 and 1000 than between 0 and 10. as an example of what you should be looking at as a performance metric is something like symmetric percentage error or explained variance like in the stable baselines implementation.
For the entropy loss something similar is happening. As the probability distribution becomes more concentrated entropy loss will increase. This is because the entropy of a 1-hot distribution is 0 and as it tends towards a uniform distribution that value increases however in standard PPO you subtract the entropy from the loss to incentivise higher entropy. You may have noticed that that would imply growing negative values but you are using differential entropy because of the dirichlet distribution instead of discrete entropy for a categorical which means that the entropy values you can get can be negative. In essence the higher the entropy loss the more deterministic your policy not less.
In short you've done a great job and have nothing to worry about.


text:
What does the branching factor mean in the time complexity of Breadth-First Search (BFS)
Can someone explain where my math is off here?
I am confused on the b - Branching Factor and how to calculate the worst-case scenario when running BFS. In a worst-case scenario BFS would have to hit every single node from the root node to the goal node giving the complexity of 1 (root node) + b (number of nodes at depth 1) + b^2 (number of nodes at depth 2) + ... + b^d = b^d.
Based on the following image does that mean the Time Complexity is has an upper bound of O(b^d) = O(2^4) = 16 nodes + 1 = 17 Nodes?

Image Reference
Of course, if this was a complete binary tree with max nodes at depth of 4 there would be a total of 1 + 2 + 4 + 8 + 16 = 31 Nodes so the previous complexity does not make sense.
Thank You!

$O(.)$ does not give you the worst case in an exact sense, but in an asymptotic sense, indeed it can be off by any constant, so $O(b^2)$ might even be $1000000\cdot b^d$

Math is just fine. Its all about the definition of $\mathrm{O}(\cdot)$. Checkout the wiki page about Big-O Notation.
Basically, your function $T(b) = 1 + b + b^{2} + \cdots + b^{d}$ belongs to the class of functions $\mathrm{O}(b^{d})$ if its upper bound is at most $b^{d}$ multiplied by some constant.
Note that:
$$ T(d) = 1 + b + b^{2} + \cdots + b^{d} \leq (d+1)b^{d}, $$
which means that $T(d) \in \mathrm{O}(b^{d})$.
You could also obviously say that $T(d) \in \mathrm{O}(b^{d+1})$. If you want to be more specific about the complexity class to which your function belongs, then you can use Big-Theta Notation instead.
Your function $T(b) = 1 + b + b^{2} + \cdots + b^{d}$ belongs to the class of functions $\mathrm{\Theta}(b^{d})$ if its upper bound is at most $b^{d}$ multiplied by some constant and its lower bound is at least $b^{d}$ multiplied by some constant.
You have:
\begin{align}
T(d) &= 1 + b + b^{2} + \cdots + b^{d} \leq (d+1)b^{d} \\
T(d) &= 1 + b + b^{2} + \cdots + b^{d} \geq b^{d},
\end{align}
which means that $T(d) \in \mathrm{\Theta}(b^{d})$.
HOWEVER, you can still say that $T(d) \in \Theta(b^{d+1})$....
\begin{align}
T(d) &= 1 + b + b^{2} + \cdots + b^{d} \leq b^{d+1} \\
T(d) &= 1 + b + b^{2} + \cdots + b^{d} \geq \frac{1}{b}b^{d+1}.
\end{align}
So, people mostly just say that the complexity class is "exponent of $b$", without thinking too much whether the power is $d$ or $d+1$.
Also this youtube video is highly recommended.


text:
Deriving ELBO for Diffusion Models
I am trying to read through the proof of ELBO for diffusion models on pg. 8 of this paper. However, I do not see how the author arrived at Eqn (45) from Eqn (44). Specifically, I do not know how they simplified the equation by rewriting it in terms of the KL divergence. I do know the formula for KL divergence but it does not seem like a direct application in this case. Any help is appreciated. I have included an image below of the proof for convenience.

Not really an answer, but same question asked here

Indeed the notations here are a little involved and confusing since there exist multiple joint conditional random variables here. First please note here all involved conditional random variables are all conditioned only on the fixed value of the input data $x_0$, so we can tentatively and effectively ignore it when considering other (conditional) random variables. Next let's look at the second term in your asked line (44): $$\mathbb{E}_{q(x_{T−1}, x_T | x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]$$
Notice compared to the usual KL divergence, there's an additional fixed conditioning value of $x_{T−1}$ as given data involved in the posterior $q$-distribution of the random variable $x_T$ in above term. And since the random variable $x_{T−1}$ is taken jointly with the random variable $x_T$ given $x_0$ over the expectation in above term, we have to take expectation wrt the random variable $x_{T−1}$ after rewriting the above term as KL divergence. With these ideas we can expand the above term as:
$$\mathbb{E}_{x_{T−1}|x_0 \sim q(x_{T−1}| x_0)}[\mathbb{E}_{x_T|x_{T-1}, x_0 \sim q(x_T|x_{T-1}, x_0)} [\log \frac{p(x_T)}{q(x_T | x_{T −1})}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[\mathbb{E}_{q(x_T|x_{T-1})} [\log \frac{q(x_T | x_{T −1})}{p(x_T)}]] \\= -\mathbb{E}_{q(x_{T−1}| x_0)}[D_{KL}(q(x_T | x_{T −1})||p(x_T))]$$
Similar process can be applied to the third term to explain the additional expectation operator of the rewritten KL divergence.


text:
ST-GCN: graph convolution operator in Geometry-Aware Interaction Network (GAIN)
I need help implementing the model in this paper:
They have adopted spatio-temporal graph convolution operator in ST-GCN [section 3.1.2]. I've found there is popular libraries available for GCN: pytorch_geometric and torch_geometric_temporal, but I'm new to the graph NNs.
This is the overal network architecture in the paper, and I'm tring to implement the geometric feature part. For the dense landmark detection I'll be using mediapipe's FaceMesh landmarks (because the dense landmark detection model used in the paper is not available publicly).
Currently what I've written is as below:
import torch
from torch.nn import Linear
from torch_geometric.nn import GCNConv
from torch_geometric.typing import Adj, OptTensor

class GAIN(torch.nn.Module):
    def __init__(
        self,
        in_channels: int,
        hidden_channels: int,
        out_channels: int,
    ):
        super().__init__()
        self._graph_conv_1 = GCNConv(
            in_channels=in_channels,
            out_channels=hidden_channels,
        )
        self._graph_conv_2 = GCNConv(
            in_channels=hidden_channels,
            out_channels=hidden_channels,
        )
        self._graph_conv_3 = GCNConv(
            in_channels=hidden_channels,
            out_channels=hidden_channels,
        )
        self._graph_conv_4 = GCNConv(
            in_channels=hidden_channels,
            out_channels=hidden_channels,
        )
        self._graph_conv_5 = GCNConv(
            in_channels=hidden_channels,
            out_channels=hidden_channels,
        )
        self._graph_conv_6 = GCNConv(
            in_channels=hidden_channels,
            out_channels=hidden_channels,
        )
        self.linear = Linear(hidden_channels, out_channels)

    def forward(
        self,
        x: torch.FloatTensor,
        edge_index: Adj,
        edge_weight: OptTensor = None
    ):
        x = self._graph_conv_1(x, edge_index, edge_weight)
        x = self._graph_conv_2(x, edge_index, edge_weight)
        x = self._graph_conv_3(x, edge_index, edge_weight)
        x = self._graph_conv_4(x, edge_index, edge_weight)
        x = self._graph_conv_5(x, edge_index, edge_weight)
        x = self._graph_conv_6(x, edge_index, edge_weight)
        x = self.linear(x)
        return x

def train_gcn():
    # I'm not sure what should be hidden and out channels, just setting 64 and 256.
    # I guess in channel should be 2 because node feateure is (x, y) coordinate.
    model = GAIN(in_channels=2, hidden_channels=64, out_channels=256)

    criterion = torch.nn.BCELoss()
    optimizer = torch.optim.SGD(model.parameters(), momentum=0.9, weight_decay=0.0001, lr=0.1)

    data = None # I don't know

    # Training loop
    for epoch in range(65):
        # Clear gradients
        optimizer.zero_grad()
        # Forward pass
        y_hat = model(data.x, data.edge_index)
        # Calculate loss function
        loss = criterion(y_hat, data.y)
        # Calculate accuracy
        acc = accuracy(y_hat.argmax(dim=1), data.y)
        # Compute gradients
        loss.backward()
        # Tune parameters
        optimizer.step()

        # Print metrics every 10 epochs
        if epoch % 10 == 0:
            print(f'Epoch {epoch:>3} | Loss: {loss:.2f} | Acc: {acc*100:.2f}%')

What I'm struggling with are:

I don't know how to interpret formula (3) in the code.
How to construct input X and edge matrices.
Overall am I implementing it correctly?

Please help me implement it.

'Please help me implement it' is not a question that can be answered on this forum. You will need to ask specific questions that you struggle with when implementing it. What do you not understand about the paper that makes it difficult for you to implement it?

@RobinvanHoorn I've updated the question


text:
How to prompt an LLM for a real variable list length?
I struggle to get a variable list out of an LLM. Even with a prompt like
short list of the most important steps to build a house, with a variable length between 5 and 15 entries

The resulting list has most of the time exactly 15 entries, sometimes exactly 10, but never eg. 6 or 8 also never 13. Why? How to overcome this?
I also tried prompts like … maximum 15 entries, however this also produces always exactly 15 entries, never less.
The topic of the lists can vary and I expect the ideal list will depend on the complexity of the topic, sometimes 6 entries will have all relevant information, but sometimes more will be needed. The LLM should fiddle this out by itself.
Is this a general LLM problem? I experienced this with Open AIs GPT-3.5.

The behavior you're observing is likely due to the nature of how GPT-3.5 generates text. It's a complex model trained on a large dataset, but it doesn't have an inherent understanding of the concepts of "lists" or "variable length" in the same way humans do.
When you ask for a "variable length" list, GPT-3.5 doesn't have a direct way to understand what that means. Instead, it might default to generating lists with a fixed number of items because it doesn't have any specific instructions on how to handle variable lengths.
To overcome this, you might need to approach the problem in a slightly different way. Instead of asking for a variable length list directly, you can ask for a list with a specific range, and then manually trim it down to the desired length afterwards.
For example, you could ask for a list with a range between 5 and 15 items, and then manually remove any excess items once you receive the response.
Keep in mind that while this method might work better for getting variable length lists, it's not foolproof. GPT-3.5 might still produce lists with a consistent length due to its training data and patterns it learned during training.
If you absolutely need precise control over the length of the list, you might need to implement some post-processing on the generated output to trim it down to your desired length. This could be done using some form of list summarization or filtering algorithm after receiving the response from the model.

Sadly stripping list entries later will be not feasible in my current case. However you explanation why this can't work is really helpful, thanks!


text:
How to guess a card in the game of Cambio with limited information?
I need help with a probability problem in the card game of Cambio. In this game, two players are dealt four cards each from a deck of 52 cards . At the start of the game, the bottom cards of each player are observed and there are four hidden top cards (two in each player's hand). I am using minimax for my AI where a shallow copy of the game object is generated. Then possible moves are calculated based on the dealt card from the top of the deck. If 7,8 was at the top of the deck, our AI is able to peek a card in its hand(enemy’s hand if the card is 9 or 10). The problem is we are not allowed to reveal the card. Ai has to guess the probabilities of every card type that could be the card we want to reveal.  For example, here:

Heart 7, spades 8, spades, diamond 7. And x's are hidden.
How would one guess the card above Heart 7? We would have to assume that either all the hidden cards are of type 7, none of the cards are or some of the cards are of type 7.

I'm not sure I understand the game rules from your description. I expect the question summary is fine if someone already knows the rules, but if you could link something more complete, like a Wikipedia article then it may help.

Also you are asking about guessing a hidden card. The answer is likely to be a probability distribution - it may help to understand why you want this guess data - e.g. are you writing a bot that can play the game?


text:
Multiple Loss Functions For Proper Parameter Updates
I am working on a model on PyTorch where it has two loss functions each fed from two separate input datasets. I want to update the model parameters based on these loss functions, ic_loss, res_loss, and also record their respective values each epoch. I highlighted by ### 1 ###, ### 2 ### and ### 3 ### the lines of code that I modified. It seems to me that both Code 1) and Code 2) do the same thing, but I doubt otherwise. Here are three alternatives from the training part of my code that I am not sure of which to pick:
Code 1)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      self._optimizer.step() ### 2 ###
      self._optimizer.zero_grad() ### 3 ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 2)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss.backward() ### 1 ###
        self._optimizer.step() ### 2 ###
        self._optimizer.zero_grad() ### 3 ###
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 3)
    self._model.train()

    for epoch in range(1, self._epoch_number + 1):

      loss_info_string = f'Epoch number: [{epoch} / {self._epoch_number}] - '
      ic_loss_val = 0.0
      res_loss_val = 0.0

      for ic_input in self._ic_loader:

        u_hat = self._model(torch.cat(ic_input, dim=1))
        ic_loss = self._ic_loss(u_hat, ic_input)
        ic_loss_val += ic_loss.item()

      ic_loss_val /= len(self._ic_loader)

      loss_info_string += f'Initial Condition error/loss: {ic_loss_val:.6e}'

      for res_input in self._res_loader:

        u_hat = self._model(torch.cat(res_input, dim=1))
        res_loss = self._res_loss(u_hat, res_input)
        res_loss_val += res_loss.item()

      res_loss_val /= len(self._res_loader)

      loss_info_string += f' - Residual error/loss: {res_loss:.6e}'

      (ic_loss + res_loss).backward() ### I ###
      self._optimizer.step() ### II ###
      self._optimizer.zero_grad() ### III ###

      loss_info_string += f' - Total error/loss: {ic_loss_val + res_loss_val:.6e}'

Code 1) and Code 3) seem to be the same except for if I used a batch size different from the entire datasets since Code 1) would accumulate gradients for each batch whereas Code 3) would use only the last batch. Therefore, I discarded Code 3). Is that correct?
Also based on what I tried to described above, which one should I consider using, Code 1) or Code 2)? I hope everything is clear. Thanks in advance!

For your use case, you want to update the model with a single .backward() call to a combined loss, similar to what you're doing in version 3. However, version 3 only calls the gradient update after going through both dataloaders - you want to do it every batch. Something like this
ic_iter = iter(self._ic_loader)
res_iter = iter(self._res_loader)
for i in range(iterations):
    ic_input = next(ic_iter)
    u_hat = self._model(torch.cat(ic_input, dim=1))
    ic_loss = self._ic_loss(u_hat, ic_input)

    res_input = next(res_iter)
    u_hat = self._model(torch.cat(res_input, dim=1))
    res_loss = self._res_loss(u_hat, res_input)

    loss = ic_loss + alpha*res_loss # alpha scaling parameter
    loss.backward()
    self._optimizer.step()
    self._optimizer.zero_grad()

If the dataloaders are different lengths, you'll have to figure out the right cadence for refreshing the iterators.

Thanks to your reply that made me remember that I code on Python, I found an even simpler solution to this by making use of sum() function with embedding the "for-loop" as generators for both cases and summing them up for backward() as such: ic_loss = sum(self._ic_loss(self._model(torch.cat(ic_input, dim=1)), ic_input) for ic_input in self._ic_loader) with (ic_loss + res_loss).backward().


text:
Why Does the Model not Improve in PyTorch?
I have a simple curve fitting problem in hand. I wrote some code in PyTorch as follows:
class MyDatasetV1(torch.utils.data.Dataset):

  def __init__(self, dataset):

    # Initialize a dataset

    assert isinstance(dataset, list), '"dataset" must be of "tuple" type'
    assert isinstance(dataset[0], torch.Tensor), '"x" must be of "torch.Tensor" type!'
    assert isinstance(dataset[1], torch.Tensor), '"y" must be of "torch.Tensor" type!'

    self.x = dataset[0]
    self.y = dataset[1]
    self.length = self.x.shape[0]

  def __len__(self):

    # Get the number of elements in entire dataset

    return self.length

  def __getitem__(self, index):

    return self.x[index], self.y[index]

class MyModelV2(torch.nn.Module):

  def __init__(self, input_size, output_size, hiddens, weights, biases, batchnorms, activations, dropouts):

    # Initialize a custom fully-connected model

    super(MyModelV2, self).__init__()

    assert len(hiddens) + 1 == len(weights), 'Number of hidden layers must match the number of "weights" units/tensors!'
    assert len(hiddens) + 1 == len(biases), 'Number of hidden layers must match the number of "bias" units/scalars!'
    assert len(hiddens) + 1 == len(batchnorms), 'Number of hidden layers must match the number of "batch normalization" units!'
    assert len(hiddens) + 1 == len(activations), 'Number of hidden layers must match the number of "activation" functions!'
    assert len(hiddens) + 1 == len(dropouts), 'Number of hidden layers must match the number of "dropout" units!'

    self.weights = weights
    self.biases = biases
    self.batchnorms = batchnorms
    self.activations = activations
    self.dropouts = dropouts
    self.layers_size = [input_size]
    self.layers_size.extend(hiddens)
    self.layers_size.append(output_size)
    self.layers = torch.nn.ModuleList()

  def build(self):

    # Build a model with given specifications

    for index in range(len(self.layers_size) - 1):

      layer = torch.nn.Linear(self.layers_size[index], self.layers_size[index + 1])

      if self.weights[index]:

        self.weights[index](layer.weight)

      if self.biases[index]:

        self.biases[index](layer.bias)

      self.layers.append(layer)

      if self.batchnorms[index]:

        self.layers.append(torch.nn.BatchNorm1d(self.layers_size[index + 1]))

      if self.dropouts[index]:

        self.layers.append(torch.nn.Dropout(self.dropouts[index]))

      self.layers.append(self.activations[index])

  def forward(self, x):

    # Forward pass for a given input

    for layer in self.layers:

      x = layer(x)

    return x

def set_weight(weights):

  return torch.nn.init.xavier_uniform_(weights)

def set_bias(biases):

  return torch.nn.init.zeros_(biases)

torch.manual_seed(7)

model = MyModelV2(1, 1, [64, 64], 3 * [set_weight], 3 * [set_bias], 3 * [False], 3 * [torch.nn.ReLU()], 3 * [False])
model.build()

criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())

x = torch.linspace(-10, 10, 1000 * 1).reshape((1000, 1))
y = 0.1 * x * torch.cos(x) + 0.05 * torch.normal(1, 2, size=(1000, 1))

ds = MyDatasetV1([x, y])
ds_loader = torch.utils.data.DataLoader(ds, batch_size=32, shuffle=True)

def get_training_loss(model, training_loader, criterion, optimizer):

  # Training loop for a given model

  model.train()
  training_loss = 0.0

  for x_train, y_train in training_loader:

    optimizer.zero_grad()
    y_hat_train = model(x_train)
    train_loss = criterion(y_hat_train, y_train)
    train_loss.backward()
    optimizer.step()
    training_loss += train_loss.item()

  # Calculate the average training loss

  training_loss /= len(training_loader)

  return training_loss

EPOCHS = 100

for epoch in range(1, EPOCHS + 1):

  tr = get_training_loss(model, ds_loader, criterion, optimizer)

  print(f'Epoch number: {epoch} - Training error/loss: {tr:.6e}')

def predictor(model, x):

  # Predict after training for a given model

  model.eval()

  with torch.no_grad():

    x = model(x)

  return x

y_hat = predictor(model, x)

plt.figure(dpi=120)
plt.plot(x.numpy(), y.numpy(), 'ro', markersize=1.5, label='(x, y)')
plt.plot(x.numpy(), y_hat.numpy(), 'bo', markersize=1.5, label='(x, $\hat{y}$)')
plt.xlabel('x')
plt.ylabel('y')
plt.tight_layout()
plt.legend()
plt.show()

Even though I tried different models, i.e., 32, 32 or 64, 32, 16 neurons at hidden layers, etc., I ended up having a zero prediction from all as shown in the figure below.

I reviewed my model many times, but I could not figure out the issue here. What is wrong with it? Thanks in advance!

For whom this post and reply might be useful,
If the model is examined carefully, the ReLU activation function at the end before the output neurons is what causes the problem. Since the dataset consists of positive and negative values, using ReLU activation function just before the output neurons prevents the model from learning the negative values of y in the dataset as seen in the plot above.

Well done to find this bug. Your answer does show why this kind of question gets low interest. At the very least the expert helping has to carefully read all the code, they probably have to get more involved and run it. I've answered a few similar and they have been very time consuming


text:
Toy dataset: Radial VAE
I'm evaluating disentanglement in toy datasets seeing as we have such little understanding of the phenomena. I'm using various tools from differential geometry. Now I want to train a VAE on the following dataset:
class Radial(Dataset):
    def __init__(self, num_samples=1000, num_radii=5, num_angles=8, r_noise=0.0, angle_noise=0.0):
        """
        Initialize the dataset with random samples in polar coordinates.
        The radial distances are equally spaced between 0.5 and 2.5.
        The angles are equally spaced between 0 and 2 * pi.

        Parameters:
        - num_samples: Total number of samples in the dataset
        - num_radii: Number of unique radial distances
        - num_angles: Number of unique angles
        - r_noise: Standard deviation of Gaussian noise added to the radial distances
        - angle_noise: Standard deviation of Gaussian noise added to the angles
        """
        self.num_samples = num_samples
        self.num_radii = num_radii
        self.num_angles = num_angles

        # Generate the radial distances and angles
        radii = np.linspace(0.5, 2.5, num_radii)
        angles = np.linspace(0, 2 * np.pi, num_angles, endpoint=False)

        # Create the grid of (r, theta)
        r, theta = np.meshgrid(radii, angles)

        # Flatten and repeat to create the dataset
        r_flat = r.flatten()
        theta_flat = theta.flatten()
        repeats = num_samples // (num_radii * num_angles)

        self.r = np.tile(r_flat, repeats)
        self.theta = np.tile(theta_flat, repeats)

        # Add Gaussian noise to the radial distances and angles
        self.r += np.random.normal(0, r_noise, self.r.shape)
        self.theta += np.random.normal(0, angle_noise, self.theta.shape)

        # Convert to Cartesian coordinates (x, y)
        self.x = self.r * np.cos(self.theta)
        self.y = self.r * np.sin(self.theta)

        # Convert to torch tensors
        self.data = torch.tensor(np.column_stack((self.x, self.y)), dtype=torch.float32)
        self.gen_factors = torch.tensor(np.column_stack((self.r, self.theta)), dtype=torch.float32)

    def __len__(self):
        return self.num_samples

    def __getitem__(self, index):
        return self.data[index]

The architecture that I came up with is as follows:
import torch
import torch.nn as nn
import torch.nn.functional as F

class Layer(nn.Module):
    def __init__(self, input_dim, output_dim, act_func=nn.Tanh()):
        super(Layer, self).__init__()
        if act_func is None:
            self.act_func = lambda x: x
        else:
            self.act_func = act_func
        self.linear_map = nn.Linear(input_dim, output_dim)
        self.out_features = output_dim
        self.in_features = input_dim

    def forward(self, x):
        return self.act_func(self.linear_map(x))

class PolarLayer(nn.Module):
    def __init__(self):
        super(PolarLayer, self).__init__()

    def forward(self, x):
        r, theta = x.split(1, dim=-1)
        x = r * torch.cos(theta)
        y = r * torch.sin(theta)
        return torch.cat([x, y], dim=-1)

class EuclLayer(nn.Module):
    def __init__(self):
        super(EuclLayer, self).__init__()

    def forward(self, x):
        # Assuming x is of shape [batch_size, 2]
        # where x[:, 0] = radius (r) and x[:, 1] = angle (theta)

        r = x[:, 0]
        theta = x[:, 1]

        # Convert to Cartesian coordinates
        x_cartesian = r * torch.cos(theta)
        y_cartesian = r * torch.sin(theta)

        # Concatenate to form output tensor of shape [batch_size, 2]
        output = torch.stack([x_cartesian, y_cartesian], dim=1)

        return output

# Define the Encoder module
class Encoder(nn.Module):
    def __init__(self, in_features, features, out_features):
        super(Encoder, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.layers = nn.ModuleList([
            PolarLayer(), nn.Linear(in_features, features[0])
        ] + [
            Layer(features[i], features[i + 1]) for i in range(len(features) - 1)
        ])
        self.fc_mu = Layer(features[-1], out_features)
        self.fc_log_var = Layer(features[-1], out_features)

    def forward(self, x):
        x = self.layers[0](x)
        for layer in self.layers[1:]:
            x = F.tanh(layer(x))
        mu = self.fc_mu(x)
        log_var = self.fc_log_var(x)
        return mu, log_var

    def forward_layers(self, x, indx):
        assert indx < len(self.layers)

        if indx == 0:
            x = self.layers[0](x)
            for layer in self.layers[indx:]:
                x = layer.forward(x)
        else:
            for layer in self.layers[indx:]:
                x = layer.forward(x)
        return x

# Define the Decoder module
class Decoder(nn.Module):
    def __init__(self, in_features, features, out_features):
        super(Decoder, self).__init__()
        self.in_features = in_features
        self.out_features = out_features

        self.layers = nn.ModuleList([
            nn.Linear(in_features, features[0])
        ] + [
            nn.Linear(features[i], features[i + 1]) for i in range(len(features) - 1)
        ] + [
            nn.Linear(features[-1], out_features), EuclLayer()
        ])

    def forward(self, x):
        for layer in self.layers[:-1]:
            x = F.tanh(layer(x))
        x = self.layers[-1](x)
        return x

class VAE(nn.Module):
    def __init__(self, encoder, decoder):
        super(VAE, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.layers = encoder.layers

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)
        eps = torch.randn_like(std)
        return mu + eps * std

    def forward(self, x):
        mu, log_var = self.encoder(x)
        z = self.reparameterize(mu, log_var)
        x_reconstructed = self.decoder(z)
        return x_reconstructed, mu, log_var

My main concern is whether or not to introduce polar coordinates. Without the conversion, my latent traversals for the radius look fine but on the angle it's simply a straight line, which is incorrect. Now I've tried a few different combinations for the architecture, tanh seems to work best for the activation function. However the reconstruction is always incorrect despite some very strong dampening of the KL divergence (for lambda = 0 to 0.5 through training, loss = recon_loss + lambda*KL). The traversals are sometimes better (I've included one that's on the poorer side) but they're not very consistent. Is there any other way of incorporating polar coordinates into the VAE?
The training script is below for reproducibility. Current default arguments lead to these plots.

from torch.optim.lr_scheduler import ReduceLROnPlateau
import matplotlib.pyplot as plt
import argparse
import os
from torch.utils.data import DataLoader
from torch.optim import Adam
from models.data.gen_factors import Radial
from models.unsupervised.vae.model import Encoder, Decoder, VAE
import torch.nn.functional as F
import torch
from tqdm import tqdm

def plot_reconstructions(model, dataset, save_dir, num_samples=10):
    """
    Plot original and reconstructed data points.
    """
    plt.figure(figsize=(12, 6))

    # Sample data points from the dataset
    sampled_data = [dataset[i] for i in range(num_samples)]
    sampled_data = torch.stack(sampled_data)

    # Get the reconstructed data points
    with torch.no_grad():
        reconstructed_data, _, _ = model(sampled_data)

    plt.subplot(1, 2, 1)
    plt.scatter(sampled_data[:, 0], sampled_data[:, 1], c='b', label='Original')
    plt.title('Original Data Points')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid(True)
    plt.legend()

    plt.subplot(1, 2, 2)
    plt.scatter(reconstructed_data[:, 0], reconstructed_data[:, 1], c='r', label='Reconstructed')
    plt.title('Reconstructed Data Points')
    plt.xlabel('x')
    plt.ylabel('y')
    plt.grid(True)
    plt.legend()

    plt.savefig(f"{save_dir}/recon.png")

def plot_latent_traversal(model, save_dir, dataset, num_points=10):
    """
    Plot latent space traversal and observe how it affects the generated data.
    """
    fig, ax = plt.subplots(1, 3, figsize=(18, 8))

    # Retrieve the unique radii and angles from the dataset
    unique_r = torch.unique(dataset.gen_factors[:, 0])
    unique_theta = torch.unique(dataset.gen_factors[:, 1])

    # Sample some unique radii and angles
    sampled_r = unique_r[torch.randint(0, len(unique_r), (int(num_points**0.5), ))]
    sampled_theta = unique_theta[torch.randint(0, len(unique_theta), (int(num_points**0.5), ))]

    # Create a grid in the latent space corresponding to these radii and angles
    z_r, z_theta = torch.meshgrid(sampled_r, sampled_theta)

    # Flatten and create a tensor of shape [num_points * num_points, 2]
    z = torch.stack([z_r.flatten(), z_theta.flatten()], dim=1)

    # Decode the latent variables to generate data
    with torch.no_grad():
        generated_data = model.decoder(z)

    # Plotting
    ax[0].scatter(generated_data[:, 0], generated_data[:, 1], c='g', label='Generated Data')
    ax[0].set_title('Latent Space Traversal')
    ax[0].set_xlabel('x')
    ax[0].set_ylabel('y')
    ax[0].grid(True)
    # Generate traversal values
    traversal_values_radius = torch.linspace(0, 2.5, num_points)
    travel_values_angles = torch.linspace(0, 2*torch.pi, num_points)
    traversal_values = [traversal_values_radius, travel_values_angles]
    for i in range(model.encoder.out_features):
        z = torch.zeros(num_points, model.encoder.out_features)
        z[:, i] = traversal_values[i]

        with torch.no_grad():
            generated_data = model.decoder(z)

        ax[i+1].scatter(generated_data[:, 0], generated_data[:, 1], c='g', label=f'Latent Dim {i+1}')
        ax[i+1].set_title(f'Latent Dimension {i+1} Traversal')
        ax[i+1].set_xlabel('x')
        ax[i+1].set_ylabel('y')
        ax[i+1].grid(True)

    plt.savefig(f"{save_dir}/latent_traversal.png")

# Define the VAE loss function (negative ELBO)
def vae_loss(x_reconstructed, x, mu, log_var):
    # Reconstruction loss (MSE)
    recon_loss = F.mse_loss(x_reconstructed, x, reduction='sum')

    # KL divergence loss
    kl_loss = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())

    return recon_loss, kl_loss

def train(model, loader, criterion, optimizer, lambda_):

    torch.manual_seed(args.seed)
    model.train()
    epoch_loss = 0
    store_loss = []
    store_recon_loss = []
    store_kl_loss = []

    for x in loader:
        # Forward pass
        x_reconstructed, mu, log_var = model(x)

        # Compute loss
        recon_loss, kl_loss = criterion(x_reconstructed, x, mu, log_var)
        loss = recon_loss + lambda_*kl_loss

        # Backward pass and optimization
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()
        store_loss.append(loss.item())
        store_recon_loss.append(recon_loss.item())
        store_kl_loss.append(lambda_*kl_loss.item())
    return epoch_loss / len(loader.dataset), store_loss, store_recon_loss, store_kl_loss

def main(args):
    model_name = f"R{args.num_radii}-A{args.num_angles}-Nr{args.radial_noise}-Na{args.angle_noise}"
    save_dir = f"models/unsupervised/vae/saved_models/{model_name}"
    plot_dir = f"models/unsupervised/vae/figures/{model_name}"
    os.makedirs(save_dir, exist_ok=True)
    os.makedirs(plot_dir, exist_ok=True)

    # Initialize the VAE model
    features = [128, 64]
    encoder = Encoder(in_features=args.in_dim, features=features, out_features=args.out_dim)
    decoder = Decoder(in_features=args.out_dim, features=list(reversed(features)), out_features=args.in_dim)
    vae = VAE(encoder, decoder)
    assert args.n_samples > args.num_radii*args.num_angles, "Number of samples must be greater than the number of radii times angles"
    num_samples = args.n_samples - (args.n_samples % (args.num_radii*args.num_angles))
    # Create the dataset and dataloader
    if args.dataset == "radial":
        dataset = Radial(num_samples, num_radii=args.num_radii, num_angles=args.num_angles, r_noise=args.radial_noise, angle_noise=args.angle_noise)
    dataloader = DataLoader(dataset, batch_size=args.batch_size, shuffle=True)
    # Initialize the optimizer
    optimizer = Adam(vae.parameters(), lr=args.learning_rate)
    # Training loop
    num_epochs = args.num_epochs
    store_loss = []
    store_recon_loss = []
    store_kl_loss = []
    for epoch in range(num_epochs):
        lambda_ = min(0.5, epoch/args.num_epochs)
        epoch_loss, store_loss_, store_recon_loss_, store_kl_loss_ = train(vae, dataloader, vae_loss, optimizer, lambda_=lambda_)
        store_loss += store_loss_
        store_recon_loss += store_recon_loss_
        store_kl_loss += store_kl_loss_

        if epoch % args.SAVE_LOG == 0:
            torch.save(vae.state_dict(), os.path.join(save_dir, f"model_{epoch}.pth"))
        if args.verbose:
            print(f"Epoch [{epoch+1}/{num_epochs}], Loss: {epoch_loss:.4f}")

    print("Training complete.")
    if args.PLOT_LOSS:
        plt.plot(store_loss)
        plt.plot(store_recon_loss)
        plt.plot(store_kl_loss)
        plt.legend(["Total loss", "Reconstruction loss", "KL loss"])

        plt.savefig(f"{plot_dir}/loss.png")

    if args.PLOT_MODEL:
        plot_reconstructions(vae, dataset, plot_dir, num_samples)
        plot_latent_traversal(vae, plot_dir, dataset, num_samples)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--num_epochs', type=int, default=400)
    parser.add_argument('--batch_size', type=int, default=128)
    parser.add_argument('--learning_rate', type=float, default=0.01)

    parser.add_argument('--out_dim', type=int, default=2)
    parser.add_argument('--in_dim', type=int, default=2)

    parser.add_argument('--n_samples', type=int, default=3000)
    parser.add_argument('--dataset', type=str, default='radial')
    parser.add_argument('--num_radii', type=int, default=5)
    parser.add_argument('--num_angles', type=int, default=16)

    parser.add_argument('--radial_noise', type=float, default=0.01)
    parser.add_argument('--angle_noise', type=float, default=0.01)

    parser.add_argument('--seed', type=int, default=2)
    parser.add_argument('--PLOT_LOSS', type=bool, default=True)
    parser.add_argument('--PLOT_MODEL', type=bool, default=True)
    parser.add_argument('--SAVE_LOG', type=int, default=1)
    parser.add_argument('--verbose', type=bool, default=True)
    args = parser.parse_args()
    main(args)

```

Can you please put your specific question in the title? This post is already too long and it may help to immediately show that actual question in the title.


text:
Calculating state value using bellman equation question:
Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the corresponding state transitions, except that actions that would take the agent off the grid leave the state unchanged. We model this as an undiscounted,episodic task where the reward is -1 for all transitions. Suppose that the agent follows equi-probable random policy. Calculate the state values where * is placed.(Hint: Bellman expectation equation)

Grid(list of rows):
[

[  0,  *,-20,-22],

[-14,-18,-20,  *],

[   ,-20,-18,-14],

[-22,-20,   ,  0]

]

Should the * be initialized as 0 and then calculated using the equation?

Welcome to AI Stack Exchange. This is definitely solvable, but it would help if you could clarify what you mean by "the equation". We can either confirm your suggestion is correct, or show you what's wrong with it, but we cannot do either unless you show which equation you mean. There are multiple Bellman equations, so if that's what you mean, identify which one(s) you were considering using

@NeilSlater I should have mentioned the equation. I have edited the post and added the equation now. I had this question in one of my tests and so I had copied it as it is. Even the missing values were copied as they were in question so I copied them in the same way .

OK, so you don't need the missing values to do the calculation. They were clearly removed because they are in symmetrical positions to the *, so you could correctly assume the problem is symmetrical and simply copy them :-)

Should the * be initialized as 0 and then calculated using the equation?

This would work if you repeated the calculation multiple times, and that would be similar to a solution method called value iteration. However, it is not what is intended here. Instead, if you expand the Bellman equation for $v_{\pi}(s)$ using your knowledge of existing values and the policy, you will get terms in your unknown on both sides (because one action in each case will stay in the same state) which you can resolve to directly find the value for the marked squares.
To work through this for the * on the top line, we can call the value $x$ and expand the Bellman equation as follows (working clockwise from "move up" action):
$$x = 0.25 * (-1 + x) + 0.25 * (-1 - 20) + 0.25 * (-1 - 18 ) + 0.25 * (-1 + 0)$$
$$3x = -1 - 21 - 19 -1$$
$$3x = -42$$
$$x = -14$$


text:
How random should an untrained generative AI output really be?
I am developing a particular implementation of VAE, and, how usually one does while implementing any architecture, I passed a random input to the model to test if everything worked fine (e.g. check for size mismatches, device shenanigans, etc..). While I usually just check that input, intermediate and final shapes are correct, this time I decided to also visualize the ouput on that random input, not expecting much more than pure noise.

And while of course it isn't anything semantically significant (especially since the input itself has no information, technically), it also isn't complete garbage, presenting itself way less random than the input, and clearly having regions where a certain tone prevails on others.
Out of curiosity, I also tested just the decoder part of the network, by passing a normal latent sample.

It seems clear that the main culprit of this curious behaviour is the decoder network, which would lead me to believe that something similar may also happen with decoder based models such as GAN's.
Now this leaves me with a burning desire to understand: why?
Of course, we don't really have any guarantees on the distribution of the output of a random network (or at least I think), and the decoder I am using is also a pretty complicated one, so God knows what those millions of random parameters are doing, but at the same time, I feel like I cannot be the first person to have wondered what kind of patterns are expected, if any.
The only reason I can think of that could maybe explain this behaviour, and I don't really have much to prove it, is that since there are so many inputs and the parameters to the network, it is easy to hit some combinations of layer_input/layer_weight for which some kind of cascade effect triggers, which in turn produces these regions.
I know this may be too much of an open ended question, but at the same time, I feel like it is too interesting not to ask, especially since I am not looking for answers specifically related to my model architecture but rather for a more general understanding of this phenomena. Is this behaviour expected? Could this be indicative of a good/bad model architecture or weight initialization? Is this just my primate brain looking for patterns where there really aren't any?
Thank you so much in advance for any responses!
EDIT: I also tried on a non-random input (FFHQ-256 for those wondering), and in the output you can actually see that, inspite the network being untrained, one can make out some faint outlines of the face!

For your first input/output diagram have you visualized and compared with several trials with same seeded random input but different encoder/decoder randomly initialized weights, having similar patterns in the VAE output? For the bottom diagrams since the latent are conditioned on input, sure there'll be footprint in the output even without any training.

@cinch It the output patterns positions and color wildly vary depending of the network initialization. I also tried different weight initialization strategies, such as xavier, kaiming, normal, orthogonal, and that varies the pattern types. I will probably add these findings in the post later, as for now I am at uni :D

Do you have linked or shared weights between encoder and decoder networks? For autoencoders that is usually an optional setting (and it may default to true in the library you are using). If so, that could go some way to explaining visible influence in the untrained network. There may be other architecture choices that impact this, so maybe worth sharing details of the model you are working with


text:
why learn an observation model when training latent space model in model based rl
I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.
At 1:52 of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o_t | s_t)$ instead of a state model $p(s_t | o_t)$? For me, it seems more intuitive to get a state from a state model $p(s_t|o_t)$ and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?
Also at 4:38 of the lecture, following objective function is presented.
$max_\phi \frac{1}{N} \Sigma_{i=1}^{N} \Sigma_{t=1}^{T} E_{(s_t, s_{t+1}) \sim p(s_t, s_{t+1}|o_{1:T}, a_{1:T})}[logp_\phi(s_{t+1,i} | s_{t,i}, a_{t, i}) + logp_\phi(o_{t, i} | s_{t, i})]$
In the objective function, $logp_\phi(o_t | s_t)$ is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why $p_\phi(o_t|s_t)$ would be preferable over $p_\phi(s_t|o_t)$ as an objective function term.
I tried to interpret $p_\phi(o_t|s_t)$ term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of $p_\phi(s_t|o_t)$. However, I do not think that it can be interpreted as a log likelihood since $E_{s_t \sim p_\phi(s_t|o_t)}[logp_\phi(o_{t, i} | s_{t, i})] = \Sigma_{s_{t,i}} logp_\phi(o_{t, i} | s_{t, i}) * p_\phi(s_{t,i}|o_{t,i}) \neq logp_\phi(o_{t,i})$
In summary, my questions are following.

Why learn an observation model $p_\phi(o_t|s_t)$ instead of a state model $p_\phi(s_t|o_t)$
Why have the reconstruction term in the loss instead of $logp_\phi(s_t|o_t)$?

Thank you

The problem is the definition of what's $o_t$ and $s_t$:

$o_t$ is the (partially) observable part
$s_t$ is ideally a perfect model of the world

now, by definition, you don't have access directly to $s_t$, otherwise half of the problem would be solved on the fly
So, answering your questions:

Because there is no way to learn it, instead as proposed in the video, you use a NN to approximate a posterior, which takes as input all the previous observations and actions, and gives you a useful state representation (think about an RNN as encoder, the final latent vector is the state representation)
because you want to learn a state representation that carries information about your future, which is $o_t$ (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last "latent vector" $s_t$ should carry information useful to predict what's about to happen $o_t$)

@platoDev no prob, good luck with the course

Thanks to your answer, it now seems obvious to me that observation is the only thing that is observable thus a reconstruction term should appear in the objective function. However, I have come to another question. What would be the possible interpretation of maximizing Σ(|)∗(_t|_t) where summation is over s_t? I still cannot get a grasp of it. It certainly is not likelihood.

@platoDev can you please post a new question with more informations and reference from where you see this notation? thanks