text:
Since $f_c$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. Why avoid this?
Chapter 1.2.1.5 Uncertainty of Probabilistic Machine Learning: An Introduction by Kevin P. Murphy says the following:

We can capture our uncertainty using the following conditional probability distribution:
$$p(y = c \mid \mathbf{x}; \mathbf{\theta}) = f_c(\mathbf{x}; \mathbf{\theta}) \tag{1.7}$$
where $f: \chi \to [0, 1]^C$ maps inputs to a probability distribution over the $C$ possible output labels. Since $f_c(\mathbf{x}; \mathbf{\theta})$ returns the probability of class label $c$, we require $0 \le f_c \le 1$ for each $c$, and $\sum_{c = 1}^C f_c = 1$. To avoid this restriction, it is common to instead require the model to return unnormalized log-probabilities. We can then convert these to probabilities using the softmax function, which is defined as follows
$$\text{softmax}(\mathbf{a}) \triangleq \left[ \dfrac{e^{a_1}}{\sum_{c^\prime = 1}^C e^{a_{c^\prime}}}, \dots, \dfrac{e^{a_C}}{\sum_{c^\prime = 1}^C e^{a_{c^\prime}}} \right] \tag{1.8}$$
This maps $\mathbb{R}^C$ to $[0, 1]^C$, and satisfies the constraints that $0 \le \text{softmax}(\mathbf{a})_c \le 1$ and $\sum_{c = 1}^C \text{softmax}(\mathbf{a})_c = 1$.

But why would we want to "avoid this restriction"? $f_c$ is a probability, so the restrictions $0 \le f_c \le 1$ for each $c$ and $\sum_{c = 1}^C f_c = 1$ are necessary by definition. Also, why would we do this and then require the model to return log-probabilities only to then use the softmax function to convert it back into a probability?

Your intuition is correct. The restrictions you wrote down are necessary by definition.
The author means that it is hard to build a machine learning model that gives probability by design ( returns $f_c$ directly). However we know very well how to model a function that outputs some real number. So the way to go is to create a model that gives some unnormalised log-probability ( essentially any number ) first and then transform it into $f_c$ by softmax function.
Hope it helps.


text:
Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?
About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a clear answer. But, I thought. If we feed in each sequence straight into the Attention mechanism, then the linear layers, which are supposed to transform these inputs, won't be able to accept that, because sequences can be different length, and linear layers' input shape is fixed. And if we process each token independently, we have to store the other tokens somewhere, and then have loops to iterate through each token etc.
So, I decided to find some code, where people create that Multi-Head attention, and here's what I found:
import tensorflow as tf
from tensorflow.keras import layers

class MultiHeadAttention(layers.Layer):
    def __init__(self, model_dim, n_heads, rate=0.1, initializer='glorot_uniform'):
        super(MultiHeadAttention, self).__init__()
        self.n_heads = n_heads
        self.model_dim = model_dim

        assert model_dim % self.n_heads == 0

        self.head_dim = model_dim // self.n_heads

        self.wq = layers.Dense(model_dim, kernel_initializer=initializer)
        self.wk = layers.Dense(model_dim, kernel_initializer=initializer)
        self.wv = layers.Dense(model_dim, kernel_initializer=initializer)

        self.dropout1 = layers.Dropout(rate)
        self.dropout2 = layers.Dropout(rate)

        self.wo = layers.Dense(model_dim, kernel_initializer=initializer)

    def split_heads(self, x, batch_size):
        x = tf.reshape(x, (batch_size, -1, self.n_heads, self.head_dim))
        return tf.transpose(x, perm=[0, 2, 1, 3])

    def call(self, q, k, v, mask=None):
        batch_size = tf.shape(q)[0]

        q = self.wq(q)
        k = self.wk(k)
        v = self.wv(v)

        q = self.split_heads(q, batch_size)
        k = self.split_heads(k, batch_size)
        v = self.split_heads(v, batch_size)

        dh = tf.cast(self.head_dim, tf.float32)
        qk = tf.matmul(q, k, transpose_b=True)
        scaled_qk =  qk / tf.math.sqrt(dh)

        if mask is not None:
            scaled_qk += (mask * -1e9)

        attn = self.dropout1(tf.nn.softmax(scaled_qk, axis=-1))
        attn = tf.matmul(attn, v)

        attn = tf.transpose(attn, perm=[0, 2, 1, 3])
        original_size_attention = tf.reshape(attn, (batch_size, -1, self.model_dim))

        output = self.dropout2(self.wo(original_size_attention))
        return output
class TransformerBlock(layers.Layer):
    def __init__(self, emb_dim, n_heads, mlp_dim,
                 rate=0.1, initializer='glorot_uniform', eps=1e-6, activation='gelu'):
        super(TransformerBlock, self).__init__()
        self.attn = MultiHeadAttention(emb_dim, n_heads, initializer=initializer)
        self.mlp = tf.keras.Sequential([
            layers.Dense(mlp_dim, activation=activation, kernel_initializer=initializer),
            layers.Dense(emb_dim, kernel_initializer=initializer),
            layers.Dropout(rate)
        ])
        self.ln1 = layers.LayerNormalization(epsilon=eps)
        self.ln2 = layers.LayerNormalization(epsilon=eps)

    def call(self, inputs, mask=None):
        x = self.ln1(inputs)
        x = inputs + self.attn(x, x, x, mask)
        x = x + self.mlp(self.ln2(x))
        return x

So, here I see that they basically feed in the entire sequence into the Multi-Head attention:
x = self.ln1(inputs)
x = inputs + self.attn(x, x, x, mask)
But again, sequences can be different length, but linear layers accept inputs of fixed length. Is it me not understanding something? I also read about Padding and Masking, that is used during training. Is it also used during inference?

TL;DR YES.

If the sequence length of $Q, K, V$ is $L$, the embedding size is $E$, and the number of heads is $H$, then weight matrices are of the order $E \times (E // H)$ to transform $E$-sized embeddings to $(E // H)$ sized embeddings. We will have $H$ such embeddings; we get the same $E$-sized vectors upon concatenation. The attention weights will be of size $L \times L$ to enforce every token is attending to every other token in self-attention mode.

During the training, we would process the entire sequence at once and train the Transformer with the teacher-forcing algorithm usually. If the input sequences vary in length, we would truncate or pad them to a same length (of 512 tokens for instance).
During the inference, the generation of the output sequence is done via ancestral sampling. We start with a special token (SOS - Start-of-sequence) and sample the next token, and then feed this new token back in again and continue.

Yes, the query, key and value are the entire sequence. If the sequence is smaller than the maximum sequence length (which is the size of the linear layers), then you use padding tokens to complete the sequence and masking in the attention. If it's longer you cannot process the entire sequence with a standard self attention layer.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Real world example of an Knowledge based system
I was learning knowledge representation and reasoning in AI, in which till now, I have studied predicate logic and first order logic. Forward chaining, Backward chaining, Refutation resolution method etc. for proofs.
What I didn't understand is, is there any real world example where this kind of AI is being used ?
By AI, I have always heard of Mainly machine learning (with its subclasses defined as supervised, unsupervised and reinforcement learning).
Can anyone give an example if there is any knowledge based AI system which uses this type of algorithms / rules for inference. Or is this field just outdated for today's scenerios ?

Knowledge representation is used extensively in the bioinformatics community where ontologies are used to infer classification hierarchies in the annotation of biological data. See for example the Ontology Lookup Service (OLS) (Disclaimer: I am a maintainer of OLS).
As an example, look at liver disease. On the righthand side you will see some of the axioms used in its definition. These are axioms used in inferring the hierarchical structure of the ontology and the placement of liver disease in the hierarchy :

How is this useful? When a user sees that some data is annotated with EFO_0001421 (liver disease) they can get additional context information from the ontology. I.e., that liver disease is considered to be a digestive system disease and an endocrine system disease. Moreover, by looking at the children of liver disease, they can find all the diseases that are considered to be liver diseases.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
My small BERT can't even overfit on a sentiment analysis task
I'm trying to train (from scratch) a miniature BERT model on SST2, a simple binary sentiment analysis task with inputs of maybe 5-20 words at a time. As you can see in my code, my approach is a little non-standard in a few ways:

The model is quite small, 4 attention heads of 64 dimensions each and only four transformer layers.
I'm not using a tokenizer, i.e. I'm just using unicode code points as tokens, and restricting the vocabulary to just the characters in this dataset (so my vocabulary is about 50 tokens).

Nevertheless, I'm surprised that my model can't even seem to overfit on the training set. After something like 10 epochs, I see no drop in training loss at all. Should I just train longer? Am I doing something fundamentally wrong?
import datasets
from tokenizers.normalizers import BertNormalizer
from transformers.models.bert import BertForSequenceClassification, BertConfig

import torch
from torch.nn.functional import cross_entropy
from torch.optim import Adam
from torch.utils.data import DataLoader

def normalize(sentence):
    normalized = normalizer.normalize_str(sentence)
    normalized = normalized.replace("æ", "ae")
    return normalized

sst2 = datasets.load_dataset("sst2")

normalizer = BertNormalizer(
    clean_text=False,
    handle_chinese_chars=False,
    strip_accents=True,
    lowercase=True
)

train_text = sst2["train"]["sentence"]
test_text  = sst2["test"]["sentence"]
characters = set()
max_sentence_length = 0
for sentence in [*train_text, *test_text]:
    normalized = normalize(sentence)
    characters.update(normalized)
    max_sentence_length = max(max_sentence_length, len(normalized))

character_to_id = {}
for index, character in enumerate(characters):
    character_to_id[character] = index

n_vocabulary = len(characters)
n_heads = 4
dimension = 64
n_layers = 4

config = BertConfig(
    vocab_size=n_vocabulary,
    num_hidden_layers=n_layers,
    num_attention_heads=n_heads,
    hidden_size=n_heads * dimension,
    intermediate_size=256
)
model = BertForSequenceClassification(config).to("cuda")

n_train = len(train_text)
token_ids = torch.empty((n_train, max_sentence_length), dtype=torch.int32)
attention_mask = torch.zeros((n_train, max_sentence_length))
for i in range(len(train_text)):
    normalized = normalize(train_text[i])
    for j, c in enumerate(normalized):
        if j >= max_sentence_length:
            token_ids = torch.cat((token_ids, torch.empty(n_train, 1)), dim=1)
            attention_mask = torch.cat((attention_mask, torch.zeros(n_train, 1)), dim=1)
            max_sentence_length += 1
        token_ids[i, j] = character_to_id[c]
    attention_mask[i, :j + 1] = 1

train_x = token_ids.to("cuda")
train_y = torch.tensor(sst2["train"]["label"]).to("cuda")
train_mask = attention_mask.to("cuda")

batch_size = 128
loader = DataLoader(
    list(zip(train_x, train_y, train_mask)),
    batch_size=batch_size
)

optimizer = Adam(model.parameters())
initial_loss = None
while True:
    for batch_x, batch_y, batch_mask in loader:
        outputs = model(input_ids=batch_x, attention_mask=batch_mask)
        loss = cross_entropy(outputs.logits, batch_y)
        if initial_loss is None:
            initial_loss = float(loss)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        print(f"{float(loss):.4} ({100*float(loss)/initial_loss:.2}%)     ", end="\r")
    print("epoch")
```

How did you decide 10 epochs and 4 Transformers?

Regarding 4 Transformers, is "num_hidden_layers" equaling 4?  When I see the document, it looks like the number of layer in the Transformer (encoder) not 4 Transformers...(document: https://huggingface.co/docs/transformers/model_doc/bert)

One obvious issue with my script is that I never called model.train() (I'm not sure what effect this has with the BERT model). I also found that the batch size can be reduced a little, which speeds things up. However, the real problem seems to be that my BERT model is too big.
Using updated code with the above changes made, I ran an experiment comparing two versions of the model, a larger one (same parameters as above) and a smaller one:

Large model: 4 heads, 64 dimensions per head, FFN dimension of 256 and 4 layers
Small model: 4 heads, 16 dimensions per head, FFN dimension of 16 and 4 layers

Both models are trained with a batch size of 64, and I see the following relative training loss curves (i.e. proportion of mean epoch training loss to initial training loss):
epoch | 1       2       3       4       5       6
------------------------------------------------------
large | 99.07%  99.03%  98.99%  98.98%  98.98%  98.94%
small | 96.7%   94.8%   93.6%   92.4%   91.5%   90.7%

The large model is definitely converging, but very slowly (and its convergence rate would have seemed even slower with the larger batch size). By the way, the smaller model does generalize too, with about 58% test accuracy after 6 epochs.
import datasets
from tokenizers.normalizers import BertNormalizer
from transformers.models.bert import BertForSequenceClassification, BertConfig

import torch
from torch.nn.functional import cross_entropy
from torch.optim import Adam
from torch.utils.data import DataLoader

def normalize(sentence):
    normalized = normalizer.normalize_str(sentence)
    normalized = normalized.replace("æ", "ae")
    return normalized

def preprocess(dataset):
    sentences = dataset["sentence"]
    n_data = len(sentences)
    max_sentence_length = 1
    token_ids = torch.empty((n_data, max_sentence_length), dtype=torch.int32)
    attention_mask = torch.zeros((n_data, max_sentence_length))
    for i in range(len(sentences)):
        normalized = normalize(sentences[i])
        for j, c in enumerate(normalized):
            if j >= max_sentence_length:
                token_ids = torch.cat((token_ids, torch.zeros(n_data, 1, dtype=torch.int32)), dim=1)
                attention_mask = torch.cat((attention_mask, torch.zeros(n_data, 1)), dim=1)
                max_sentence_length += 1
            token_ids[i, j] = character_to_id[c]
        attention_mask[i, :j + 1] = 1

    x = token_ids.to("cuda")
    y = torch.tensor(dataset["label"]).to("cuda")
    mask = attention_mask.to("cuda")

    return x, y, mask

print("Preprocessing data")

sst2 = datasets.load_dataset("sst2")

normalizer = BertNormalizer(
    clean_text=False,
    handle_chinese_chars=False,
    strip_accents=True,
    lowercase=True
)

train_text = sst2["train"]["sentence"]
test_text  = sst2["validation"]["sentence"]
characters = set()
max_sentence_length = 0
for sentence in [*train_text, *test_text]:
    normalized = normalize(sentence)
    characters.update(normalized)

character_to_id = {}
for index, character in enumerate(characters):
    character_to_id[character] = index

train_x, train_y, train_mask = preprocess(sst2["train"])
test_x, test_y, test_mask = preprocess(sst2["validation"])

n_vocabulary = len(characters)
n_heads = 4
head_dimension = 64
ffn_dimension = 256
n_layers = 4

print("Loading model")
print("  n attention heads:", n_heads)
print("  head dimension:", head_dimension)
print("  n layers:", n_layers)
print("  ffn dimension:", ffn_dimension)

config = BertConfig(
    vocab_size=n_vocabulary,
    num_hidden_layers=n_layers,
    num_attention_heads=n_heads,
    hidden_size=n_heads * head_dimension,
    intermediate_size=ffn_dimension
)
model = BertForSequenceClassification(config).to("cuda")
model.train()

batch_size = 64
print("  batch size:", batch_size)
loader = DataLoader(
    list(zip(train_x, train_y, train_mask)),
    batch_size=batch_size
)
print()

n_batches = len(loader)

optimizer = Adam(model.parameters())
initial_loss = None
epoch_number = 1
while True:
    print("epoch", epoch_number)
    batch_index = 0
    mean_loss = 0
    for batch_x, batch_y, batch_mask in loader:
        outputs = model(input_ids=batch_x, attention_mask=batch_mask)
        loss = cross_entropy(outputs.logits, batch_y)
        if initial_loss is None:
            initial_loss = float(loss)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        mean_loss += float(loss) / n_batches

        batch_index += 1
        print(f"  {batch_index}/{n_batches}     ", end="\r")

    epoch_number += 1

    with torch.no_grad():
        outputs = model(input_ids=test_x, attention_mask=test_mask)
    predictions = outputs.logits.argmax(dim=1)
    test_accuracy = (predictions == test_y).sum() / predictions.shape[0]
    print(f"  train loss: {mean_loss:.4}")
    print(f"  relative loss: {100*(mean_loss/initial_loss):.4}%")
    print(f"  test accuracy: {100*test_accuracy:.4}%")

    # print(" ", train_loss)
    # print(" ", test_accuracy)
```


text:
Unclear point in derivation of action-value function
I did not understand how third equality follows from the second equality. Could some expert explain?

As $q_{\pi}(s,a)$ is defined in S&B RL book as follows:

we define the value of taking action a in state s under a policy $\pi$, denoted $q_{\pi}(s,a)$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$

Therefore similar to state value, your highlighted step here is trying to represent your action value's $\mathbb{E}_{\pi}[G_{t+1}|S_t=s,A_t=a]$ component by the state values of all possible entering states $s'$ after taking action $a$ following the same policy $\pi$ thereafter. As for the other component involving $R_{t+1}$ you've probably already known it's only determined by the environment under above conditions.


text:
Discrepancies in the Exclusion of Elements in Image vs. Text Generation
(The following comments concern DALLE, I have not tested it with other image generating tools, but would be curious to hear if the same happens)
When generating images, it seems that ChatGPT (i.e. DALLE) fails to exclude specific elements from an image.
As an example, here is the image I got based on the simple prompt:

Generate a picture of a zoo, but make sure that no elephants are in the picture.

 Here is the prompt that ChatGPT generated and used for image generation:
 "A bustling zoo scene filled with a variety of animals such as giraffes, zebras, lions, and monkeys. The zoo is lively with visitors walking around, looking at different animal exhibits. There are trees, benches, and pathways that lead to different sections of the zoo. Food stands and souvenir shops are scattered throughout, adding to the busy atmosphere. The sky is clear and sunny, perfect for a day out. Make sure there are no elephants in the picture".

When it comes to text generation, this does not seem to be an issue. Using the prompt

Write a small poem about a zoo, but make sure that no elephants are mentioned.

I have never gotten a result that mentioned elephants. If the comment concerning elephants is left out however, most poems will talk about animals in the zoo and then also mention them.
Given the observed challenges in the exclusion of elements (like elephants) in pictures, but the relative ease of that for texts, what are the fundamental reasons for this? Is this due to some inherent differences in training or model architecture in both applications?

The difference is due to qualities of text training data used in training the two types of model.
The LLM models are trained on large amounts of text, and learn grammar rules very well. There are lots of examples of negation in the training data. The trained models can process what negation means in phrases, and what that may mean for the following phrases, i.e. what they are likely to generate when a term such as "no elephants" is in the previous context.
The image models are trained on images paired with image descriptions. The text is generally simpler, and there is less of it overall than with LLM training data (hundreds of millions of sentences instead of hundreds of billions). Part of the image training process involves creating a shared multimodal "concept space" where text and image descriptions both resolve to the same descriptive vector*. This allows the generator to measure how far away a generation in progress is from an image with the same description as its text input. It can then use that to adjust the image pixels towards something with a more similar concept vector to the text.
The thing that is generally missing from the image/text pairs is negation. It is rare to have images described by what is not in them, and definitely not text generally covering all the details in a complex image - like your zoo image - that are not there. There are therefore very few places in the "concept space" which cover the negation of some element or other. In fact you often get the opposite effect in a generator like DALL.E - the more you mention an element, positively or negatively, the more prominently it will appear in the image.
This issue is common across image generators. In Google's blog about their model Parti, they give an example of this as one of the failure modes.
Image generators can and do allow for this. They allow the use of negative prompts (Stable Diffusion), or prompt weights which can be negative (Midjourney). These work by adding the negation of the concept vector, based on the text in the separate negative prompt. The image, whilst generating, will be adjusted to not be like, or not contain, the concept in the negative prompt.
I don't know why DALL.E does not support the use of negative prompts.

* It's more complex than that, what I describe is very roughly how CLIP guidance works.


text:
Is the sequence 1-1-2-3-Exit possible in the following Markov reward process?
Is the sequence 1-1-2-3-Exit possible in the following Markov reward process?
The probability of transitioning from state 1 to itself is 0.
Source: https://maelfabien.github.io/rl/RL_2/#markov-process-mp

No, it's not possible, and you correctly explained why.


text:
How do I code so that the embedding output and input share the same weight matrices?
I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In
our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [30]" [30] here is the paper Using the output embedding to improve language models by Ofir Press and Lior Wolf.
Excuse me but

could you explain why we need output embedding between encoder stack and the decoder stack? Isn't the output of encoder d_model dimension vectors (d_model = 512 in the original paper) so they don't need embedding layer in-between? Can I treat this as a linear layer?

could you explain how the weights can be shared between the pre-softmax linear layer and the input embedding? Since in the original paper, it was about the translation between German and English, shouldn't the weight matrices be different for pre-softmax linear layer and the input embedding since the number of vocabularies for two languages aren't the same?

Thank you for reading my question!

There are three weight matrices that can be shared:

The weights of the embedding layer for the encoder -- that is the layer that embeds the source sequence before forwarding through the transformer encoders. If your source vocab has size $V_\text{src}$ then this linear layer will have weight matrix of shape $ V_\text{src} \times d_\text{model}$.
The weights of the embedding layer for the decoder --  If your target vocab has size $V_\text{tgt}$ then this linear layer will have weight matrix of shape $ V_\text{tgt} \times d_\text{model}$.
The weights of the final linear layer that maps from the decoder embeddings into your target vocabulary space. The weight matrix of this layer has shape $ d_\text{model} \times V_\text{tgt}$

So regarding your first question:
There is nothing that you need to do with the output of the encoder.
Regarding your second question:
Since matrix 2 and matrix 3 have the same shapes you can easily share the weights. Matrix 3 is just the transpose of matrix 2.
Sharing the same weights between matrix 1 and matrix 2 and 3 is only possible if your source vocabulary matches the target vocabulary. So this makes perfect sense for tasks like text summarization or question answering, where both sequences share the same vocabulary.
In the case of machine translation where you have different vocabularies, this doesn't make much sense at first glance. But note that most of the times your are not using a word vocabulary, but a sub-word vocabulary (think character-level encodings or byte pair encodings). Now if you take a look at Table.1 in the paper from Ofir Press nad  Lior Wolf, you will see that for translating between French and English (or German and English) almost 80% of the sub-words appear in both dictionaries. So what they do is they use a single vocabulary $V_\text{total}$ that contains all subwords that appear in both languages.
I really don't think that this idea makes sense for languages where there is very little overlapping of words / sub-words / characters, e.g. translating between English and Chinese.
If you want to see how this could be implemented you can check out the code snippets that I wrote here:
https://pi-tau.github.io/posts/transformer/#token-embedding-layer
In this implementation not only the weight matrices are shared, but also the weights for the positional encodings.


text:
In a known finite episodic task, is there still value to discount?
Sutton and Barto explains that in an continuing task, $\gamma$ is used to make the reward not infinite.  In a known finite episodic task, does $\gamma$ retain any value?  It is said that changing $\gamma$ changes how near-sighted an agent is but that seems more like hyperparameterization to suit a given task, not a specific strength.
For example:
If we consider a multi-arm bandit with two states, a $\gamma$ of zero is fine because one only needs to maximize the present decision.  If the agent was only given 10000 pulls, a $\gamma$ of 1 might be noisier but the actions in a state should approach their actual values on a long enough timeline.

Yes. The use of discount factor helps reduce the variance when computing the multi-step return. Future rewards which are more uncertain receive a higher discount.

Your overall reasoning sounds reasonable. In a known finite episodic task, while the choice of $\gamma$ could be seen as a global hyperparameter as you narrated, the impact of $\gamma$ is more localized within each episode and its choice is to influence the agent's exploration-exploitation tradeoff based on the characteristics of its specific task and the environment. There is no one-size-fits-all value for $\gamma$. For instance, a short-term swing trader's $\gamma$ must be smaller than a long term investor's, on the other hand a hyperparameter of a model is generally systematically tuned by some grid search like method to achieve possibly best training and test performance without such prior insight.


text:
Can Conditional Normalizing Flows Cheat?
Consider the Conditional Normalizing Flow (informally): $f(x, z) = y$ with its inverse $f^{-1}(x, y) = z$ where $x$ is the prior and $z$ is the posterior.
Is $f^{-1}$ able to "cheat" by leaking information about $y$ directly into $z$, or do the theoretical guarantees of normalizing flows side-step this?
I want to use $z$ as a way to represent anything about $y$ that can't be deduced from only $x$ (similar to BYOL-Hindsight) so that I can randomly sample from $z$ for inference.

Formally, the functions of your conditional normalizing flow are $f(y, x) = z$ and $f^{-1}(z, x) = y$, where we model the likelihood of $y$ conditional on $x$.
The probability density can be expressed as:
$$
p_{Y \vert X}(y \vert x) = p_{Z \vert X}(z \vert x) \vert J \vert
$$
In optimizing the $p_{Y \vert X}$ term of the objective, the model learns $z$ to represent how "unpredictable" $y$ is, being incentivised to learn the highest probability that doesn't lose any information.

Is $f^{−1}$ able to "cheat" by leaking information about $y$ directly into $z$, or do the theoretical guarantees of normalizing flows side-step this?

Now, when passed through the inverse, only the information about $y$ that it "couldn't have known" is preserved through $z$, which is what $f^{-1}$ receives.
Conditional normalizing flows might not be what you're looking for in adapting BYOL-Hindsight because the likelihood consists of both what it doesn't know and what it knows it can't know. Having the actor look for places with lower probability would likely reproduce the noisy tv problem.
Perhaps you could instead use it to task the agent to find states with high probability, that the model still doesn't predict well.


text:
What are reward networks in reinforcement learning?
I am reading the following article given over here -

The goal of both inverse reinforcement learning (IRL) algorithms (e.g.
AIRL, GAIL) and preference comparison is to discover a reward
function. In imitation learning, these discovered rewards are
parameterized by reward networks.

Does this mean that the output of imitation learning is a reward value? What are the inputs then? The state and action?
I am currently referring to Levine's lecture notes. At slide 5, it seems like the output of imitation learning is a policy and not a reward.
I'd appreciate an answer that refers to papers while explaining these concepts.

In RL, reward networks (also called reward models), say $r_\theta$, have the sole role of learning (approximating) the reward function, defined as $r(s,a)$ or $r(s, a,s')$, in which the latter formulation is more general. A reward function, $r$, takes the current state $s$, selected action $a$ (by a policy, expert trajectories, or behavioral policy), and optionally the next state $s'$, to output the immediate reward, $r_t$, of timestep $t$. Reward networks should approximate $r_t$, and are employed in inverse reinforcement learning (IRL) and sometimes also in model-based RL.
Moreover, you should not confuse IRL with imitation learning (IL), as IL is more general because it also includes supervised learning methods such as behavioral cloning. In IRL, you want to learn a reward function first that is usually parameterized by a NN (therefore you have a reward network), and then use standard model-free algorithms from the learned reward to learn the policy. This is the general setting, but GAIL for example can learn the policy directly.
Note also that when you do IRL or IL you don't have access to the environment, and therefore you don't know the true reward function, but you have only access to a finite number of samples (called demonstrations) which can be from either an expert, exploratory, or sub-optimal policy.
You can refer to this survey for a comprehensive introduction about IRL, its foundations, flavors, and applications.

One representative paper for imitation learning is Ho & Ermon's "Generative Adversarial Imitation Learning" (2016). The goal in imitation learning is to learn a policy that can mimic the expert's behavior.

Consider learning a policy from example expert behavior, without interaction
with the expert or access to reinforcement signal. One approach is to recover the
expert’s cost function with inverse reinforcement learning, then extract a policy
from that cost function with reinforcement learning. This approach is indirect
and can be slow. We propose a new general framework for directly extracting a
policy from data, as if it were obtained by reinforcement learning following inverse
reinforcement learning.

Behavioral cloning, while appealingly simple, only tends to succeed with large amounts of data, due to compounding error caused by covariate shift [23, 24]. Inverse reinforcement learning (IRL), on the other hand, learns a cost function that prioritizes entire trajectories over others, so compounding error, a problem for methods that fit single-timestep decisions, is not an issue... Unfortunately, many IRL algorithms are extremely expensive to run, requiring reinforcement learning
in an inner loop.

We desire an algorithm that tells us explicitly how to act by directly learning a policy. To develop such an algorithm, we begin in Section 3, where we characterize the policy given by running reinforcement learning on a cost function learned by maximum causal entropy IRL [31, 32]... The discriminator network can be interpreted as a local cost function providing learning signal to the policy—specifically, taking a policy step that decreases expected cost with respect to the cost function.

Therefore GAIL is mainly imitation learning which learns a policy as its ultimate goal, yet it also has a discriminator network similar to GANs that acts as a local cost function and aims to distinguish expert trajectories against learned policy's trajectories, establishing a more rapid framework for simultaneously learning policy and reward from data due to its adversarial nature and bypassing an additional independent IRL step.

I think the reason why some refer to learning a reward policy and others learning an actor policy is that both are possible.
For example, GAIL: A discriminator (the reward policy) learns to tell apart expert trajectories and the actor's trajectories, and the actor tries to fool the discriminator by learning to imitate expert trajectories.
In this way, both a reward policy and actor policy are learnt simultaneously.


text:
MDP and a given policy and the correctness of the state-value function
Is the following statement correct?
"For an MDP and a given policy, the Bellman equation can be used to check the correctness of the state-value function."

What do you mean by ‘correctness’?

Generally speaking it's correct.  Bellman equation is a set of linear equations rather than a single equation for the usual tabular cases. If we put these equations together, it would be clear how to calculate all the state values. See S&B book page 60:

The value function $v_{\pi}$ is the unique solution to its Bellman equation.

$v$ is the state-value function and $\pi$ is a given policy as you specified.


text:
Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem?
Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem? I need a clear explanation.

Can you please tell us where you took this image from?

Like it's explained in the image's caption, DP techniques are based on the idea that you can reuse the solution to subproblems, so it assumes that you can break down the original problem into subproblems and that some subproblems overlap.
So, in your example, you can get to $B$ from $C$ and $A$. Similarly, you can get to $E$ from $C$ and $D$. Once you know the shortest path from $B$ to $Y$, you don't have to recalculate it. The same applies for $E$. So, to be clearer, the shortest path to $C$ and $A$ share a common subproblem because they are both connected to $B$. The same applies to the other example.
Now, if you look at your example, there are 10 segments, so you don't go through a segment more than once. If you start from $Y$, you can compute the solution in a bottom-up approach (see also the top-down approach in DP), and cache the results to any subproblem, so you don't have to re-evaluate anything.
In the case of exhaustive search, you basically try every possible combination and you don't cache anything, so you may revisit a segment more than once. For example, you will visit the segment from $B$ to $Y$ twice because there are 2 paths that have that segment, i.e. $X \rightarrow A \rightarrow B \rightarrow Y$ and $X \rightarrow C \rightarrow B \rightarrow Y$. You also visit the segment $X \rightarrow C$ several times because several paths have it. Try to enumerate all possible paths and then count all segments (even if repeated) - they should be 14 (I've just counted them, but you can count them yourself too)

Exhaustive search is just trial and error for the specified problem which has a fixed start node X and a fixed end node Y, thus every possible travel route from X to Y has to be evaluated independently which requires 14 segment evaluations of their respective reward (i.e., the negative of cost-to-go length for shortest path problem) if you count the total 5 possible ways from X to Y in your directed graph.
On the other hand, if you use model-based RL there're only 7 states as nodes and obviously its MDP's state and reward transitions are all deterministic with at most 3 actions at each state and only one action at some of the states. So by using backward dynamic programing's (DP) simplest one-sweep value iteration for this shortest path problem starting from the terminal node, the problem is decomposed into smaller subproblems represented by individual states. In many cases such as shortest path here, states closer to the terminal state have fewer successor states to consider, reducing the computational complexity and iterations to be able to directly compute optimal values within one iteration here instead of the generic algo's intermediate  estimates with many iterations as mentioned by my answer to your recent post.
And since you've already known the one-line max operator algo for value iteration derived from Bellman optimality equation (not Bellman equation) in your recent post, it's easy to check for yourself here the optimal values for the 3 states closest to terminal state Y are $v_*(B)=-2, v_*(E)=-2, v_*(C)=-1$ assuming the terminal optimal value $v_*(Y)=0$, and you only need 5 segment evaluations to arrive at these 3 optimal values (note to calculate state C's optimal value we can reuse optimal values of B & E reducing the segment evaluation count essentially). Then similarly you just need another 5 segment evaluations to get the remaining 3 states' optimal values. Then you can extract shortest path from start state X by following the action with optimal value stored already in the above described step-by-step backward DP.


text:
Export trained model offline to be used by an application
I'm trying to create a text-based game using AI. I was playing around with text classification AutoML from Vertex AI just to learn AI, and then pick the best solution for my use case. Is it possible that I can train my data online (using any cloud provider) using a text classification model, and then export the model already trained (in the form of a script, I don't know how the output is) to be just consumed inside an offline game application? All the options I see so far use API endpoints, but I want it to be totally offline, so each game client has his own version of the trained model and the dataset.
Thanks

Is it possible that I can train my data online (using any cloud provider) using a text classification model, and then export the model already trained (in the form of a script, I don't know how the output is) to be just consumed inside an offline game application?

Yes this is possible in principle, and commonly done. All mainstream machine learning libraries, such as TensforFlow, PyTorch etc, support saving model parameters to disk and loading them later, perhaps on another machine. Even within a single library there can multiple formats that this could work in, depending on whether you want to save raw parameters and code the model's structure in a script, or have the full model structure and parameters encoded in the saved file.
The downloaded model most likely will not run itself - saved files from trained ML are typically data only, not executables. You will need to write code within your game client that loads the model, and runs it when required.
The specifics of how to do this depend on the libraries and services you are using. But in principle this is no different from creating any other asset online, and transferring it to use in some other software. For instance, the same issues would arise if your service helped you design 3D assets for a graphical game.
You will need to research what is available to you in your chosen language and service (or vice-versa, look for choices that make this part work best for you). There is no one-size-fits-all solution.
In addition, you may want to check what resources you will have available on game clients, and whether the model can be made to run efficiently given those resources. Dedicated machine learning training services typically have fast GPUs to run on. When you are no longer training, and the model only needs to perform inference, which uses less resources than training, then simply using the CPU may still be fast enough for your game. It seems you are not creating something that needs to run fast in real time, so probably this will be fine, but it should be something you check early on.
There is also nothing stopping you using an API-based service to access the trained model. This has very different characteristics, including needing to account for the cost of the service, but for some use cases could still be a good fit. For example for large models that need powerful machines to run, this may be the only choice that allows clients to use the model.


text:
Why are there up to $m^2$ action values when we consider the complexity of DP based on $q(x,u)$?
Please see slide 32 in the following lecture slides on DP:
https://groups.uni-paderborn.de/lea/share/lehre/reinforcementlearning/lecture_slides/built/Lecture03.pdf
Let $m$ the size size of action space in MPD.
Why are there up to $m^2$ action values when we consider the complexity of DP based on $q(x,u)$ ?

To fully understand generically the transition dynamics from state $x$ to state $x′$ in terms of action values without state values involvement, we need to evaluate all possible combinations of actions for both states via their respective action values $q(x,u),q(x',u')$. Since for each state there are at most $m$ possible actions, this means there are at most $m^2$ possible combinations of actions for every pair of states $x$ and $x′$s. Furthermore there're at most $n^2$ state transition pairs obviously, therefore you arrive at complexity is inferior with $O(m^2·n^2)$.


text:
Is it possible to use Mini-Batches with Adam optimization?
Is it possible/advised to use Mini-Batch like accumulation with Adam optimization?
How would that works?
Do I accumulate the loss function for each sample in the batch and then run Adam, or should I divide the Loss by Number of samples in batch before running Adam?

Adam works best with mid-sized mini batches.
Too small batches can generate too much sampling noise, making Adam less stable than basic stochastic gradient descent.
Too large batches remove the computational advantages of using mini batches in the first place and slow learning.
Important: For gradient estimates in neural networks you cannot simply accumulate or average loss values. You must calculate the full gradient of the loss for each example (using back propagation). You can then aggregate and take the mean of these high dimension gradients.
Accumulating or taking mean gradients are almost the same thing in Adam, provided the batch size remains constant. That's because Adam divides all gradient components by running averages.
However, using the mean gradient is closer conceptually to what is going on: The mean gradient of a mini batch is an approximate estimate of the full batch mean gradient. Adam then processes a sequence of these estimates into update steps that roughly account for second-order gradients.
You can use Adam (and other gradient-based optimisers) separate to sampled data. E.g. when you want to find a minimum of a function that you know, instead of a mini batch, calculate the true gradient at a current position and use that gradient as the input to Adam. If you have not done that before, it's worth experimenting with for multiple gradient descent accelerators, because it helps understand how they are separate to sampling and how they work


text:
A question about the reward calculation in the Hindsight Experience Replay algorithm
I' m try to implement the HER algorithm from scratch in order to use it in the PandaReach-v3 environment.
I already developed the same algorithm for the bitflip environment and it works as expected.
So, what's now the problem?
The problem is the calculation of the new reward $r'$ given an additional goal as stated in the red box in the following picture:

The problem is, that while in the bitflip environment I just created a trivial function, that basically calculates the difference between the state $s_{t}$ and the goal $g'$, in the case of a more complex problem like the PandaReach I don't kwon how to proceed, since I cannot figure out, how the function for the calculation of the new reward given a  new goal should look like.
Basically I came up with the following two ideas:

I could implement a function, which calculates the 3D distance between the state and the goal. The function could output a one, if the calculated distance is below a small $\epsilon$ or zero otherwise;

I could use the already implemented step method (from the environment), which delivers in one step the new calculated reward. So no effort from my side. But here the problem is that I need to pass an action to the step method in order to generate a new state and calculate the new reward. And, for a new action, I need a policy. Since HER is a off policy algorithm I have a behavioural policy $\pi_{b}$ and nothing else. So my concern is whether I can/should/may use the same behavioural policy to sample an action to feed into the step method and get the new reward.

In other words: how do you calculate the new reward $r'$ in the HER algorithm?
Many thanks

HER typically assumes you have access to a reward function that can evaluate whether a certain state achieves the goal or not. If you can implement 1. then that should work.

Meanwhile I can answer my own question.
After figuring it out, what the HER algorithm really needs, I found out, that the right answer is 1: you need a reward function for calculating the new reward and the new done flag.
In case you are at the first tries and you are trying to implement it using the panda-gym environment, this is what I did:
Inside the Panda environment there is a compute_reward function, which can be used to calculate the reward. Just pass the actual state and the new goal to that function.
In code:
import gymnasium as gyms import panda_gym

...
env = gyms.make('PandaReach-v3')

...

# The compute_reward functions need an 'info' dictionary as third argument.
# Just pass an empty dictionary there.
# 'next_state' should replace with the variable you need. Since in my case
# I want to calculate the difference between 3 spatial coordinates, so I'm going to take 'next_state' and 'new_goal'. Both are 3D spatial coordinates
new_reward = env.unwrapped.compute_reward(next_state, new_goal, {})
# Or in case you are not using any wrapper new_reward =
env.computer_reward(next_state, new_goal, {})


text:
Use your own simulation to train a reinforcement learning multi-agent
I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a reinforcement learning environment to rebuild my simulation, train the model, and then extract that agent to be used in the simulation? My preferred route would be to change the simulation to enable reinforcement learning, but I have not found examples of that on the web or in literature. Is that possible? Or must the route be that we recreate our simulation in an environment?

Please clarify - are you intending to write your own learning agent, and if so do you intend to do this in C++ (with e.g. TensorFlow C++ API for a neural network) or are you hoping to use something pre-built in Python (e.g. one of OpenAI's baseline implementations)?

Also important - can the environment and agent interact over reasonably well-defined time steps/decision points, e.g. when training it is possible to move the simulation forward a known amount and use that to generate next state, reward etc?

If I can use python, my main language, that would be fantastic. If there is some benefit to using C++, I would use that. I'm not sure of the difficulty of the task, perhaps I would start easy for prototyping, then go hard (Tensorflow C++ API) after that.

Regarding the second question, I believe this is possible. Also, I might add, I am planning to do multi-agent RL, with agents that can be of different categories (planes, cruise-liners, air-balloons, etc).

According to the question, your environment is written in C++. Does it have an interface that you can script in Python for inputs (agent actions) and outputs (observations of state)? That seems to be what your question is about - how to adapt an RL system around the existing environment. If there is no suitable interface, do you have the C++ source code and able to compile some Python bindings, or only the executable?

I do not have such an interface, but I do have the source code. I agree, that is the main question. Tho it seems the other answerer of this  question says you can just modify your simulation to add reinforcement learning. But on the interface question, I have the source code and can modify it on my branch. Not at all clear on how to do that technically, haven't really seen resources online about that, it seems like a niche thing. You search RL and get a million medium posts about cart-pole.

An environment is not a magical concept in Reinforcement Learning (RL).  When the textbooks were written on RL the python "gym" concept certainly did not exist much less during the time of some of the earlier research work.
An environment is anything which the agent can sense, interact with and receive feedback from.  One could throw an agent into a robotic body an let it learn from the real world in real time.  There's no need even for a simulation.  It's just common because we can run a simulation at faster than real time and/or spread learning across many agents.
So, for your simulation to be an "environment" it need only be modified to do those three things.  Give the agent an input it can sense, accept and enact the agent's actions and evaluate it's performance in the form of a reward signal.

I see. I had come to understand it would take an interface that would output a state vector. However, I would like to use a multi-agent RL technique, with varying types of units. One step would be full information, knowing the whole state of the environment to decide initial placements. And the future steps would be partial. It would have interesting things like deploying an energy capture device, with the reward delayed. But from what you say it seems like you would just modify or replace the current algorithms used to generate each units behavior.

So yes, the environment must present the agent with a state vector.  That is the "sensing" part.  One might have an agent which is the general, pilot and sailor all at once, sensing all of the information simultaneously.  But it is also possible for each to be their own agent, with their own collection of things they can sense.  Which is better is something to test.

What about a battleship, a plane, and a missile launcher, MARL can handle all of that? They'll all have partial information, communicated to each other in timesteps.
Then the only question I have is how do I do this. Is there a guide, a paper?

I don’t have a lot of MARL experience.  I found a couple interesting papers starting here, hope it helps.   https://arxiv.org/abs/1912.03558


text:
Can/should a reward function depend on something other than state in a DQN
Question: Is it OK to have a reward function on a DQN or any RL algorithm that depends on variables other than the enviroment state? I'm asking because, so far I'm learning from tutorials, but I've never seen a reward function working with something else than enviroment.
My Case: Imagine a trading bot. In order to get the reward, I'd neet to know the position (how much of a stock/cryptocurrency I have) to know if the bot won or lost money in the taken step. But I don't want to have current position on the enviroment's state as it is not important for calibration. Important variables in enviroment are other key market indicators.
I thought on having a separate variable with auxiliar data such as current position. So reward function would rely on new_state, previous_state and aux_data.
Is this OK? Bad practice?

Reward functions often depend on variables not present in the current state.  Trading is the perfect example.  Try as we might, we are unlikely to capture every variable necessary to perfectly predict the movement of a price and in effect the agent's reward.  This is not typically a desirable property for an environment to have.
But if as you say the present positions are not important for calibration then feel free to test the hypothesis.  I can drive a car without a speedometer but do not ask me to drive with a blindfold on.  Not every missing variable has disastrous consequences despite their effect on the reward.


text:
Q learning (DQN) strategy for a multiplayer zero-sum game
I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and I want to know if it would work. Here it is:
The agent learns through self-play. It receives a +1 reward for winning (no matter what player won, you'll see later why) and 0 for every other state. The agent learns through a modified Bellman equation, where the expected future reward is SUBTRACTED and not added.

Q(st, a) = r - γ * max Q(st+1, a)

If you expand this equation, you should get this:

Q(st, a) = rt - γ * rt+1 + γ2 * rt+2 - γ3 * rt+3 + ...

Notice how the plus and minus signs alternate. Thus, your expected reward is a sum of the rewards from your turns minus the sum of your opponent's rewards. That is also why the reward for winning is always +1.
This strategy is similar to the minimax tree search, where we want the best rewards for us and the worst for our opponent.
I hope I got my point across. Please note that I'm not very good at maths and AI theory, so forgive me if I mixed some things up or have a wrong notation. Also, if this is not a good approach, which would you recommend?

This works, and is used as a standard approach for two player zero-sum games in reinforcement learning. As you stated, it is a combination of reinforcement learning with Minimax optimisation.
A very similar approach is to have reward +1 for player A winning, and reward -1 for player B winning, with A's objective to maximise reward and B's objective to minimise reward, allowing them to share the same value function directly (with your formulation, the safest thing to do is maintain two value functions, although in some cases it may work with one). This approach leaves the definition of $Q(s,a)$ unchanged from standard RL, but changes the Bellman equation (and matching standard update rule) of $Q(s,a) = r + \gamma \text{min}_{a'}Q(s',a')$.
You may see another variation, where the action variable $a$ is dropped, and the agent selects the next state from all possibilities that the game engine allows. This is called the afterstate representation, and it works effectively for simple board games because it reduces the space needed for the value function compared to an action value approach.

Instead of having one player try to maximize the Q function and another trying to minimize it, could you just reverse the board (replace every O with an X and every X with an O) after every step, then get one player to choose their action based on the normal board and the other to choose an action based on the reversed board?

@Mas that would depend on the game. Probably not a good choice for TicTacToe because the reversed board would not be valid for O


text:
Does the DoubleDQN algorithm use a target network or two separate policies?
I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:

The Double DQN algorithm has two separate policies Q1 and Q2 with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.

Q1(st, a) = r + γ * Q2(st+1, argmax Q1(st+1))

The Double DQN algorithm works like a normal DQN with a policy and a target network (a copy of the policy lagging a few steps behind), and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.

Q(st, a) = r + γ * Qtarget(st+1, argmax Q(st+1))

Which one of these solutions is correct? If both of them, which one is "better"? Or is there a third option I haven't considered?
Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.

You can use mathjax on this site to format your formulas.

For the record I did a short experiment on DQN with two policies.  i.e. using 2 different policies with totally independent collection of history.  My results were not statistically different than DDQN with a target network.  Your results may vary.

The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the deep RL literature, but it looks like what one would do if they were to take the original Double Q-Learning algorithm and place it exactly into the Deep RL format. The reason they don't do this in the Double DQN paper is because it would be very expensive to train and maintain two networks, when you are using a target network anyway. I believe they found that the target network worked well enough at reducing the maximisation bias without much more computational cost to the original DQN.


text:
What is the difference between two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts?
Here, https://lrscy.github.io/2020/07/09/Coursera-Reinforcement-Learning-Course2-Week2-Notes
(See the "Monte Carlo Control" and then "Solutions of Two Assumptions"
sections)
two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts are given as follows:
(i) One is to hold firm to the idea of approximating $q_{\pi_k}$  in each policy evaluation. However, it is also likely to require far too many episodes to be useful in practice on any but the smallest problems.
(ii) Another one is similar to the idea of GPI. On each evaluation step we move the value function toward $q_{\pi_k}$ , but we do not expect to actually get close except over many steps. One extreme form of the idea is to alternatively apply policy improvement and policy evaluation.
I did not understand the difference between (i) and (ii). For me, (i) is also a form of GPI.

This lingering question is closely related to your recent question how to derive value iteration from policy iteration. Policy evaluation (PE) required in policy iteration method needs exact or approximated state or action value evaluation during each iteration, since it's based on the idea of policy improvement theorem (PIT) and Bellman equation, so it's the original and pure (not generalized) policy iteration. And you're right it could be said as a form of GPI.
On the other extreme end, value iteration is based on the idea of solving Bellman Optimal equation, on each policy update (PU) step we move the value function toward the exact $q_{\pi_k}$, but we do not expect to actually get close except over many steps. Therefore it's a kind of generalized policy iteration.

I was asking difference between (i) and (ii).

My first section above is for your case (i), and second section is for the extreme one-sweep case (ii) which is rightfully called GPI as I've explained during so called 'evaluation' step we do not expect to actually get close to true state/action values except over many steps, therefore it's a kind of generalized policy iteration.


text:
How to reconstruct a new image using pre-trained autoencoder?
When a single image is assigned for training, an auto-encoder should be able to gradient-descend and find the full set of satisfactory weights that will reconstruct this image.

Suppose a second image is now assigned to the autoencoder:

Can the previously trained autoencoder weights (for image 1) now be reused immediately (without further training) to reconstruct the second image as well?

If you do not pose constraints on the dimensionality of the encoded representation, you can cheat by imposing $d= |x|$ and learn an identity mapping of the input quite easily. Of course, this is a perfectly useless network since your encoded representation has the same dimensionality of the input and replicates the input exactly :)

@Ciodar thank you, yes I believe the identity mapping is what I was really after... Say suppose this auto-encoder is implemented as a fully-convolutional U-net type network (all weights are kernel values or bias), skipped connections, and relu slapped onto each stage's output, then is it certain that a specific set of weights that perform the identity mapping (that maps all images into itself pixel-to-pixel) can still be found after all those convolution steps and relu information truncation?

Look at a single residual block: if you feed $x$, all you need to do is make the network zero out everything (i.e all weights to zero), to obtain $x$ as output. Maybe this question is answering exactly to that.

The latent space representation learned by the VAE may not be meaningful or informative with only one input image. Reconstruction of the input image may not be accurate or meaningful since the VAE may simply learn to memorize the input image rather than learning useful latent features for reconstruction.
Even assuming it can be trained successfully with only one image as yours, it's quite impossible to achieve your goal since these 2 images have obviously very different appearances and features. For example, your trained image has a big hat covering a large portion of space which your target output image lacks. There're many other dissimilarities. By randomly sampling the learned latent space with only one image, you can only expect similarly generated image at best.

thank you! The reason I ask is I guess because while I'm convinced that a good set of weights can be found that reconstruct a single image, I have serious fundamental doubts whether/why there must exist a single set of weights that can perform the dual purpose of reconstructing exactly both image 1 and image 2 when given each image as input... Is it certain that 1 large enough model having sufficient number of weights can always reconstruct any number of input images exactly (auto-encode many images at once), assuming we can find the correct weights that can perform all this multitasking?

While in theory a sufficiently large AE may have the capacity to memorize individual training samples (reconstruct the input images exactly), the goal of ML is not memorization but generalization, that's why DAE/CAE exist. And VAE unlike AE cannot generate exactly same image since there's sampling uncertainty inherent. For your OP since the 2nd image is not in the training set, AE won't be able to reconstruct it exactly or even closely, as it hasn't learned the specific patterns and features present in that image which is very different from your trained image. You may try fine tuning though.


text:
How to perform latent space Interpolation between two images?
I have a variational convolutional autoencoder that has trained on 2 images and outputs a linear interpolation (inserted at the bottleneck stage) between those 2 input images.

However, the result looks (rather dissapointingly) like some Powerpoint cross-fade effect.
What would be needed to obtain a more novel transition effect, for instance limbs moving from '1' and seamlessly reshaping into '7'?

You need more training images. Far more, at least a few hundred, with variations. The latent space has no meaningful form to it when you train with just two end points. The decoder will have no examples of shifting forms that it could arrange on the path between typical 1s and typical 7s.
Also, if there are separate classes within the encoded space, representations in the middle are not guaranteed to be coherent transformations between them. More training examples will help, but some parts of the latent space will be odd. In human face examples you can often observe this when latent  space crosses between glasses and no glasses, the transition can fall apart.

thank you, I think I understand how morphing works now. Although, I still can't wrap my head around how smile replacement works (where a non-smiling face is replaced by a smiling face, but the rest of the face is unchanged). Is this also done through latent space pathfinding? But how to keep the rest of the face unchanged while changing only the smile around the mouth?

@James that would be a different question, but the principle is to try and extract the "smile vector" by using all the labelled examples. It's rarely perfect, and often other image features change too


text:
Wouldn't residual connections in RNNs solve the vanishing/exploding gradient problem?
I was recently brushing up on my deep-learning basics and came back to RNNs.  LSTMs/GRUs and the Transformer architecture were invented to solve RNN's vanishing/exploding gradient problem. I was at the time also interested in image processing and the ResNet model, so I immediately thought of incorporating the residual connection into RNNs. The idea is to connect the hidden states after each block (skipping the network and activations) to allow for better training of the past blocks. Given that nobody has done this yet, I feel the idea is somehow flawed and not as genius as my mind thinks. Why would/wouldn't this work? Where did I make a mistake?

In my opinion your idea indeed holds merit. Something worth noting though is that it is cruder than the LSTM/GRU that have trainable weights that guide what features are remembered and forgotten. There are also other architectures than ResNet that uses residuals, but add gate layers to guide what is passed on.
The main thing your idea would bring is likely speed, rather than accuracy. And that is not always a bad thing.
Here are some papers where the author uses a residuals in combination with LSTM:
Residual LSTM
Highway-LSTM

Your idea is exactly the idea behind state-space models. They have a linear "residual" connection from previous hidden states, skipping activations. In fact, it works very well! I'd recommend reading the Mamba paper. They get comparable results to "transformers" while being several times more efficient, and fixing the context-length issue.
There are two major issues with LSTMs, and in general RNNs, that make training slow:

Vanishing gradients (as you mentioned), and
Only being able to train one token in a sequence at once.

Because state-space models just have a linear "residual" connection, they can train an entire sequence of tokens at once with a parallel scan algorithm (similar to the prefix sum algorithm). Note that this ability to crunch huge amounts of data is what allowed "transformers" to take off.
The issue you mentioned is actually a little broader: if you run an RNN too long the outputs will either vanish or explode. You can imagine a purely linear model, $x\mapsto Ax$, which after $n$ steps will become $A^{n}x$. If the eigenvalues are too big, things explode, while if they are too small, $A^{n}x\to 0$. The authors of Mamba considered forcing $A$ to be unitary (eigenvalues have magnitude exactly one, see section B.3), and mention that other authors have explored it in more depth to resolve this issue.


text:
Loss goes down but never below a certain treshold
I made a neural network in C#, I observe the loss goes down but never below a certain treshold.
This is XOR function error graph:
(The graph is every 4 samples, so for all the 4 possible combinations of inputs there is one combination that performs better (MIN) and one that perform worse (MAX error)

NN architecture:

4 layers
Number of neurons (2,6,6,1)
First 3 layers ReLU activated (He initi), last layer Sigmoid (Glorot Init)
Epochs 5000 (20.000 samples)
Learning rate: 0.02

Encoding

One is encoded as 0.5
Zero is encoded as -0.5

The function is almost working, in the sense it outputs almost 0 instead of -0.5 and it outputs almost 0.5 in place of 0.5.. So if I decode the input with the following
Decoding

One if output is > 0.25
Zer if output is < 0.25

it has 100% accuracy.. however since it was trained with -0.5 as encoded zero actually it is not outputting what I expect.

The second example is from MNIST:

NN architecture:

Number of layers 3
Number of neurons (784,128,10)
All activations Sigmoid with Glorot initialization
Learning rate 0.005
Epochs 1 (the graph refers to 40% of the first epoch)

Encoding:

784 pixels encoded with grey values mapped from (0,255) to (0,1)
10 channels in output where the output is set to 1 for corresponding digit.

Even though the error rapidly decrease at the beginning, it gets stuck, and the accuracy stuck aroun 11% (slightly better than guessing the correct digit)

So what are the possible causes for error staying above a certain treshold?
Code: https://github.com/Darelbi/NeuraSharp/tree/main/NeuraSharp/NeuraSharp.BuiltIn

In the first example, Sigmoid activation cannot go below 0.  Either modify your encoding or switch to a different activation (tanh for example can go to -1).
On the second issue, it is customary to use softmax activation on the outputs here which ensures all outputs sum to 1 and it can vaguely be interpreted as the network's estimated probability distribution over class (1 being 100% surity, .5 being 50%, so on).  This is not strictly necessary so you might also be looking at underfitting.  But I'd start with softmax and go from there.

Ok, It worked, for the softmax It was not necessary since i was already taking the max output lane. Turns out i swapped Sigmoid for tanh (Remembering It was Sigmoid in range -1,1). For First problem I reached 100% accuracy with  expected values. For second problema It was definitely underfitting. Now i reach accuracy 81% with some noise, I consider It great improvement. Thank you


text:
Why soft actor critic uses exponential of Q when updating policy? and what is a partition function?
Soft Actor-Critic paper proposes $\pi_{new}=\arg\min_{\pi'\in \prod}D_{KL}\left(\pi'(\cdot|s_t)\big|\big| \frac{\exp(Q^{\pi_{old}}(s_t,\cdot))}{Z^{\pi_{old}}(s_t)}\right)$
Paper says, we update the policy towards the exponential of the new Q-function(is it really a new Q function when it is $Q^{\pi_{old}}$..?) because it guarantees to result in an imporved policy in terms of its soft value.

Does it mean, Q is exponential by just a modeling/analytical reason, as it allows us to proof the soft policy improvement theorem? There is no intuitive explanation about it?

It says $Z^{\pi_{old}}(s_t)$ is a partition function which normalizes the distribution. I understand that its usage is to normalize the distribution. However, what does it mean by partition function? Can someone provide a descriptive explanation/example about it to better understand what is partition function and how it is used to normalize the distribution of $exp(Q^{\pi_{old}}(s_t,\cdot))$?

looking at energy based models might answer some of your questions.

The idea comes from physics. The exp(Q)/Z term is from the Boltzmann equation. Large language models do the same thing, though they call this the "softmax" (which is likely where the name "soft actor-critic" comes from). I'd do some reading up on statistical mechanics, information theory, and evolutionary game theory if you want to understand this on a deeper level.

Regarding "we update the policy towards the exponential of the new Q-function", it's correct that the Q-function $Q(s_t,·)$ being exponentiated in the policy improvement step is new (latest) at timestep $t$, though under the old policy $\pi_{old}$. And it's the consideration of energy based ML models rooted in statistical mechanics to motivate taking exponential of the Q-function since intuitively the soft (stochastic) policy as a probability distribution can be regarded as the Boltzmann distribution and the learnable Q-function interpreted as energy can be flexibly arbitrary so long as you have a normalizing partition function.
A quintessential ML model using partition function is the restricted Boltzmann machine (RBM) which is a special case of Markov random field whose graphical model is undirected, inducing unnormalized joint density of the input in the form of a product of factors aka factor analysis in statistics.
In all references you could see the partition function involves summation or integral of many or infinite inputs, thus it's usually intractable. And here for SAC policy improvement with continuous action space defined earlier, the inputs are just all actions in the action space $\mathcal{A}$, therefore the partition function is $Z^{\pi_{old}}(s_t)=\int_\mathcal{A} \exp Q^{\pi_{old}}(s_t,a)da$.


text:
What is the optimal policy for this MDP?
What is the optimal policy for this MDP?

Since this is model-based RL you can use policy iteration method of dynamic programming, and since this MDP is very simple you can intuitively guess an optimal policy for your initial iterated policy $\pi_0$ as $\pi_0(a_1|s_1)=1, \pi_0(a_2|s_2)=1$, where $a_1$ is the switch action and $a_2$ is the stay action.
In the first policy evaluation step we need to solve for the Bellman equation of both states:
$v_{\pi_0}(s_1)=\sum_a\pi_0(a|s_1)\big[\sum_rp(r|s_1,a_1)r+\gamma \sum_{s'}p(s'|s_1,a_1)v_{\pi_0}(s')\big]=0+0.5(0.2v_{\pi_0}(s_1)+0.8v_{\pi_0}(s_2))$
$v_{\pi_0}(s_2)=\sum_a\pi_0(a|s_2)\big[\sum_rp(r|s_2,a_2)r+\gamma \sum_{s'}p(s'|s_2,a_2)v_{\pi_0}(s')\big]=1+0.5v_{\pi_0}(s_2)$
This is a simple system of linear equations of 2 variables $v_{\pi_0}(s_1),v_{\pi_0}(s_2)$, you can easily check the solutions are $v_{\pi_0}(s_1)=\frac{8}{9},v_{\pi_0}(s_2)=2$.
Then in the first policy improvement step we need to compute action value for each state-action pair:
$q_{\pi_0}(s_1,a_1)=\sum_rp(r|s_1,a_1)r+\gamma \sum_{s'}p(s'|s_1,a_1)v_{\pi_0}(s')=0+0.5(0.2v_{\pi_0}(s_1)+0.8v_{\pi_0}(s_2))=\frac{8}{9}$
$q_{\pi_0}(s_1,a_2)=\sum_rp(r|s_1,a_2)r+\gamma \sum_{s'}p(s'|s_1,a_2)v_{\pi_0}(s')=0+0.5v_{\pi_0}(s_1)=\frac{4}{9}$
$q_{\pi_0}(s_2,a_1)=\sum_rp(r|s_2,a_1)r+\gamma \sum_{s'}p(s'|s_2,a_1)v_{\pi_0}(s')=0+0.5(0.2v_{\pi_0}(s_2)+0.8v_{\pi_0}(s_1))=\frac{5}{9}$
$q_{\pi_0}(s_2,a_2)=\sum_rp(r|s_2,a_2)r+\gamma \sum_{s'}p(s'|s_2,a_2)v_{\pi_0}(s')=1+0.5v_{\pi_0}(s_2)=2$
Therefore the improved policy is updated as $\pi_1(a_1|s_1)=1,\pi_1(a_2|s_2)=1$, which apparently already converges to $\pi_0$ initialized in the above first section. So we can conclude an optimal policy for this MDP is $\pi^*(a_1|s_1)=1, \pi^*(a_2|s_2)=1$.


text:
Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts
I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101.
I want to comprehent the proof by a simple example:
Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy $\pi$ and $\pi'$
I calculated the action-values for each step with an $\epsilon = 0.1$. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.
Why is $\Sigma q_\pi(s,a)$ in Equation 2 only summing all Action Values without incomporating the Probabilities under policy $\pi$?
Do I need different Action Values for $\pi'$ and $\pi$?

Your highlighted equation 2 is incorporating the probabilities under policy $\pi'$ which as stated in your referenced page is the $\epsilon$-greedy policy with respect to $q_{\pi}$, the coefficients ($\frac{\epsilon}{|A(s)|},(1-\epsilon)$) in each summation term are the probabilities.
Here to demonstrate policy improvement theorem you only really need action values of $\pi$ as seen in your referenced derivation, the goal is to compute and compare state values for any generic state $s$ under both policies $\pi$ and $\pi'$.

Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$.

@bake_thi that sounds right.


text:
Can I scale subsets of my dataset independently to handle different feature ranges?
I am currently making an LSTM model to predict the change in value of a stock for each day. My dataset consists of 4 years of data for 30 different stocks along with some financial metrics such as RSI, EMA... This data is concatenated with a few macroeconomic indicators and some market interest indicators obtained from google trends.
For the LSTM model I have scaled the data between -1 and 1. However, because the data for each stock is so different (some stocks are valued at close to 1$ per share whereas others trade at $600 per share), I have scaled the data on a stock by stock basis using the min, max scaler. The issue I have is I don't know how to inverse the scale to get usable results for comparison?
Should I have scaled the entire dataset as one even though this will lead to some stocks having almost not difference from one day to the next?

Scaling individual parts of a dataset individually is generally not that trivial. Just remember that this means that the statistical properties of the subsets also may change. E.g., let's assume that your model just takes the mean of all sub-datasets. To combine the information from each sub-dataset you could simply averaging over all means. However, if the sub-datasets were differently scaled, averaging over them is not reasonable. The same holds for ML models.
However, if you are careful and properly scale the results of each sub-dataset then this might make perfectly sense.
To answer your last question: yes, scaling the entire dataset e.g. to (-1, 1) or any other reasonable range would probably be the easiest approach here. If this doesn't work due to too small difference then you still could think about non-linear scaling.


text:
How does the Belman optimality equation with altered transition probabilities in the second equation follow?
Sutton-Barto, page 102 (second edition).
How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the summation in second equation where the action variable is $a$ (the one over which max is taken) but the variable in the second part in the summation is $a^\prime$ which is independent of the variable over which max is taken. This second part is understandable because the actions
are random. But why is still the variable in the first part of the summation in the second equation $a$  which is the variable over which max is taken?

Your second equation is the BOE (Bellman optimality equation) of the altered transition probabilities for the new environment where any $\epsilon$-soft policy is moved inside as explained in the same page:

The new environment has the same action and state set as the original and behaves as follows. If in state $s$ and taking action $a$, then with probability $1-\epsilon$ the new environment behaves exactly like the old environment. With probability $\epsilon$ it repicks the action at random, with equal probabilities, and then behaves like the old environment with the new, random action... Let $\overset{\sim}{v}_*$ and $\overset{\sim}{q}_*$ denote the optimal value functions for the new environment. Then a policy $\pi$ is optimal among $\epsilon$-soft policies if and only if $v_\pi=\overset{\sim}{v}_*$

Therefore the max operator needs to range over the action $a$ in the first part of your second equation for learning the general (non-$\epsilon$-soft) optimal policy of the new environment. The uniformly repicked random actions $a'$ are summed out in the second part of your second equation, thus they're not really variables but placeholders.

but why is it "a" in the first part  which is the same variable as the variable over which max is taken?

As stated in my answer here the first part action $a$ is the real single non-dummy variable to be applied by max operator of BOE, it's the action of a general non-ϵ
-soft/greedy optimal policy of the new environment, as in any generic model-based value iteration dynamic programming method. The possible confusing part is the transition probability $p$ here belongs to the old environment, you have to use the altered form to express the new environment's transition probability based on the said given $p$.


text:
Optimality of two policies versus variance of returns from a state
If for an MDP there exist two optimal policies, it may be possible that the variance of their returns are different for a given state.
This is correct, right?

Yes, optimality is defined in the expectation of a random variable, variance is not taken in consideration
Consider 2 bandits, one that has $R^1 \sim N(0, 1)$ and the second one that has $R^2 \sim N(0,100)$ ($R$ being the reward function), then any policy $\pi$ will maximize the expected reward but for sure the greedy policy that always picks the first bandit has a lower variance than the greedy one that takes every time the second bandit


text:
Using AI to search for references in a dataset
I'm a long-time programmer just starting to look at AI so I'm not even sure what I need to ask or search on.  All examples I'm finding out there on Onnx seem to be dealing with images.
What I'd like to do is read in a dataset, in simplistic concept, a book.  In more complex terms, multiple tables in a DB all containing relative data but in different formats.
Take for example, a dataset that contains information on cooking:

A table that contains ingredients
A table that contains cooking utensils
a table that contains recipes
a table that contains cooking processes

Then I'd like to issue a request like:
"What do I need to cook chicken parmesan?"
Or
"What do I need to make a steak?"
I assume this is definitely possible.
I've done some work with flattening queries from users and have that working.  It's helped a lot, but I'm finding writing all ways that a user could ask a question is exhausting.
Can someone point me to an example that does something similar to this?  Or explain what I need to focus my attention on in learning this process?  I don't have a lot of time to spend learning the details of AI, but I would like to start working with it in a small way.

Well I gotta double check and see if everything is up to date with my assumptions. (I usually answer questions on reddit so I'm still getting used to this).
Assumption 1: There is a database of json data.
Ie.
{
name: "john",
fav_num: 2,
}
Assumption 2: You want to ask a question, and then get an AI that is able to go through the database and find "relevant" answer for you. So even if you say, "Give me a recipe for a juicy meal" it'll offer something like a steak versus a fruit juice meal.
If these two assumptions are true then you're going to be looking at semantic-text-embedding!
https://js.langchain.com/docs/get_started/quickstart here is a pretty good source that talks and shows code examples. It's probably the most complete source on how to do this stuff.
I'll walk you through the high-level logic behind how it works.

It takes in text
It transforms it into a vector based on meaning
2a. So basically if you put in the word, "big" it'll have a very similar meaning as "huge" so it'll recognize it as the same thing. This is important because it gives you the same meaning from different words. So we can search based on meaning.
We then search based on the vector using something like cosine similarity or euclidean distance.
We return the text from the database which has the closest similarity to the semantic text-embedding of the prompt.

Ie. What semantic-text-embedding does is storing all these different words into the same meaning.
https://openai.com/blog/introducing-text-and-code-embeddings

+1 for the samples.  This is exactly my case, both assumptions are true.  I am doing #1, and #2.  #3 is where I am getting hung up.  I will dig more into the link you provided. Thanks!

Good information there.  One question, I'm looking at on device AI and populating it with my data.  It looks like this piece is left out on the link?


text:
Book(s) recomendation for Probability Theory Foundations
I am in the process of learning ML and AI. I've been taking some courses, and I now understand the foundations of the big picture of Machine Learning. I've been using Pattern Recognition and Machine Learning as my go-to-book.
After a few reads of the chapters I've covered so far, I think I understand the general concepts in each of the methodologies, BUT I do recognize that my probability foundations are weak, therefore I want to improve that: Are there any recommendations on (good) books about probability theory that I could benefit from?
*(good) : by good I mean a book that gives you not only the theory and practice exercises, but especially a book that provides you with the Intuition and practical meaning of the topic being taught; I believe that is why I love Bishop's book because he tries to explain what the equations mean, what the concept implies, and what in practice is used. I looking for the same but on probability theory.
More specifically, I found that I could not completely grasp the Bayesian framework. I would want to go over it in a probability book. I believe it doesn't hurt from the foundations all the way to the full Bayesian framework theory. by the way, if there are any compilations of topics on which I should focus, it would be highly appreciated as well.
Thanks in advance,
Cheers

well bishop just published a new book, though it's not strictly on prob theory, and depends what you want to learn about probability theory

Thanks, I did notice his new Deep Learning book. It’s definitely in queue for reading… thanks!

This right here is the money shot.
https://ocw.mit.edu/courses/18-05-introduction-to-probability-and-statistics-spring-2022/pages/classes-reading-and-in-class-materials/
That's essentially what you're looking for. They get into Bayesian Inferences at Unit 2. This is better than a textbook, it has courses, lectures, problem sets, and if you're ever confused just ask a LLM for an  answer; and yes it has a textbook inside of it too.

This is just an amazing material! As you mentioned, just what I’m looking for !
Thanks for the recommendation, this definitely answers my question. Thanks Joseph

of course! Always happy to help :D


text:
Is there a relationship between tokens and parameters in LLMs?
What the question says.
In a transformer architecture, is there a relationship between number of tokens and number of parameters?
Can you have a LLM with a small number of parameters but a large context window (many tokens) or viceversa?

Transformers parameters count is invariant to the context window (which by definition can be infinite, though the $O(n^2)$ complexity might hurt)
Consider that the context window that you see in the LLMs documentations usually refers to the one used in training, but nothing stops you from taking a model train with a 10k context window and using it with a 50k context window (but usually performances deplete quickly extending the context window)

For the details, I'd suggest you to read the original paper, but the main point is the following:
Transformers are made by 2 main building blocks, attention and FFNN
The FFNN applies a transformation element-wise, thus it's for sure able to apply it to any sequence length size
The Attention block is nothing more than a (dynamic) linear combination of all the tokens... if you take the attention formula, and multiply the matrices, you will find that the output shape is the same as the input, thus making it length-free

Isn't there a maximum number of input tokens specified prior to training in the normal transformer architecture?

@Ggjj11 no, it’s done mainly to make the training faster by using tensor operations, but no, you can feed any sequence to the same architecture

Could you please help via describing in your answer why the model architecture theoretically can compute it's output even with arbitrary length inputs (let's say for num_tokens 100,1000,10000,...?) the attention output must work with arbitrary length of this dimension (which is true, because the self-attention matrix just dynamically could scale to nxn entries). But then ... I think you are right, but it would be nice to see a written explanation with all the details taking into account the classification head for next token prediction :)


text:
Xavier vs He initialization with tanh
I'm a student and in the lecture, I learned that He initialization is better than Xavier if you use ReLU activation function.
In addition, I also learned that Xavier initialization is better than He initialization.
However, I once implemented a simple MLP with 3 hidden layers using tanh, for function approximation as below:
2 input neurons
8 hidden neurons
1 output neuron
In the result, He got better result than Xavier, which is not as I expected.
Their final loss was similar, but He converged faster.
Can anyone help me why this happens? uniform Xavier and norm Xavier did not showed big difference.

the ablation studies on this topic have strong assumptions on the input distribution

Xavier (or Glorot) initialization is designed to prevent both sigmoid and tanh activations to saturate, i.e., to yield values close to 0 or 1 (for sigmoid) or -1, +1 (for tanh) which causes the gradient during backprop to be zero or so, therefore halting learning. The same happens for the ReLU, when its output is zero. So, He init, again, tries to prevent that.
Now, for simple problems and shallow networks you can't really expect big differences, which should be larger for more complex problems (e.g., learn from images or high-dimensional data) and deeper networks, for which He init is also optimized for (think about ResNets that have up to hundreds of layers.)
In your case, it may have happened that the He init prevented the tanh to saturate (even better than Xavier). But to check that you should plot a per-layer (and per-neuron) histogram of the activations to see that values should not be at the extremes but rather around zero.


text:
sample one node from each graph in a batch with pytorch
I am working with pygeometirc Batch objects. But, for those unfamiliar with it, I will pose my question as follows:
I am given a 1d tensor Y and a 1d tensor batch.
Y has the shape (num_of_nodes, ) and is the nodes' probabilities concatenated over the different graphs.
batch has the shape (num_of_nodes, ) and specifies the graph from which each node comes. Example:
Y = [0.2, 0.8, 1.]
batch = [0, 0, 1]
The first two nodes come from the first graph and the last from the second graph in the batch. Note that batch is always sorted.
I want to sample one node from each graph in the batch in a vectorized way. I am aware of a sequential way by splitting the vector and creating Categorical distributions, but this is too inefficient.
Example output:
out = [0, 2] or out = [1, 2]
Restarting the count at each graph (out=[0, 0] or out=[1, 0]) is fine, but not desirable.

This question is better suited for StackOverflow has it is more about coding rather than AI per se, but I did something like this recently, so here is what I ended up with:
First, you generate a random scalar for each node, then you identify the maximum value for each graph. This will point to a random node, one per graph. Here is a rough sketch:

Randomly sample a vector $z \sim U(0, 1) \in \mathbb{R}^N$, with $N$ being the number of nodes
Perform scatter with max-reduction: graph_choose = scatter(z, batch, reduction='max')
Perform gather: graph_choose = graph_choose[batch]
Compare the random vector with the graph-maximums: choice_mask = z == graph_choose. choice_mask will have a single True entry per graph, which is your randomly selected node.
(optionally) Apply mask to collect the random nodes the nodes: X_select = X[choice_mask]

Thank you so much. Choosing the maximum over uniform distributions should lead to a Discrete Uniform distribution over the nodes. Nonetheless, I'm not looking for a DUniform. I have the sampling probabilities given to me for each node. Are you aware of a sampling method that uses the same logic you used by perhaps augmenting z by p (given probabilities).

I tried multiplying z*p element-wise and choosing the maximum. Nonetheless, this proved to be mathematically wrong.

Ah, I misread, sry. Yes, you can extend the method to probabilities. Instead of the uniform distribution, you can use a Gumbel distribution: 1) Convert your probabilities to logits 2) sample a vector from the Gumbel distribution centered at your logits G(<your logits>, 1), continue with step 2) of my answer. This should work as the Gumbel distribution can be used as a substitute for categorical sampling.

I'll leave this here for any future reader: https://arxiv.org/abs/1411.0030

Nice, I hope this question will not be closed, I will update my answer later and also reference this article :)


text:
Does a random forest classifier always get 100% accuracy on its own training data?
Due to the way that decision trees work, do random forest classifiers always get 100% accuracy on their own training data? My random forest classifier got 100% accuracy on its own training data, so I'm not sure if this is a fluke or if there is a logical reason for this. The model got 99.4% accuracy on its testing data, so it may have just had high accuracy in general.

No, random forest is not guaranteed to get 100% accuracy on the test data.  But it also doesn’t mean it’s overfit because relatively high scores on test data are common with random forests.
The trees of a random forest are trained to leaf purity.  BUT, they are only trained on a subset of the samples/features.  A pure decision tree rarely generalizes well so samples not in a given tree's subset of the training samples might be wrong.  The voting mechanism of random forest tends to even out all of the mistakes and provide a good score but a sample in a training set may be so misrepresented by enough decision trees to outweigh the correct votes resulting in an incorrect prediction.


text:
What does "position-wise" fully connected mean?
I understand the architecture of a position-wise feed-forward network as described in (https://nn.labml.ai/transformers/feed_forward.html). However I do not understand what "position wise" refers to. I assume "position" refers to where a data point is inside the input vector/matrix. What makes this architecture "position-wise"?

Position-Wise basically means that the architecture takes position of a given value in the vector into considiration, and changing position of a given value in the input will impact the output.


text:
Understanding different methods of covariance parametrization
In the paper https://arxiv.org/pdf/2112.02143 it is noted that there are different ways to "parametrize" covariance. (page 4)

What does "parametrizing" covariance mean exactly? In the model described in the paper, prediction uncertainty in the form of covariance is estimated using an MLP. The resulting covariance matrix is used in an uncertainty-reduction loss function to train the model. I do not really understand what different parametrizations of covariance there are. As far as I know there exists simply one covariance of model predictions.
I also looked at the paper https://arxiv.org/pdf/1910.14215 which they reference. In it there are mentions of epistemic and aleatoric uncertainty however I do not really see how one can distinguish between the 2 when building this model, or if that is even what is meant by covariance "parametrization".

Parametrizing something is literally what the term is saying: "how do you put parameters so that you can end up having it via such parameters"
For example, you can parametrize the covariance with a MLP, thus the MLP "spits out" the covariance matrix. This way, such matrix can change for every input.
However, nothing stops you from having a fixed covariance, where every entry of your matrix is a parameter (however in this way, it's fixed, and does not vary based on $x$, as they do in their paper)
To conclude, what they do, is to ask the neural network to output both $\mu$ and $\Sigma$ based on the current input (thus the NN parametrize the covariance)... note that later they say that they just use the diagonal


text:
CNN model to infer classes from unlabelled, unpartitioned data
I'm wondering if there exists a CNN model (class of models?) designed to receive a single set of unlabelled, unpartitioned data, and attempt to infer classes. For instance, you could theoretically hand such a model a dataset containing some images of red squares and some images of blue circles, and the model would learn to distinguish the two types of images (perhaps by some kind of dimensionality reduction technique or Gaussian mixture model on the feature vectors).
I feel like I've heard of similar techniques before, but my extensive Googling failed me. If there are similar but not identical types of models to what I've described, or if for some reason the problem I've described is generally very difficult and therefore no models exist, then I'd love to know also.

what you want is a combination of dimensionality reduction technique and clustering.
Indeed, you can't do naive clustering as the feature space is way too large, thus you need some sort of lower dimensional representation. To do so, you can even naively use an autoencoder, and use the latent representation as a lower dimensional feature space.
On such manifold, you can then apply usual clustering techniques, such as DBScan, Kmeans and so on

Thanks @Alberto! Do you know of any algorithms that currently exist like this? Designing something from scratch is a bit above my paygrade

@Tom i don't have any specific end to end tool for this problem, sorry