text:
Why does the loss stops reducing after a point in this Transformer Model?
Context
I was making a Transformer Model to convert English Sentences to German Sentences. But the loss stops reducing after some time.
Code
import string
import re
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Embedding, LSTM, RepeatVector, Dense, Dropout, BatchNormalization, TimeDistributed, AdditiveAttention, Input, Concatenate, Flatten
from tensorflow.keras.layers import Activation, LayerNormalization, GRU, GlobalAveragePooling1D, Attention
from tensorflow.keras.optimizers import Adam
from tensorflow.nn import tanh, softmax
import time
from tensorflow.keras.losses import SparseCategoricalCrossentropy, CategoricalCrossentropy
from numpy import array
from tensorflow.keras.utils import plot_model
from sklearn.utils import shuffle
import time
import tensorflow as tf
from numpy import array
import numpy as np
from tensorflow.keras.models import load_model
from tensorflow.keras.datasets.imdb import load_data

def load_data(filename):
    file = open(filename, 'r')
    text = file.read()
    file.close()
    return text

def to_lines(text):
    return text.split('\n')

def clean_data(pair):
    pair = 'start_seq_ ' + pair + ' end_seq_'

    re_print = re.compile('[^%s]' % re.escape(string.printable))
    table = str.maketrans('', '', string.punctuation)
    tokens = [token.translate(table) for token in pair.split()]
    tokens = [token.lower() for token in tokens]
    tokens = [re_print.sub('', token) for token in tokens]
    tokens = [token for token in tokens if token.isalpha()]
    return tokens

lines = to_lines(load_data('/content/drive/My Drive/spa.txt'))

english_pair = []
german_pair = []
language = []
for line in lines:
    if line != '':
        pairs = line.split('\t')
        english_pair.append(clean_data(pairs[0]))
        german_pair.append(clean_data(pairs[1]))

        language.append(clean_data(pairs[0]))
        language.append(clean_data(pairs[1]))

english_pair = array(english_pair)
german_pair = array(german_pair)
language = array(language)

def create_tokenizer(data):
    tokenizer = Tokenizer()
    tokenizer.fit_on_texts(data)
    return tokenizer

def max_len(lines):
    length = []
    for line in lines:
        length.append(len(line))
    return max(length)

tokenizer = create_tokenizer(language)

vocab_size = len(tokenizer.word_index) + 1

max_len = max_len(language)

def create_sequences(sequences, max_len):
    sequences = tokenizer.texts_to_sequences(sequences)
    sequences = pad_sequences(sequences, maxlen=max_len, padding='post')
    return sequences

X1 = create_sequences(english_pair, max_len)
X2 = create_sequences(german_pair, max_len)
Y = create_sequences(german_pair, max_len)

X1, X2, Y = shuffle(X1, X2, Y)

training_samples = int(X1.shape[0] * 1.0)

train_x1, train_x2, train_y = X1[:training_samples], X2[:training_samples], Y[:training_samples]
test_x1, test_x2, test_y = X1[training_samples:], X2[training_samples:], Y[training_samples:]

train_x2 = train_x2[:, :-1]
test_x2 = test_x2[:, :-1]
train_y = train_y[:, 1:].reshape(-1, max_len-1)
test_y = test_y[:, 1:].reshape(-1, max_len-1)

train_x2 = pad_sequences(train_x2, maxlen=max_len, padding='post')
test_x2 = pad_sequences(test_x2, maxlen=max_len, padding='post')

train_y = pad_sequences(train_y, maxlen=max_len, padding='post')
test_y = pad_sequences(test_y, maxlen=max_len, padding='post')

All code above just prepares the Data, so if you want you can skip that part.
Code After this starts implementing the Transformer Model.
class EncoderBlock(tf.keras.layers.Layer):
    def __init__(self, mid_ffn_dim, embed_dim, num_heads, max_len, batch_size):
        super(EncoderBlock, self).__init__()
        # Variables
        self.batch_size = batch_size
        self.max_len = max_len
        self.mid_ffn_dim = mid_ffn_dim
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.attention_vector_len = self.embed_dim // self.num_heads
        if self.embed_dim % self.num_heads != 0:
            raise ValueError('I am Batman!')

        # Trainable Layers
        self.mid_ffn = Dense(self.mid_ffn_dim, activation='relu')
        self.final_ffn = Dense(self.embed_dim)

        self.layer_norm1 = LayerNormalization(epsilon=1e-6)
        self.layer_norm2 = LayerNormalization(epsilon=1e-6)

        self.combine_heads = Dense(self.embed_dim)

        self.query_dense = Dense(self.embed_dim)
        self.key_dense = Dense(self.embed_dim)
        self.value_dense = Dense(self.embed_dim)

    def separate_heads(self, x):
        x = tf.reshape(x, (-1, self.max_len, self.num_heads, self.attention_vector_len))
        return tf.transpose(x, perm=[0, 2, 1, 3])

    def compute_self_attention(self, query, key, value):
        score = tf.matmul(query, key, transpose_b=True)
        dim_key = tf.cast(tf.shape(key)[-1], tf.float32)
        scaled_score = score / tf.math.sqrt(dim_key)
        weights = tf.nn.softmax(scaled_score, axis=-1)
        output = tf.matmul(weights, value)
        return output

    def self_attention_layer(self, x):
        query = self.query_dense(x)
        key = self.key_dense(x)
        value = self.value_dense(x)

        query_heads = self.separate_heads(query)
        key_heads = self.separate_heads(key)
        value_heads = self.separate_heads(value)

        attention = self.compute_self_attention(query_heads, key_heads, value_heads)

        attention = tf.transpose(attention, perm=[0, 2, 1, 3])
        attention = tf.reshape(attention, (-1, self.max_len, self.embed_dim))

        output = self.combine_heads(attention)
        return output

    def get_output(self, x):
        attn_output = self.self_attention_layer(x)
        out1 = self.layer_norm1(x + attn_output)

        ffn_output = self.final_ffn(self.mid_ffn(out1))

        encoder_output = self.layer_norm2(out1 + ffn_output)
        return encoder_output

class DecoderBlock(tf.keras.layers.Layer):
    def __init__(self, mid_ffn_dim, embed_dim, num_heads, max_len, batch_size):
        super(DecoderBlock, self).__init__()
        # Variables
        self.batch_size = batch_size
        self.max_len = max_len
        self.mid_ffn_dim = mid_ffn_dim
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.attention_vector_len = self.embed_dim // self.num_heads
        if self.embed_dim % self.num_heads != 0:
            raise ValueError('I am Batman!')

        # Trainable Layers

        self.query_dense1 = Dense(self.embed_dim, name='query_dense1')
        self.key_dense1 = Dense(self.embed_dim, name='key_dense1')
        self.value_dense1 = Dense(self.embed_dim, name='value_dense1')

        self.mid_ffn = Dense(self.mid_ffn_dim, activation='relu', name='dec_mid_ffn')
        self.final_ffn = Dense(self.embed_dim, name='dec_final_ffn')

        self.layer_norm1 = LayerNormalization(epsilon=1e-6)
        self.layer_norm2 = LayerNormalization(epsilon=1e-6)
        self.layer_norm3 = LayerNormalization(epsilon=1e-6)

        self.combine_heads = Dense(self.embed_dim, name='dec_combine_heads')

        self.query_dense2 = Dense(self.embed_dim, name='query_dense2')
        self.key_dense2 = Dense(self.embed_dim, name='key_dense2')
        self.value_dense2 = Dense(self.embed_dim, name='value_dense2')

    def separate_heads(self, x):
        x = tf.reshape(x, (-1, self.max_len, self.num_heads, self.attention_vector_len))
        return tf.transpose(x, perm=[0, 2, 1, 3])

    def compute_self_attention(self, query, key, value):
        score = tf.matmul(query, key, transpose_b=True)
        dim_key = tf.cast(tf.shape(key)[-1], tf.float32)
        scaled_score = score / tf.math.sqrt(dim_key)
        weights = tf.nn.softmax(scaled_score, axis=-1)
        output = tf.matmul(weights, value)
        return output

    def masking(self, x):
        b = []
        for batch in range(x.shape[0]):
            bat = []
            for head in range(x.shape[1]):
                headd = []
                for word in range(x.shape[2]):
                    current_word = []
                    for represented_in in range(x.shape[3]):
                        if represented_in > word:
                          current_word.append(np.NINF)
                        else:
                          current_word.append(0)
                    headd.append(current_word)
                bat.append(headd)
            b.append(bat)
        return b

    def compute_masked_self_attention(self, query, key, value):
        score = tf.matmul(query, key, transpose_b=True)
        score = score + self.masking(score)
        score = tf.convert_to_tensor(score)

        dim_key = tf.cast(tf.shape(key)[-1], tf.float32)
        scaled_score = score / tf.math.sqrt(dim_key)
        weights = tf.nn.softmax(scaled_score, axis=-1)
        output = tf.matmul(weights, value)
        return output

    def masked_self_attention_layer(self, x):
        query = self.query_dense1(x)
        key = self.key_dense1(x)
        value = self.value_dense1(x)

        query_heads = self.separate_heads(query)
        key_heads = self.separate_heads(key)
        value_heads = self.separate_heads(value)

        attention = self.compute_masked_self_attention(query_heads, key_heads, value_heads)

        attention = tf.transpose(attention, perm=[0, 2, 1, 3])
        attention = tf.reshape(attention, (-1, self.max_len, self.embed_dim))

        output = self.combine_heads(attention)
        return output

    def second_attention_layer(self, x, encoder_output):
        query = self.query_dense2(x)
        key = self.key_dense2(encoder_output)
        value = self.value_dense2(encoder_output)

        query_heads = self.separate_heads(query)
        key_heads = self.separate_heads(key)
        value_heads = self.separate_heads(value)

        attention = self.compute_self_attention(query_heads, key_heads, value_heads)

        attention = tf.transpose(attention, perm=[0, 2, 1, 3])
        attention = tf.reshape(attention, (-1, self.max_len, self.embed_dim))

        output = self.combine_heads(attention)
        return output

    def get_output(self, x, encoder_output):
        masked_attn_output = self.masked_self_attention_layer(x)
        out1 = self.layer_norm1(x + masked_attn_output)

        mutli_head_attn_output = self.second_attention_layer(out1, encoder_output)
        out2 = self.layer_norm2(out1 + mutli_head_attn_output)

        ffn_output = self.final_ffn(self.mid_ffn(out2))
        decoder_output = self.layer_norm3(out2 + ffn_output)
        return decoder_output

embed_dim = 512
mid_ffn_dim = 1024

num_heads = 8
max_len = max_len
batch_size = 32

encoder_block1 = EncoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)
encoder_block2 = EncoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)
encoder_block3 = EncoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)

decoder_block1 = DecoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)
decoder_block2 = DecoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)
decoder_block3 = DecoderBlock(mid_ffn_dim, embed_dim, num_heads, max_len, batch_size)

# Define Loss and Optimizer
loss_object = SparseCategoricalCrossentropy()
optimizer = Adam()

embedding = Embedding(vocab_size, embed_dim, name='embedding')
position_embedding = Embedding(vocab_size, embed_dim)

final_transformer_layer = Dense(vocab_size, activation='softmax')

def positional_embedding(x):
    positions = tf.range(start=0, limit=max_len, delta=1)
    positions = position_embedding(positions)
    return x + positions

def train_step(english_sent, german_sent, german_trgt):
    with tf.GradientTape() as tape:
        english_embedded = embedding(english_sent)
        german_embedded = embedding(german_sent)

        english_positioned = positional_embedding(english_embedded)
        german_positioned = positional_embedding(german_embedded)

        # Encoders
        encoder_output = encoder_block1.get_output(english_positioned)
        encoder_output = encoder_block2.get_output(encoder_output)
        encoder_output = encoder_block3.get_output(encoder_output)

        # Decoders
        decoder_output = decoder_block1.get_output(german_positioned, encoder_output)
        decoder_output = decoder_block2.get_output(decoder_output, encoder_output)
        decoder_output = decoder_block3.get_output(decoder_output, encoder_output)

        # Final Output
        transformer_output = final_transformer_layer(decoder_output)

        # Compute Loss
        loss = loss_object(german_trgt, transformer_output)

    variables = embedding.trainable_variables + position_embedding.trainable_variables + encoder_block1.trainable_variables + encoder_block2.trainable_variables
    variables += encoder_block3.trainable_variables + decoder_block1.trainable_variables + decoder_block2.trainable_variables + decoder_block3.trainable_variables
    variables += final_transformer_layer.trainable_variables

    gradients = tape.gradient(loss, variables)
    optimizer.apply_gradients(zip(gradients, variables))

    return float(loss)

def train(epochs=10):
    batch_per_epoch = int(train_x1.shape[0] / batch_size)
    for epoch in range(epochs):
        for i in range(batch_per_epoch):
            english_sent_x = train_x1[i*batch_size : (i*batch_size)+batch_size].reshape(batch_size, max_len)
            german_sent_x = train_x2[i*batch_size : (i*batch_size)+batch_size].reshape(batch_size, max_len)
            german_sent_y = train_y[i*batch_size : (i*batch_size)+batch_size].reshape(batch_size, max_len, 1)

            loss = train_step(english_sent_x, german_sent_x, german_sent_y)

            print('Epoch ', epoch, 'Batch ', i, '/', batch_per_epoch, 'Loss ', loss)

train()

And the Code is done! But the loss stops reducing at around value of 1.2 after some time. Why is this happening?
Maybe Important
I tried debugging the model, by passing random input integers, and the model was still performing the same way it did when I gave real Sentences as input.
When I tried training the model with just 1 training sample, the loss stops reducing at around 0.2. When I train it with 2 training samples, the result was the approximately the same as when I trained it with 1 training sample.
When I stopped shuffling the dataset the loss gone till around 0.7 and again stopped learning.
I tried simplifying the model by removing some encoder and decoder blocks but the results were approximately the same. I even tried making the model more complex but the results were again approximately the same.

Is it supposed to translate English to German? Does it use dictionaries for both languages? How is success measured?

@markmark yes it is supposed to translate English to German. No it does not uses dictionary, it uses array to store the data. I am measuring success in terms of loss. And also when saw the predicted sentence it is not that good

I'm far from an expert, but here some ideas that I have. The problem is very complex for several reasons - the model needs to learn the relationship between the dictionaries (which are huge, and there's not always one to one match between words) and also it needs to learn the concept of grammar and semantics. That might need huge processing power. Maybe limiting number of allowed words in the training sets might help a bit? Not sure what architecture is proper for this task, but training many hidden layers is difficult to train without a strong GPU too.

Another point is that it's hard to evaluate the success of translation numerically (there probably is, but I'm not sure SparseCategoricalCrossentropy is good enough for this), which makes it difficult to come up with an applicable loss function. For example, you might need some sort of custom loss function which will describe the distance in translations. Such function would assign different success values to different translations (e.g. translating "This is very good" as "Das ist sehr gut" vs "Das gut" vs a bunch of random words)


text:
What is the process of inventing deep neural network models? How researchers deal with long training times?
After reading this topic on GitHub how long time it takes to train YOLOV3 on coco dataset I was wondering how researchers deal with long training times while inventing new architectures.
I imagine that to evaluate the model you need to train it first. How do they make tweaks in their architectures, e.g. tweaking layers, adding pooling, dropout, etc., if training can take a few days? Is it pure art and it is designed roughly or is it a more deliberate process?
What are the steps of engineering new architecture using deep neural networks?

Part of the answer is that new architectures are often variants of existing architectures. There are some rough heuristics people follow (e.g. using powers of 2 in layer sizes, changing layer sizes according to a schedule, adding normalization layers, etc.). However, even when building off an existing architecture, there can still be a lot of toying around with different configurations and hyperparameters to require copious training time.
Researchers working for a company will likely have the luxury of some serious compute, e.g. GPU clusters or cloud services like AWS. This can really make a huge difference over training a model on your PC GPU. Further, many models can be trained at once. Many academic institutions also employ cluster computing on dedicated GPU clusters - for example, here at the University of Utah our department has a cluster with 8 really excellent GPUs.
One last thing I will note, is that it's not always necessary to train the model to completion on the full training set to get a sense of which network configuration and hyperparameter settings are working better than others. If I have a massive dataset that will take days to train on, I may take a smaller subset of the training data and do some quick comparisons between different models to get a feel for what changes have the greatest impact on test performance. It's not completely reliable, but it can guide intuition to narrow down the number of models you want to train on the full dataset to convergence. For example, using this technique I was able to find that predictions from my model were fairly invariant to the exact layer sizes I was using and was far more sensitive to a hyperparameter in my optimization solver.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can Machine Learning be used to synthesize engine sounds?
I'm working on a project to equip model locomotives with sound boards. I'm in the process of designing the board at the moment, and the idea is to allow users to load their own sound files onto an SD card plugged into the board.
Conventionally, model locomotive sounds are collected from high-fidelity microphones placed on and around the real engine in question. The engine is started up then put through idle and all of the different notches, as well as dynamic braking, horn and bell sounds, etc. This practice is very expensive because you have to find a willing (usually small) railroad or museum, pay for travel expenses, and diesel fuel ain't exactly cheap at the volumes these engines go through. Secondly, newer engines are hard to record because railroads aren't exactly in the business of letting hobbyists tape microphones all over their money making machines. As such, the main cost for a sound board comes not from the circuit's BOM cost, but from the effort required to get sounds from locomotives.
What there's plenty of are YouTube videos of amateur rail enthusiasts taking videos of locomotive sightings at close(ish) proximity, including startups and shutdowns. My question is - is there a way to take a bunch of different audio recordings of the same engine, remove noise and the doppler effect, and from that create a profile that can be used to simulate what the engine might sound like at different throttle notches? Is machine learning the right tool for this?

Deep learning has been used to create a super-resolution photo from many low-resolution photos (https://arxiv.org/pdf/1808.03344.pdf), and also adding things to the images (e.g. make a frowning person smile). I don't know of an application that creates a super-resolution audio from many low and noisy resolution audios, and then predict the audio at different notches/braking/horn/etc, but I don't see why it can't -- it just hasn't been done before (that I know of). Do you plan on developing a model of this yourself?

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Using an LSTM for model-based RL in a POMDP
I am trying to set up an experiment where an agent is exploring an n x n gridworld environment, of which the agent can see some fraction at any given time step. I'd like the agent to build up some internal model of this gridworld.
Now the environment is time-varying, so I figured it would useful to try using an LSTM so the agent can learn potentially useful information about how the environment changes. However, since the agent can only see some of the environment, each observation that could be used to train this model would be incomplete (i.e. the problem is partially-observable from this perspective). Thus I imagine that training such a network would be difficult since there would be large gaps in the data - for example, it may make an observation at position [0, 0] at t = 0, and then not make another observation there until say t = 100.
My question is twofold

Is there a canonical way of working around partial observability in LSTMs? Either direct advice or pointing to useful papers would both be appreciated.
Can an LSTM account for gaps in time between observations?

Thanks!

Note that, in many problems, the state is only partially observable. Even in the Atari games, for example, an observation (i.e. a frame/image from the game) may not correspond to a state. In the DQN paper, they stacked multiple successive observations to build the state, given that multiple successive observations may contain more info about the current situation/state of the agent.

There is this question, which is very similar to yours. After reading that question and answer, can you please tell us what is your specific question at that point, or if your question is a duplicate of that one?


text:
Algorithms for training a two motor powered rc car without steering servo
Previously, I have build a donkey car where the steering of the two front wheels was done using a motor servo. This project was a success and the car was able to drive autonomously after training was done.
source Donkey Car 2
Now:
I have this  Rc Car kit that has two motors on the back, powering two wheels and a trolley wheel in front.
The steering is supposed to be done by playing around with the two back motors.
My question is:
Is there any method modify the donkey car code, so I can train the model?
Considering that Donkey Car uses the angle of the servo to train the model, and now I just have the information of the two back wheels and no servo steering the vehicle
Not sure if there approach that is specific to this concept.


text:
How to handle long sequences with transformers?
I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during the train time. I want to use a transformer model. I have 2 questions:

If I want to embed the 400 dimensional input feature vector into another space before feeding into the transformer, what are the pros and cons of using let's say 1024 and 64 for the embedding space dimension? Should I use a dimension more than 400 or less?
When doing position embedding, I cannot use a maximum position length of 10 million as that blows up the memory. What is the best strategy here if I want to use maximum position length of 512? Should I chunk the 10 million steps into blocks of size 512 and feed each block separately into the transformer? If so, how can I connect the subsequent blocks to take full advantage of parallelization while keeping the original chronological structure of the sequence data?


text:
Which NN would you choose to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$?
Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you choose ? How many layers/neurons ? Which activation functions ? Which loss function ?
Intuitively, I would go with one hidden layer, 2 neurons, $L^2$ loss, and maybe the Bent identity for the output and a sigmoid in the hidden layer ?
What are the advantages of doing something "fancier" than that ?
Would you also have chosen to use a NN for this job or would you have considered a regression SVM for example or something else (knowing that precision is the goal)?

It depends on the complexity of your problem. $\mathbb{R}^2 \rightarrow \mathbb{R}^1$ looks simple, but I can give you some nonsense complicated examples that need a deep network. So, the complexity of the problem sets the number of layers and neurons. The kind of problem will determine the architecture of your network (if it needs memory or not). In most cases, the mean squared error is ok. However, for the activation function, I would go with the ReLU.
If the SVM is good enough for your problem, then go with it. Your question is general and an exact answer needs more information about the problem.

Previously, you had written that the ReLU is better. I removed the "better", but maybe you should explain why you're suggesting the ReLU. Maybe you should also mention the universal approximation theorems, given the generality of the question, if you heard of them.


text:
Can I start with perfect discriminator in GAN?
In many implementations/tutorials of GANs that I've seen so far (e.g. this), the generator and discriminator start with no prior knowledge. They continuously improve their performance with training. This makes me wonder — is it possible to use a pre-trained discriminator? I have two motivations for doing so:

Eliminating the overhead of training the discriminator
Being able to use already existing cool models

Would the generator be able to learn just the same, or is it dependent on the fact that they start from scratch?

Some pretty interesting comments on the subject here: https://www.reddit.com/r/MachineLearning/comments/dwzb81/r_training_gans_with_a_pretrained_discriminator/

If you start with perpect discriminator, loss function will be saturated, and gradient of loss will be very small, so feedback for the generator also will be small, and learning will be slow down as a result. Actually, it is allways desired for discriminator and generator to learn balancedly. Additionally, it is claimed that Wasserstein Loss take care of this problem.
You can find more information in this article. (I strongly suggest to read)
Also from the paper " Towards principled methods for training generative adversarial networks":

In theory, one would expect therefore that we would first train the discriminator as close as we can to optimality (so the
cost function on $\theta$ better approximates the JSD), and then do gradient steps on  $\theta$, alternating these
two things. However, this doesn’t work. In practice, as the discriminator gets better, the updates
to the generator get consistently worse. The original GAN paper argued that this issue arose from
saturation, and switched to another similar cost function that doesn’t have this problem. However,
even with this new cost function, updates tend to get worse and optimization gets massively unstable.

Note 1: $\theta$ is parameters of generator
Note 2: JSD: Jensen–Shannon divergence. For optimal discrimantor, loss is equal to $L(D^{ \ast }, g_{\theta})=  2 JSD(P_{r}, P_{g}) - 2log(2)$
Also from the paper of Wasserstein Loss:

The fact that the EM distance is continuous and differentiable a.e. means that
we can (and should) train the critic till optimality. The argument is simple, the more we train the critic, the more reliable gradient of the Wasserstein we get, which is actually useful by the fact that Wasserstein is differentiable almost everywhere. For the JS, as the discriminator gets better the gradients get more reliable but the true gradient is 0 since the JS is locally saturated and we get vanishing gradients ... In Figure 2 we show a proof of concept of this, where we train a GAN discriminator and a WGAN critic till optimality. The discriminator learns very quickly to distinguish between fake and real, and as expected provides no reliable gradient information. The critic, however, can’t saturate, and converges to a linear function that gives remarkably clean gradients everywhere.

I find this a very good answer. However, I wonder if relativistic GANs wouldn't cope with this and even benefit from the pretrained discriminator.

@David Thanks for the comment. I honestly heared Relativistic GAN first time from you. Seems interesting paper.


text:
Why is KL divergence used so often in Machine Learning?
The KL Divergence is quite easy to compute in closed form for simple distributions -such as Gaussians- but has some not-very-nice properties. For example, it is not symmetrical (thus it is not a metric) and it does not respect the triangular inequality.
What is the reason it is used so often in ML? Aren't there other statistical distances that can be used instead?

This is an interesting question. Is it because the KL divergence can be approximately computed more cheaply than the other statistical distances? The KL divergence can be approximated with Monte Carlo rollouts, even if you don't have a closed-form solution (because it's defined as an integral). I am not familiar with the details of the other statistical distances, such as the Wasserstein metric, but I guess they may not be so cheap even to approximately compute (at least, that's one of the reasons that I can think of now why the KL divergence is more used than other measures).

The KL divergence has also an information-theoretic interpretation, but I don't think this is the main reason why it's used so often. However, that interpretation may make the KL divergence possibly more intuitive to understand. Moreover, the KL divergence formula is quite simple. I don't really know how the other measures are defined (apart from the TVD), but maybe that (the simplicity) is also one reason. As you point out, the KL divergence has also some apparent disadvantages, so your question is more than valid.

Anyway, I don't feel like providing an answer now, because I am not familiar with the details of other measures/metrics/distances, but maybe later I will investigate this issue, if I have some time.

In ML we always deal with unknown probability distributions from which the data comes. The most common way to calculate the distance between real and model distribution is $KL$ divergence.
Why Kullback–Leibler divergence?
Although there are other loss functions (e.g. MSE, MAE), $KL$ divergence is natural when we are dealing with probability distributions. It is a fundamental equation in information theory that quantifies, in bits, how close two probability distributions are. It is also called relative entropy and, as the name suggests, it is closely related to entropy, which in turn is a central concept in information theory. Let's recall the definition of entropy for a discrete case:
$$
H = -\sum_{i=1}^{N} p(x_i) \cdot \text{log }p(x_i)
$$
As you observed, entropy on its own is just a measure of a single probability distribution. If we slightly modify this formula by adding a second distribution, we get $KL$ divergence:
$$
D_{KL}(p||q) = \sum_{i=1}^{N} p(x_i)\cdot (\text{log }p(x_i) - \text{log }q(x_i))
$$
where $p$ is a data distribution and $q$ is model distribution.
As we can see, $KL$ divergence is the most natural way to compare 2 distributions. Moreover, it's pretty easy to calculate. This article provides more intuition on this:

Essentially, what we're looking at with the KL divergence is the
expectation of the log difference between the probability of data in
the original distribution with the approximating distribution. Again,
if we think in terms of $log_2$ we can interpret this as "how many
bits of information we expect to lose".

Cross entropy
Cross-entropy is commonly used in machine learning as a loss function where we have softmax (or sigmoid) output layer, since it represents a predictive distribution over classes. The one-hot output represents a model distribution $q$, while true labels represent a target distribution $p$. Our goal is to push $q$ to $p$ as close as possible. We could take a mean squared error over all values, or we could sum the absolute differences, but the one measure which is motivated by information theory is cross-entropy. It gives the average number of bits needed to encode samples distributed as $p$, using $q$ as the encoding distribution.
Cross-entropy based on entropy and generally calculates the difference between two probability distributions and closely related to $KL$ divergence. The difference is that it calculates the the total entropy between the distributions, while $KL$ divergence represents relative entropy. Corss-entropy can be defined as follows:
$$
H(p, q) = H(p) + D_{KL}(p \parallel q)
$$
The first term in this equation is the entropy of the true probability distribution $p$ that is omitted during optimization, since the entropy of $p$ is constant. Hence, minimizing cross-entropy is the same as optimizing $KL$ divergence.
Log likelihood
It can be also shown that maximizing the (log) likelihood is equivalent to minimizing the cross entropy.
Limitations
As you mentioned, $KL$ divergence is not symmetrical. But in most cases this is not critical, since we want to estimate the model distribution by pushing it towards real one, but not vice versa. There is also a symmetrized version called Jensen–Shannon divergence:
$$
D_{JS}(p||q)=\frac{1}{2}D_{KL}(p||m)+\frac{1}{2}D_{KL}(q||m)
$$
where $m=\frac{1}{2}(p+q)$.
The main disadvantage of $KL$ is that both the unknown distribution and the model distribution must have support. Otherwise the $D_{KL}(p||q)$ becomes $+\infty$ and $D_{JS}(p||q)$ becomes $log2$
Second, it should be noted that $KL$ is not a metric, since it violates triangle inequality. That is, in some cases it won't tell us if we are going the right direction when estimating our model distribution. Here is an example taken from this answer. Given two discrete distributions $p$ and $q$, we calculate $KL$ divergence and Wasserstein metric:

As you can see, $KL$ divergence remained the same, while the Wasserstein metric decreased.
But as mentioned in comments, Wasserstein metric is highly intractable in a continuous space. We still can use it by applying the Kantorovich-Rubinstein duality used in Wasserstein GAN. You can also find more on this topic in this article.
The 2 drawbacks of $KL$ can be mitigated by adding noise. More on it in this paper

You say "The most common way to calculate the distance between real and model distribution is KL divergence", but you're not explaining why. The OP was already aware of the KL divergence can be used for and its disadvantages, so not sure why you're listing them here again. Apart from saying that the Wasserstein metric can be intractable, you're not really answering the question. Note that the question was not "What are the disadvantages of the KL divergence?".

Thanks for your feedback. I spent more time and added more information.

I kind of agree with @nbro. This post is very informative and great intro to "What is KL?", but doesn't answer why KL is so popular in ML (additionally, OP already knows that KL is). This post says "KL is used to measure difference in 2 probability distributions, and is the most natural way to do so", but there are many other ways to measure the difference of 2 distributions and some are more natural/intuitive than KL (examples: wasserstein distance, cosine similarity, mean squared residuals, etc). This post doesn't explain to readers why KL is used over all the other metrics.

I tried to show the connection between entropy, KL and cross-entropy. Entropy is the central concept in information theory, and KL, in turn, is the basics of ML.

The measures you named is not natural for comparing 2 probability distributions (except Wasserstein, but it has its own drawbacks). MSE/MSR is natural for regression problems, cosine is natural for vectors

This question is very general in the sense that the reason may differ depending on the area of ML you are considering. Below are two different areas of ML where the KL-divergence is a natural consequence:

Classification: maximizing the log-likelihood (or minimizing the negative log-likelihood) is equivalent to minimizing KL divergence as typical used in DL-based classification where one-hot targets are commonly used as reference (see https://stats.stackexchange.com/a/357974). Furthermore, if you have a one-hot vector $e_y$ with $1$ at index $y$, minimizing cross-entropy $\min_{\hat{p}}H(e_y, \hat{p}) = - \sum_y e_y \log \hat{p}_y = - \log \hat{p}$ boils down to maximizing the log-likelihood. In summary, maximizing the log-likelihood is arguably a natural objective, and KL-divergence (with 0 log 0 defined as 0) comes up because of its equivalence to log-likelihood under typical settings, rather than explicitly being motivated as the objective.
Multi-armed bandits (a sub-area of reinforcement learning): Upper confidence bound (UCB) is an algorithm derived from standard concentration inequalities. If we consider MABs with Bernoulli rewards, we can apply Chernoff's bound and optimize over the free parameter to obtain an upper bound expressed in terms of KL divergence as stated below (see https://page.mi.fu-berlin.de/mulzer/notes/misc/chernoff.pdf for some different proofs).

Let $X_1, \dots, X_n$ be i.i.d. Bernoulli RVs with parameter $p$.
$$P(\sum_i X_i \geq (p+t)n) \leq \inf_\lambda M_X (\lambda) e^{-\lambda t} = \exp(-n D_{KL}(p+t||p)).$$

Comments are not for extended discussion; this conversation has been moved to chat.


text:
Why can't we combine both training and validation data, given that both types of data are used for developing the model?
Sorry if I sound confused. I read that data to be fed to a machine are divided into training, validation and test data. Both training and validation data are used for developing the model. Test data is used only for testing the model and no tuning of the model is done using test data.
Why is there a need to separate out training and validation data since both sets of data are for developing/tuning the model? Why not keep things simple and combine both data sets into a single one?

There are two possible forms of overfitting. First related to the training only (fitting weights) and second related to  architecture (fitting hyperparameters) and these two must be checked in two different stages. When you check performance of the given model you have two do this on unseen data, so you fit weights on training data and check it on validation (unseen to the weights fitting process) set and next you fit hyperparameters on validation set and check it on test (unseen to the hyperparameters fitting process) set.

Thanks. Upvoted. I see. Training data is for tuning weights. Validation data is for tuning hyperparameters. Did I understand correctly?

and test set is to test it all. Simplified, but yes.

Usually we are splinting the training dataset because we want to fine tune, find the best hyper-parameters, for our model. If we combine the validation and training dataset given a network complex enough we could achieve perfect performance for the given task. But having very good performance on the training dataset does not mean our model is useful. It might be the case that our model is very good on the training dataset, but it fails to generalize, so when given samples outside of the training dataset the performance will significantly drop (this is called over-fitting). By splitting the dataset, we are testing how well our model is performing on new/unseen data.


text:
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go human player. What kind of reinforcement model does it use? Why is this particular model appropriate for Go game?

I think their program is not just based on a single algorithm, and they used several algorithms in their design. However they have used deep reinforcement learning (There are several methods available, see for instance: "Human-level control through deep reinforcement learning"). The main difference between deep reinforcement learning and traditional Q- learning is that, tradition Q learning can only learn applicably when we have few states and actions say 10 or 20.


text:
DDQN Agent in Othello (Reversi) game struggle to learn
This is my first question on this forum and I would like to welcome everyone.
I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems to be properly initialized does not learn against random opponent. Actually the score is about 50-60% won games out of nearly 500. Generally if it gets some score after first 20-50 episodes it stays on the same level. I have doubts on the process of learning and how to decide when the agent is trained. Current flow is as follows:

Initialize game state.
With epsilon greedy policy choose the action to make based on currently available actions depending on game state
Get opponent to make his action
Get the reward as number of flipped places that remain after opponent move.
Save the observation to replay buffer
If number of elements in replay buffer equals or higher than batch size do the training.

What I do not know is when do I know when to stop the training. Previously this agent trained against MinMax algorithm learned how to win 100% games because MinMax played exactly the same every time. I would like the agent to generalize the game. Right now I save the network weights after the game is won but I think it does not matter. I can't see that this agent find some policy and improve over time. Whole code for the environment, agent and training loop can be found here: https://github.com/MikolajMichalski/RL_othello_mgr I would appreciate any help. I would like to understand how the RL works :)

Hi Mikołaj and welcome! The best strategy for the game of reversi is somewhat counterintuitive. It's actually preferable to have little pieces of your color on the board at the beginning or midgame, and only in the very end flip the situation in your favor. The way you currently define your reward function encourages the agent to be greedy in the beginning, which may lead to losing the control of the corners (which are important). You want the agent to learn to win by itself, and your reward function should only award points for winning/drawing and losing.

Hi, thank you for the answer. I changed reward to be only given after the episode is finished. Now it's counted as learning_agent_score - opponent_score. I can see a bit of improvement but is is not satisfying enough. I may experiment with hyper parameters. I have other question... Is it a proper way to train agent against random opponent? I thought about the policy of setting opponents weights to learning agent weights every time the win percentage gets higher so the trained agent plays against better opponent over time.

To best of my knowledge, self-play would preferred over playing versus random opponent in terms of learning efficiency, since the state space is huge. Training network against itself using some minmax variation should be able to pick up some strategies. You can then evaluate the performance of the network against random opponent. Though, it might take a good while before agent learns anything the game is very complex (less than, but comparable to chess).

As I did some experiments with hyperparameters and did not see much progress... I am coming back with more questions :)

What would be the best network architecture in this case? Mine is 1st layer - input 64 of size with ReLU, 2nd layer - hidden with ReLU- 32 of size, 3rd layer - 64 of size linear and 4th with softmax. I am not sure if this is proper approach but I can't find good source about building good network arch.
Currently I am getting minibatches with random samples for training. Is it good as the states sequentions for each games are  highly dependent?

And another one:
3. Should the training be done after each episode or each step?

Hello again! I worked on a similar problem before - I tried to solve mancala game with DDQN (which I could not solve with that algorithm and currently trying some others). For the network architecture I only found vague suggestion -- seems that this topic is not well understood within the community. I'm inclined to think that you may need more neurons in each layer (a couple of hundreds), because neurons need to look at relations between pieces too and that's a huge space.

The second question - yes, that's fine. DDQN should converge to optimal policy in theory, regardless of what you're feeding to it.
The third - the training should be done at the end of the episode, provided, there's enough samples.

I would also suggest to test your algorithm on a simpler game for a sanity check. Maybe not as simple as tic-tac-toe, but something that  has smaller state/action space and hence is easier for agent to find patterns in


text:
Interpretation of Inner Product in a two-tower model
I have seen at quite a few places the use of two-tower architecture. This(Fig 6) is one of the examples. Each tower computes embedding of a concept which is orthogonal to the concepts in the rest of the towers. Also, I have seen that the next step on getting an n-dimensional embedding from the towers an inner product is taken to measure Cosine similarity.
I would like to know how to interpret this Inner Product. The two n-dimensional embeddings represents different concepts. What is the meaning of similarity here ?


text:
How does the support vector machine constraint imply that sample selection bias will not systematically affect the output of the optimisation?
I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3.4. Support vector machines, the author says the following:

3.4. Support vector machines
In its basic form, the support vector machine (SVM) algorithm (Joachims, 2000a) learns the parameters $a$ and $b$ describing a linear decision rule
$$h(x) = \text{sign}(a \cdot x + b),$$
whose sign determines the label of an example, so that the smallest distance between each training example and the decision boundary, i.e. the margin, is maximized. Given a sample of examples $(x_i, y_i)$, where $y_i \in \{ -1, 1 \}$, it accomplishes margin maximization by solving the following optimization problem:
$$\text{minimize:} \ V(a, b) = \dfrac{1}{2} a \cdot a \\ \text{subject to:} \ \forall i : \ y_i[a \cdot x_i + b] \ge 1$$
The constraint requires that all examples in the training set are classified correctly. Thus, sample selection bias will not systematically affect the output of this optimization, assuming that the selection probability $P(s = 1 \mid x)$ is greater than zero for all $x$.

How does the constraint that all examples in the training set are classified correctly imply that sample selection bias will not systematically affect the output of the optimisation? Furthermore, why is it necessary to assume that the selection probability is greater than zero for all $x$? These are not clear to me.


text:
Designing Policy-Network for Deep-RL with Large, Variable Action Space
I am attempting a project involving training an agent to play a game using deep reinforcement learning.
This project has a few features that complicate the design of the neural network:

The action space per state is very large (can be over 1000 per state)
The set of actions available in each state very wildly between states, both in size and the actions available.
The total action-space (the union of each state's action-space) is way too large to enumerate.
The action space is discrete, not continuous.
The game is adversarial, multi-agent.

Most RL neural networks I've seen involve the input of a state, and an output of a constance action size, where each element of the output is either an action's q-score or probability. But since my game's action space non-constant per state, I believe this design will not work for this game.
I have seen an alpha-go style network, which outputs a probability for all actios ever possible, and then zeros out actions not possible in the given state and re-normalized the probabilities. However, since the total action-space in my game is way to large to enumerate, I don't believe this solution will work either.
I have seen several network designs for large, discrete action spaces, including:

design a network to input a state-action pair and output a single score value, and train it via a value-based algorithm (such as q-learning). To select an action given a state, pass in every possible state-action pair to get each action's score, then select the action with the highest score.
(Wolpertinger architecture) have a network output a continous embedding, which is then mapped to a discrete action, and train it via deterministic policy gradient.
divide actions into a sequence of simpler actions, and train an RNN to output a sequence of these simpler actions. Train this network via a value-based algorithm (such as q-learning).

However, all of these solutions are designed for either value-based or deterministic policy gradient algorithms; none of them output probabilities over the action space. This seems to be an issue since at least a very large portion of the multi-agent deep-RL algorithms I've seen involve a network that outputs a probability over the action-space. Therfore, I don't want to limit myself to value-based and deterministic-policy algorithms.
How can I design a neural network that outputs a probability over the action space for my game?
If not, what would be some good solutions to this problem?

"I have seen an alpha-go style network, which outputs a probability for all actios ever possible, and then zeros out actions not possible in the given state and re-normalized the probabilities." Can you tell me the name of this paper. I am curious on some details of this approach. Besides, I hope this answer can help you.


text:
Are inputs into AlphaZero the same during the evaluate step in MCTS and during test time?
From the AlphaZero paper:

The input to the neural network is an N × N × (M T + L) image stack that represents state using a concatenation of T sets of M planes of size N × N . Each set of planes represents the board position at a time-step t − T + 1, ..., t, and is set to zero for time-steps less than 1

From the original AlphaGo Zero paper:

Expand and evaluate (Figure 2b). The leaf node $s_L$ is added to a queue for neural network evaluation, $(d_i(p), v) = f_\Theta(d_i(s_L))$, where $d_i$ is a dihedral reﬂection or rotation selected uniformly at random from i∈[1..8]

Ignoring the dihedral reflection, the formula in the original paper $f_\Theta(s_L)$ implies that only the board corresponding to $s_L$ is passed to the neural network for evaluation when expanding a node in MCTS, not including the 7 boards from the 7 previous time steps. Is this correct?


text:
Optimum Discriminator for label smoothed GAN
I was reading the paper called Improved Techniques for Training GANs. And, in the one-sided label smoothing part, they said that optimum discriminator with label smoothing is
$$ D^*(x)=\frac{\alpha \cdot p_{data}(x) + \beta \cdot p_{model}(x)}{p_{data}(x) + p_{model}(x)}$$
I could not understand where this is come from. How did we get this result?
Note: By the way, I knew how to find optimal discriminator in vanilla GAN i.e.
$$ D^*(x) = \frac{p_{r}(x)}{p_{r}(x) + p_g(x)} $$

I did not yet write it down in detail but to my understanding deriving this result should work like deriving the optimal discriminator in the vanilla GAN (which does not use label smoothing). In case of label smoothing your integrands just get constants alpha and beta...

The equation most likely comes from one of the following references:

David Warde-Farley and Ian Goodfellow. Adversarial perturbations of deep neural networks. In Tamir Hazan, George Papandreou, and Daniel Tarlow, editors, Perturbations, Optimization, and Statistics, chapter 11. 2016. Book in preparation for MIT Press.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception Architecture for Computer Vision. ArXiv e-prints, December 2015.

I was not able to retrieve the book but Szegedy et al. discuss on page 6 (see "Model Regularization via Label Smoothing").
Ian Goodfellow discusses this topic on page 31 of his tutorial:

I. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. ArXiv e-prints April 2017.

A similar question was asked and answered in the Data Science community: https://datascience.stackexchange.com/questions/28764/one-sided-label-smoothing-in-gans

Thanks for your answer, but I was expecting someone to explain step by step formulation of this result. Still, I got helpful links


text:
Training speed in GPU vs CPU for LSTM
I was experimenting seq2seq model which was the bi-LSTM encoder/decoder with attention. When I compared the training times on GPU vs CPU while varying the batch size, I got

CPU on the small batch size (32) is fastest. It's even ~1.5 time
faster than GPU using the same batch size.
When I increase the batch size (upto 2000), GPU becomes faster than CPU due to the
parallelization.

(2) looks reasonable to me. However, I am a bit perplexed by the observation (1). The average sequence length is around 15~20. Even with the batch size 32, I expected that GPU should be faster but it turned out not. I used PyTorch LSTM.
Does it look normal? In RNN-style seq2seq, could CPU be faster than GPU?

How did you plot that graph? Did you draw it manually? Please, put your specific question in the title. Anyway, this could be because, with a smaller batch size, you need to move data more regularly to the GPU, and this could be expensive. However, this is just a guess.


text:
Considerations when doing image classification where the object is not the subject
I've come across two types of image classification tasks

cat/dog classification the whole picture is either a cat or a dog. Simple.
this image contains a cat classification. There's a whole chaotic scene, and the image may contain a cat nestled in there somewhere.

Type 2 seems to be way more prevalent in real life application. Here are just some examples:

Determining the sex of an insect. Maybe the male and female look pretty much the same, but the male has a small bump in some location that takes up a tiny part of the image.
Determining the presence of an animal call in an audio spectrogram.
Finding defects in road surfaces.

My question is: for task type 2, what are some key modifications we'd make to the normal approach of post-training on an ImageNet trained architecture like Resnet? Shouldn't the architecture be modified somehow to be more fitted to task type 2?
Before someone mentions using object detection algorithms I'd like to add the rule that we only have global image labels, not bounding box annotation or co-ordinates of any sort.


text:
Does the alpha/beta value of parent nodes change if the alpha beta value of the child node changes?
I want to do alpha-beta pruning on this tree:

Consider nodes J and K. K is the max. Therefore, node D has an alpha value of 20, node B has a beta value of 20.

Move to Node E. Pass the beta value of 20 to node E. Node L has an alpha value of 30, therefore, at this point 30 (alpha) > 20 (beta) and we can prune the E to M branch.

Now is my question. My original beta value at node B was 20, and the alpha value passed up to node A was 20. Then, in step 2, I changed the alpha value to be 30. Do i then change the beta value at node B to be 30, and the alpha value at node A to be 30? (and therefore pass 30 as the alpha value to node C)? Or do I keep the original value of 20 at node B, A and C?

I couldn't understand your question clearly, however, it think you are making a slim mistake. let's look at the flowing code from "Russell " and do pruning step by step:

Assume your are In D and you have traversed its both children, Alpha at D becomes 20. We then return back to B, Beta becomes 20 (note that Alpha is -Inf in D). We go to E then L and then back to E. 30 is greater than Beta, so the rest is pruned. Not that Alpha remains -Inf in E, though its value here isn't important. we go back to B the value of Beta remains on changed 20. If we go to F and return back to B Beta remains unchanged and is still 20 (Note that Alpha is still -Inf). We return to A, Alpha becomes 20, And Beta remains +Inf.
So, Considering the code, the max node does not get the Alpha value from its children, rather it decides on its own by setting Alpha = max(Alpha, and returned value of its child) after visiting each child. the same is true for Beta.


text:
Confusion between function learned and the underlying distribution
Let us assume that I am working on a dataset of black and white dog images.
Each image is of size $28 \times 28$.
Now, I can say that I have a sample space $S$ of all possible images. And $p_{data}$ is the probability distribution for dog images. It is easy to understand that all other images get a probability value of zero. And it is obvious that $n(S)= 2^{28 \times 28}$.
Now, I am going to design a generative model that sample from $S$ using $p_{data}$ rather than random sampling.
My generative model is a neural network that takes random noise (say, of length 100) and generates an image of the size $28 \times 28$. My function is learning a function $f$, which is totally different from the function $p_{data}$. It is because of the reason that  $f$ is from $R^{100}$ to $S$ and $p_{data}$ is from $S$ to $[0,1]$.
In the literature, I often read the phrases that our generative model learned $p_{data}$ or our goal is to get $p_{data}$, etc.,  but in fact, they are trying to learn $f$, which just obeys $p_{data}$ while giving its output.
Am I going wrong anywhere or the usage in literature is somewhat random?

You're right! The generative model $f$ is not the same as the probability density  (p.d.f.) function $p_{data}$. The kind of phrases you've referred to are to be interpreted informally. You learn $f$ with the hope that sampling a latent vector $z$
from some known distribution (from which it is easy to sample), results in $f(z)$ that has the probability density function $p_{data}$. However, merely learning $f$ does not give you the power to estimate what $p_{data}(x)$ is for some image $x$. Learning $f$ only gives you the power to sample according to $p_{data}(\cdot)$ (if you've learned an accurate such $f$).


text:
Simple DQN too slow to train
I have been trying to solve the OpenAI lunar lander game with a DQN taken from this paper
https://arxiv.org/pdf/2006.04938v2.pdf
The issue is that it takes 12 hours to train 50 episodes so something must be wrong.
import os
import random
import gym
import numpy as np
from collections import deque
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import Model

ENV_NAME = "LunarLander-v2"

DISCOUNT_FACTOR = 0.9
LEARNING_RATE = 0.001

MEMORY_SIZE = 2000
TRAIN_START = 1000
BATCH_SIZE = 24

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.01
EXPLORATION_DECAY = 0.99

class MyModel(Model):
    def __init__(self, input_size, output_size):
        super(MyModel, self).__init__()
        self.d1 = Dense(128, input_shape=(input_size,), activation="relu")
        self.d2 = Dense(128, activation="relu")
        self.d3 = Dense(output_size, activation="linear")

    def call(self, x):
        x = self.d1(x)
        x = self.d2(x)
        return self.d3(x)

class DQNSolver():

    def __init__(self, observation_space, action_space):
        self.exploration_rate = EXPLORATION_MAX

        self.action_space = action_space
        self.memory = deque(maxlen=MEMORY_SIZE)

        self.model = MyModel(observation_space,action_space)
        self.model.compile(loss="mse", optimizer=Adam(lr=LEARNING_RATE))

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def act(self, state):
        if np.random.rand() < self.exploration_rate:
            return random.randrange(self.action_space)
        q_values = self.model.predict(state)
        return np.argmax(q_values[0])

    def experience_replay(self):
        if len(self.memory) < BATCH_SIZE:
            return
        batch = random.sample(self.memory, BATCH_SIZE)
        state_batch, q_values_batch = [], []
        for state, action, reward, state_next, terminal in batch:
            # q-value prediction for a given state
            q_values_cs = self.model.predict(state)
            # target q-value
            max_q_value_ns = np.amax(self.model.predict(state_next)[0])
            # correction on the Q value for the action used
            if terminal:
                q_values_cs[0][action] = reward
            else:
                q_values_cs[0][action] = reward + DISCOUNT_FACTOR * max_q_value_ns
            state_batch.append(state[0])
            q_values_batch.append(q_values_cs[0])
        # train the Q network
        self.model.fit(np.array(state_batch),
                        np.array(q_values_batch),
                        batch_size = BATCH_SIZE,
                        epochs = 1, verbose = 0)
        self.exploration_rate *= EXPLORATION_DECAY
        self.exploration_rate = max(EXPLORATION_MIN, self.exploration_rate)

def lunar_lander():
    env = gym.make(ENV_NAME)
    observation_space = env.observation_space.shape[0]
    action_space = env.action_space.n
    dqn_solver = DQNSolver(observation_space, action_space)
    episode = 0
    print("Running")
    while True:
        episode += 1
        state = env.reset()
        state = np.reshape(state, [1, observation_space])
        scores = []
        score = 0
        while True:
            action = dqn_solver.act(state)
            state_next, reward, terminal, _ = env.step(action)
            state_next = np.reshape(state_next, [1, observation_space])
            dqn_solver.remember(state, action, reward, state_next, terminal)
            dqn_solver.experience_replay()
            state = state_next
            score += reward
            if terminal:
                print("Episode: " + str(episode) + ", exploration: " + str(dqn_solver.exploration_rate) + ", score: " + str(score))
                scores.append(score)
                break
        if np.mean(scores[-min(100, len(scores)):]) >= 195:
            print("Problem is solved in {} episodes.".format(episode))
            break
    env.close
if __name__ == "__main__":
    lunar_lander()

Here are the logs
root@b11438e3d3e8:~# /usr/bin/python3 /root/test.py
2021-01-03 13:42:38.055593: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-03 13:42:39.338231: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2021-01-03 13:42:39.368192: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.368693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.8095GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2021-01-03 13:42:39.368729: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-03 13:42:39.370269: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-03 13:42:39.371430: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-03 13:42:39.371704: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-03 13:42:39.373318: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-03 13:42:39.374243: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-03 13:42:39.377939: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-03 13:42:39.378118: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.378702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.379127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 13:42:39.386525: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3411185000 Hz
2021-01-03 13:42:39.386867: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4fb44c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-03 13:42:39.386891: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-03 13:42:39.498097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.498786: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4fdf030 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-03 13:42:39.498814: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
2021-01-03 13:42:39.498987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.499416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.8095GHz coreCount: 20 deviceMemorySize: 7.92GiB deviceMemoryBandwidth: 298.32GiB/s
2021-01-03 13:42:39.499448: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-03 13:42:39.499483: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2021-01-03 13:42:39.499504: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2021-01-03 13:42:39.499523: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2021-01-03 13:42:39.499543: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2021-01-03 13:42:39.499562: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2021-01-03 13:42:39.499581: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2021-01-03 13:42:39.499643: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.500113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.500730: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 13:42:39.500772: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-01-03 13:42:39.915228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-03 13:42:39.915298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2021-01-03 13:42:39.915322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2021-01-03 13:42:39.915568: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.916104: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-01-03 13:42:39.916555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6668 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Running
2021-01-03 13:42:40.267699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10

This is the GPU stats
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   53C    P2    46W / 198W |   7718MiB /  8111MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

As you can see, TensorFlow does not compute on the GPU but reserves the memory so I'm assuming it's because the inputs of the neural networks are too small and it uses the CPU instead.
To make sure the GPU was installed properly, I ran a sample from their documentation and it uses the GPU.
Is it an issue with the algorithm or the code? Is there a way to utilize the GPU in this case?
Thanks!

Programming issues are off-topic here. We focus on the theoretical aspects of AI. Please, take a look at https://ai.stackexchange.com/help/on-topic for more info. So, this post seems to be off-topic. If I'm wrong, please, let me know.

Would it be better on stack overflow?

I think so. On SO, you will probably find more people interested in these issues.

Would it be possible to move https://datascience.stackexchange.com/questions/87458/simple-dqn-to-slow-to-train
to SO?

I'm not a moderator at Data Science SE (so I can't directly do it myself), but I think if you ping a moderator there, he/she may do it. In any case, this question may also be on-topic on DS SE.

Is it training at all? Or is agent performance not improving over time? Q learning can be pretty unstable. I would recommend logging the sum of rewards received by the agent at the end of each episode and the model loss to help in the debugging process. The sum of rewards will show you if the agent is improving over time and the model loss will give you a rough idea about how stable the convergence is. I would recommend using tensorboard to log these metrics (https://www.tensorflow.org/tensorboard/get_started#using_tensorboard_with_other_methods). You will be able to monitor these metrics throughout the training process. You could also just print these metrics at the end of every epoch and monitor them in your console. You really just need someway to see what's going on during training.
In the paper you linked, it also mentioned double q learning, which in your code does not seem to be implemented. Vanilla q learning can have a reputation of being overoptimistic in the values that it assigns to states. This results in compounding approximation errors, which tend to destabilize learning. Using double q learning may help speed up convergence. If you need help with double q learning check out this paper: https://arxiv.org/pdf/1509.06461.pdf, and this github page: https://github.com/jihoonerd/Deep-Reinforcement-Learning-with-Double-Q-learning/blob/master/ddqn/agent/ddqn_agent.py
If you use double q learning, you may have to write your own custom training loop. This can be achieved by using the gradient tape object. Make sure to wrap this new function in a tf.function decorator. This will tell the TensorFlow back-end to compile that bit of code, making it run faster (https://www.tensorflow.org/guide/function). There are also some handy speed up tips in this post (https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic). They even wrap the environment step functions in tf.functions.The article uses actor-critic, which is a combination of policy gradient and q learning techniques, but you can swap out their neural network update code with the q learning functionality that you need.
If you need help with double q learning check out this paper: https://arxiv.org/pdf/1509.06461.pdf, and this github page: https://github.com/jihoonerd/Deep-Reinforcement-Learning-with-Double-Q-learning/blob/master/ddqn/agent/ddqn_agent.py

Thanks for the detailed answer. I was trying to understand Q learning before switching to double Q learning. The agent overall is learning, it doesn't land yet but it learned how to fly between the flags and the score is improving. I will try to implement the optimization mentioned in your answer but I would like to know based on your experience if the slowness observed is normal. It played only 50 episodes in 12 hours (overnight)

I haven't messed with the lunar lander environment specifically, but the OpenAI page for the environment mentions that an environment ends only when the lander crashes or comes to a rest. It could be the case that the lander is just flying for a really long time, which would extend the timesteps for a given episode. Maybe try swapping lunar lander out for cart-pole and see if you still get long training times. Cart-pole has a fixed number of timesteps per episode. If you still get long training times, something else in your code is likely wrong.


text:
What are the disadvantages to using a distance metric in character recognition prediction
I am reading this paper, that is discussing the use of distance metrics for character recognition predicton.
I can see the advantages of using a distance metrics in predictions like character recognition: You can model a set of known character's pixel vectors, and then measure a new unseen vector against this model, get the distance, and if the distance is low, predict that this unseen vector belongs to a particular class.
I'm wondering if there's any disadvantages to using distance metrics as the cost function in character recognition? For example, I was thinking maybe the distance calculation is slow for large images (would have to calculate the distance between each item in two long image vectors)?

As I see it, the question boils down to the comparison between distance (function/metric) based Optical Character Recognition (OCR) and (for example) OCR done by means of Convolutional Neural Networks (CNNs). Particularly, it focuses on the cons of the former option.
There are a few potential problems associated with using distance based OCR systems.
First of all, this approach requires an appropriate distance metric to deliver good results. Different types of distance metrics/functions are sensitive to different features in the input images. For example, some functions might penalize absolute differences, while others penalize squared differences, where the latter punishes differences of magnitude $> 1$ stronger than the former, while the former penalizes differences of (absolute) magnitude $< 1$ stronger than the latter. Which type of distance metric works best for a given problem has commonly to be determined empirically.
Also, distance based OCR systems may possibly be more sensitive to, for example, different levels in illumination than CNNs. While both sorts of classifiers could profit from data augmentation (i.e. adding variations of the existing training data to the dataset), OCR based on CNNs has the benefit that CNN training procedures produce classifiers that commonly generalize well to slight variations in the incoming data, while some slight novel tilt (or again variation in illumination) may break the distance based classification procedure, but may have not too detrimental effects on CNN based classifiers due to their rather strong generalization capabilities.
Of course, one can also try to increase the robustness of distance based OCR systems, but this is commonly associated with developing exhaustive preprocessing pipelines to standardize the appearance of incoming character images.
Thus, in terms of system design, it is often easier to set up a CNN-based neural network architecture and train it (possibly drawing upon regularization to boost generalization of the trained system even further) than trying to design complicated preprocessing strategies to shift, rotate, and normalize character images exhaustively to boost performance of a distance based OCR system.
To sum up, after all, there are a lot of potential issues related to using plain distance (metric/function) based OCR systems, most of which, however, can strongly be alleviated when drawing upon clever (pre-)processing pipelines to standardize input images before performing distance based classification.
From all those issues mentioned above, the strongest disadvantage of this technique, however, (which cannot be alleviated so easily) is that it doesn't generalize too easily to novel input data, where CNNs might perform better with appropriate training data (+ augmentation and regularization in general).
You also mentioned the aspect of computational cost associated with distance based classification. First of all, I am not an expert in computational complexity/cost related questions. However, most image related computations can efficiently be executed on graphics cards. So, subtracting images pixel-wise, for example,is not very costly given it can be executed on a graphics card. But of course, when using distance based OCR, the computational cost associated with computing distances and comparing distance values scales with the reference dataset size.
Using CNNs, the time needed to perform a classification is always the same irrespective of the training dataset size.


text:
Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?
I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard.
My implementation doesn't batch XP together just applies the update after every episode. It turns out my implementation is more sample efficient than the reference implementation on simple problems (like CartPole or LunarLander) even though I haven't added the critic yet! Of course this could be due to number of reasons, for example I've only done parameter search on my implementation.
But it would make sense anyway: bigger batch size is generally considered better only because the GPU is faster processing many samples parallel. Is this the reason here? It would make sense but it is surprising for me as I thought sample efficiency is considered more important than computing efficiency in RL.


text:
Semantic segmentation CNN outputs all zeroes
I'm using MATLAB 2019, Linux, and UNet (a CNN specifically designed for semantic segmentation). I'm training the network to classify all pixels in an image as either cell or background to get segmentations of cells in microscopic images. My problem is the network is classifying every single pixel as background, and seems to just be outputting all zeroes. The validation accuracy improves a little at the very start of the training but than plateaus at around 60% for the majority of the training time. The network doesn't seem to be training very well and I have no idea why.
Can anyone give me some hints about what I should look into more closely? I just don't even know where to start with debugging this.
Here's my code:
    % Set datapath
    datapath = '/scratch/qbi/uqhrile1/ethans_lab_data';

    % Get training and testing datasets
    images_dataset = imageDatastore(strcat(datapath,'/bounding_box_cropped_resized_rgb'));
    load(strcat(datapath,'/gTruth.mat'));
    labels = pixelLabelDatastore(gTruth);
    [imdsTrain, imdsVal, imdsTest, pxdsTrain, pxdsVal, pxdsTest] = partitionCamVidData(images_dataset,labels);

    % Weight segmentation class importance by the number of pixels in each class
    pixel_count = countEachLabel(labels); % count number of each type of pixel
    frequency = pixel_count.PixelCount ./ pixel_count.ImagePixelCount; % calculate pixel type frequencies
    class_weights = mean(frequency) ./ frequency; % create class weights that balance the loss function so that more common pixel types won't be preferred

    % Specify the input image size.
    imageSize = [512 512 3];

    % Specify the number of classes.
    numClasses = 2;

    % Create network
    lgraph = unetLayers(imageSize,numClasses);

    % Replace the network's classification layer with a pixel classification
    % layer that uses class weights to balance the loss function
    pxLayer = pixelClassificationLayer('Name','labels','Classes',pixel_count.Name,'ClassWeights',class_weights);
    lgraph = replaceLayer(lgraph,"Segmentation-Layer",pxLayer);

    %% TRAIN THE NEURAL NETWORK

    % Define validation dataset-with-labels
    validation_dataset_with_labels = pixelLabelImageDatastore(imdsVal,pxdsVal);

    % Training hyper-parameters: edit these settings to fine-tune the network
    options = trainingOptions('adam', 'LearnRateSchedule','piecewise', 'LearnRateDropPeriod',10, 'LearnRateDropFactor',0.3, 'InitialLearnRate',1e-3, 'L2Regularization',0.005, 'ValidationData',validation_dataset_with_labels, 'ValidationFrequency',10, 'MaxEpochs',3, 'MiniBatchSize',1, 'Shuffle','every-epoch');

    % Set up data augmentation to enhance training dataset
    aug_imgs = {};
    numberOfImages = length(imdsTrain.Files);
    for k = 1 : numberOfImages
        % Apply cutout augmentation
        img = readimage(imdsTrain,k);
        cutout_img = random_cutout(img);imwrite(cutout_img,strcat('/scratch/qbi/uqhrile1/ethans_lab_data/augmented_dataset/img_',int2str(k),'.tiff'));
    end
    aug_imdsTrain = imageDatastore('/scratch/qbi/uqhrile1/ethans_lab_data/augmented_dataset');
    % Add other augmentations
    augmenter = imageDataAugmenter('RandXReflection',true, 'RandXTranslation',[-10 10],'RandYTranslation',[-10 10]);
    % Combine augmented data with training data
    augmented_training_dataset = pixelLabelImageDatastore(aug_imdsTrain, pxdsTrain, 'DataAugmentation',augmenter);

    % Train the network
    [cell_segmentation_nn, info] = trainNetwork(augmented_training_dataset,lgraph,options);

    save cell_segmentation_nn

Similar to other answers, I don't know Matlab that well but you could try the following steps to debug your problem.
Make sure you can overfit to a single instance
from your dataset, pull out a single image with a good amount of true positives in it. Duplicate that images B times (where B = Batch Size) and then try to train your network with only that small dataset. If you can't overfit to a single instance, then something is really wrong and you should validate all functional aspects of your network. If you can overfit, then it's probably more of an algorithmic or data imbalance issue.
Validate Functional Aspects of the Network
Make sure your input data and labels are correct.
Validate that your images are being correclty inputted into the network. You can do this by printing out any images right before they go into the training function. For the labels, make sure to manually inspect a few labels to be sure that the labels correctly match up the images.
Make sure that your loss function is correct.
Add a unit test or two to your loss function to validate that it is doing what it should be doing. Create a simiple example that you can easily validate.
Validate the rest of the non-algorithmic functionality
Anything that isn't a design choice should be validated. Make sure that the weights are the correct sizes, each layer has the correct number of weights, the intermediate features have the correct shape, etc.
Data Imbalance
If you were able to overfit to a single image, and validated the functional aspects of your network then you may be facing a data imbalance issue. Look at your dataset and see what % of your instances are true positives vs. true negatives. If you have an exteme imbalance (like 10% 90%) or something like that, then build a dataset that is more balanced and see if you can fit your data. If you fit the data with that more balanced dataset, then there's plenty of ways to fix your data imbalance issue. Google around for data imbalance and you should get a few good ideas. Some include focal loss, upsampling, etc...
Check your receptive field
The receptive field on a network is basically the area that can be looked at by the network for a specific region. This is controlled by the size of the convolution filters, stride, etc. If the data that you are segmenting doesn't fit within the receptive field of the model, then you might be saturating the loss function. tldr try playing around with kernel sizes.

I have never used MATLAB for ml before, so it is difficult for me to understand all your code. My first association to your problem is class imabalance. Since you seem to have got a handle on that, the problem could be dying ReLU or bloated activations. To check if the ReLU is dying, you could look at the activations of the early layers of your network. If many values are zero, it should be dying ReLU.


text:
Can you give me a piece of advise of the network sructure that would be suitable for my task?
I have 2 small images. They are basically the same, but differ in rotation and size. I should estimate the parameters for affine transform to get them similar. What network structure can be suitable for this task? For example, those based on convolutional networks did badly, because the pictures are too small.

Does it have to be with NNs or do you want to use image processing tools

Just use IMGAug library for Applying the 'zoom' augmentation on the images and a convnet (or even MLP) would have no problem in this task. Zooming on the image would be more than enough to use as, as long as your zoom is of sufficient power.
So apply "zoom" of the same factor on all images, make them bigger, and Convolutional Neural Networks would do fine. Also, If color is not an important feature, then you would be better off rescaling images b/w 0 and 1 and making them GrayScale

Instead of NNs, you can use RANSAC algorithm to calculate homography matrix, but first you need to find feature points. However, if your images are blob-like, you may not get such a successful results. Here some presentations for better understanding: cs.umd notes and csail.mit notes
(Also, there might be better image processing tools.)


text:
VAE giving near zero output when latent space dimension is large
I'm training a VAE to reconstruct some input (channels picked up by some MIMO BS for context) and I ran an experiment on the training set to see how the performance improves with the latent space dimension.
My VAE structure is as follows : Input : 2048 -> 1024 -> 512 -> Latent space dimension -> 512 -> 1024 -> Output : 2048
Here is what I get in terms of relative error when the latent space dimension goes from 2 to 100 :

Everything works as expected at the beginning, but the error starts rising up at around 50 and I have no idea why. With a large latent space dimension, the output is orders of magnitude smaller than the input, which explains the relative error of value 1.
Here is the same figure when I run the exact same experiment but with a normal autoencoder this time.
This time the results are consistent.
What's wrong with my VAE ?

This is a really interesting phenomena, my guess would be the network is unable to learn a continuous representation in the number of epochs you provided for learning. Try upping the number of epochs in the VAE for learning and see if the error drops.

@Recessive I already tried that. Went from 50 epochs to 100. Nothing changed, had the same exact behavior.

The question of how the latent space size influences the performance is answered in this & this Cross Validated posts. In short: the amount of training data upper bounds the latent dimension, and a too large latent space leads to overfitting. This however seems opposite to what you report, maybe your model becomes too large and is thus underfitting?

Old question but hopefully OP is still active.
I also agree with recessive that perhaps it is struggling to create a continuous representation of your data in the latent space. Have you looked at your KL loss and your reconstruction loss independantly?
I was wondering: have you tried changing your input? If you reduce it from 2048 dimensions to 1024, does the dip in your curve also shift proportionally?
Have you also looked at what the latent space and the reconstructions looks like? what sort of errors is it making?

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

It's been some time now so I don't know what I did of the code so I won't be able to test your suggestions. However, if I remember correctly, I solved the problem by using tanh activation functions instead of relu that I was using originally. I still have no idea what the origin of this behavior is or why changing the activation function fixes it tho.


text:
How to predict the best from a set of messages - best practice
Suppose I have a set of messages A,B,C,D and I want to produce the best message for a website user at a given time.
For training I plan to show random users a random single message [A/B/C/D] and fill these columns (i'm simplifying the data for illustration)

converted before
funnel state (e.g awareness, search, decision)
number of page views
message shown [A-D]
Time to convert (this will be updated later if there is a conversion)

I want to predict what is the best message to show to a specific user in order to maximise the chance of conversion (=min time to convert).
I'm not sure how to represent this for training and inference. Its not a simple prediction like predicting one of the given data points.
One option is to run prediction of time to buy for each of the messages but
1- its not efficient
2- It will prefer messages that are shown closer to purchase time regardless if they fit the current user time.

One way you can definitely approach the problem is by using (Deep) Reinforcement Learning (DRL).
YouTube is actually using DRL as well to suggest videos to users in order to maximize users' engagement with their website. For more information (and further references to papers explaining how other major companies implement their recommendation systems), see this paper. Just as a bit of motivation.
Other research into that direction could be found here.
Actually, there are different DRL algorithms available and each might be suitable for approaching the problem you described above from a slightly different angle.
The way how I would roughly approach your problem is as follows:
As I understand it, the goal is the following.
You want to show either of four messages to a user (only one at a time) and always display the message, which is expected to reduce some metric (=conversion time), to the user.
In that case, an DRL algorithm, or agent, could directly be trained on minimizing the conversion time without having to explicitly produce conversion time estimates per message & user. That might make phrasing the problem at hand much easier (in terms of modelling the learning task).
The outcome of training an DRL agent consists of a so-called optimal policy network, which dictates the agent which message to display to a given user, upon observing the provided user data, such that conversion time gets minimized for that user. The aforementioned network is a simple artificial Neural Network (NN) or Recurrent NN (RNN).
Training the DRL agent will consist of two steps being repeated many times in alternating order.
One step will include the sampling/generation of training data.
For this step, the algorithm (initially being untrained) will be applied to the problem at hand and predict messages for given users. Then, (in addition to the provided user data that was fed as input to the agent) each user's response (i.e. the conversion time) is recorded. This can later be used as the reward that the agent receives for having selected the selected message given the provided user data. Here, a low conversion time translates to a high reward.
In the second step, the agent's policy network will be trained in order to make the agent's predictions more accurate in the next round of generating training data. Here, the data recorded during the previous step will be used.
How the policy network is optimized largely depends on which DRL algorithm you choose. The range of different methods varies vastly. Some algorithms try to estimate utilities (i.e. measures of goodness) per action given a certain state (here, state = provided user data used to generate a prediction). In this case, an action would be choosing message A or choosing message B etc. A popular algorithm implementing this procedure is Deep Q-Learning.
However, if you rather want to predict probabilities per action, then Proximal Policy Optimization might rather be what you are looking for.
Upon convergence, i.e. stabilization of the goal metric (=convergence time per user), the algorithm has arrived at its optimal policy (ideally speaking).
The only drawback with this approach is that DRL usually requires quite a lot of training. But maybe you are lucky and pre-trained recommendation system models exist.
If there is still anything unclear, feel free to ask in the comments.


text:
What place do Agent Communications Language have in Multi-Agent Systems nowadays?
I am currently working on implementing a Multi-Agent System for Smart Grids.
There's a lot of literature for that and some things confuse me. I have read that there is FIPA, which aimed to create a unified Agent Communication Language. So multiple Agents are talking to each other and FIPA specifies how the messages should be sent and processed. However, it is pretty old.
In newer papers, where Multi-Agent Reinforcement Learning Algorithms are proposed, FIPA or generally any ACL isn't mentioned. I believe that is because in MARL, communication is done by observing the states of the other agents, rather than communicating explicitly. Also in MARL, the decision making is not based on negotiation like in FIPA but with the learned policy.
I am now super confused if I got it right.
Is FIPA still a thing I should worry about when I design my Multi-Agent System?
Is there any other thing to handle communication in MARL other than sharing states?
Any help would be really appreciated, thank you very much :)

This is a very interesting question. If you found any answer, can you share it with us?

So I don't know how correct my thoughts are, if somebody knows better I am open to suggestions.
However my understanding has changed that there are several aspects to Multi-Agent Systems.
In most AI related fields, communication is not the research focus and most things get done in simulations only, whereas in software engineering, communication protocols like FIPA or ACL are the main interest. However I didn't find lots of new literature using FIPA, my guess is that it either is used in already existing frameworks (like JADE) or other protocols are used (there is new tech since 2006 :D).


text:
Aggregating 2D object detections into 3D object detections
I have a data set of 3D images with some bounding box annotations. The images are too large to train something like YOLO 3D (would run out of memory), so I instead created slices of the 3D images with corresponding 2D bounding boxes and trained a 2D object detector. During inference, I assemble the 2D detections into 3D detections. I constructed some simple heuristics to do that, which works fine, but I am wondering if there aren't any established methods of doing that.
I appreciate if you could point me in the right direction.