text:
Are convolutional neural networks inspired by the human brain?
The Deep Learning book by Goodfellow et al. states

Convolutional networks stand out as an example of neuroscientiﬁc principles inﬂuencing deep learning.

Are convolutional neural networks (CNNs) really inspired by the human brain?
If so, how? In particular, what structures within the brain do CNN-like neuron groupings occur?

Yes, CNNs are inspired by the human brain [1, 2, 3]. More specifically, their operations, the convolution and pooling, are inspired by the human brain. However, note that, nowadays, CNNs are mainly trained with gradient descent (GD) and back-propagation (BP), which seems not to be a biologically plausible way of learning, but, given the success of GD and BP, there have been attempts to connect GB and BP with the way humans learn [4].
The neocognitron, the first convolutional neural network [1], proposed by Kunihiko Fukushima in 1979-1980, and described in the paper Neocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position, already uses convolutional and pooling (specifically, averaging pooling) layers [1]. The neocognitron was inspired by the work of Hubel and Wiesel described in the 1959 paper Receptive fields of single neurones in the cat's striate cortex.
Here is an excerpt from the 1980 Fukushima's paper.

The mechanism of pattern recognition in the brain is little known, and it seems to be almost impossible to reveal it only by conventional physiological experiments. So, we take a slightly different approach to this problem. If we could make a neural network model which has the same capability for pattern recognition as a human being, it would give us a powerful clue to the understanding of the neural mechanism in the brain. In this paper, we discuss how to synthesize a neural network model in order to endow it an ability of pattern recognition like a human being.
Several models were proposed with this intention (Rosenblatt, 1962; Kabrisky, 1966; Giebel, 1971; Fukushima, 1975). The response of most of these models, however, was severely affected by the shift in position and/or by the distortion in shape of the input patterns. Hence, their ability for pattern recognition
was not so high.
In this paper, we propose an improved neural network model. The structure of this network has been suggested by that of the visual nervous system of the vertebrate. This network is self-organized by "learning without a teacher", and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their position nor by small distortion of their shapes. This network is given a nickname "neocognitron", because it is a further extention of the "cognitron", which also is a self-organizing multilayered neural network model proposed by the author before (Fukushima, 1975)

However, Fukushima did not train the neocognitron with gradient descent (and back-propagation) but with local learning rules (which are more biologically plausible), and that's probably why he doesn't get more credit, as I think he should.
You should read at least Fukushima's paper for more details, which I will not replicate here.
Section 9.4 of the Deep Learning book also contains details about how CNNs are inspired by neuroscience findings.

In the eye, the retinal ganglion cells have a receptive field that is equivalent to some types of convolution filters, most of them edge detectors.
The brain is a big unknown, nobody knows how it does to organize, memorize, create concepts, learns the language, ... . Thus, it is not possible to establish a parallelism.
In particular, brain has a capacity of handle independence in scale and rotation that CNN's are not able to reproduce.
As general remark about NN and brain: even when it is always said that "neural network cells" are "inspired" in biological neurons, there are critical differences that made this similitude only a "inspiration". Thus, comparison of a CNN or any other kind of NN with brain is always a fuzzy comparison. The biggest difference is probably the learning capacity: the human brain learns by itself, while the neural network needs an external system (the back-propagation algorithm) that feeds the NN with the learned parameters.

Your first and third paragraphs are interesting, particularly the third because some (many?) ANNs which have CNN sub-parts use them as input layers and these then feed into non-CNNs such as LSTMs.

Are you sure about brain (or at least our visual system) being invariant to rotation? I don't think that's the case (at least generally), as there are experiments that show we cannot properly recognise faces and texts if turned upside down. In that sense, CNNs also require data augmentation, which is basically training the too on rotated versions of the input data, and we can also learn to read upside down if trained to do so.

I'm not convinced about the fact that the brain does not have external aid or guidance for learning: we have hormones and other chemical mechanisms, and I wouldn't rule those out as candidate participants in our learning process?


text:
If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?
Q-Learning is guaranteed to converge if $\alpha$ decreases over time.
On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each action-state pair is selected an infinite number of times and if $\alpha$ decreases appropriately over time.
It seems that it would be better if $\alpha$ increased over time, as it is the learning rate of the gradient of Q-function values ($R+\gamma\max_aQ(S',a)-Q(S,A)$), and, initially, they start off incredibly inaccurate, because they are initialized arbitrarily and over time converge to the true values, hence you'd want to weight them more as time increases rather than decrease it?
Why is this a convergence criterion?

Why is this a convergence criterion?

It is because $R$ and $S'$ are stochastic. A large learning rate applied when these values have variance would not converge to mean, but would wander around typically within some value proportional to $\alpha\sigma$ of the true value, where $\sigma$ is the standard deviation of the term $R + \gamma\text{max}_aQ(S',a)$. If you reduce $\alpha$ towards zero, then this expected error will also reduce to zero.
For deterministic environments, it should be possible to prove convergence with large $\alpha$.

In the special case of static policy, tabular learning and $\alpha = \frac{1}{N(s,a)}$ where $N(s,a)$ is number of visits to state $s$, action $a$, then the expected error for each Q value is the MSE from basic stats i.e. $\frac{\sigma_{TD}}{\sqrt{N(s,a)}}$

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Do I need to denormalise results in linear regression?
I have learned so far how to linear regression with one or multiple features. So far, so good, everything seems to work fine, at least for my first simple examples.
However, I now need to normalise my features for training. I'm doing this by calculating the mean and the standard deviation per feature, and then calculate the normalised feature by subtracting the mean, taking the absolute value, and dividing by the standard deviation. Again, so far, so good, the results of my tensors which I use for training look good.
I understand why I need to normalise input data, and I also understand why one can do it like this (I know that there are other ways as well, e.g. to map values to a 0-1 interval).
Now I was wondering about two things:

First, after having trained my network, when I want to make a prediction for a specific input – do I need to normalise this as well, or do I use the un-normalised data? Does it make a difference? My gut feeling says, I should normalise it, as it should make a difference, but I'm not sure. What should I do here, and why?
Second, either way, I get a result. Now I was wondering whether I need to denormalise this? I mean, it should make a difference, shouldn't it? If so, how? How do I get from the normalised result value to a denormalised one? Do I just need to reverse the calculation with mean and standard deviation, to get the actual value?

It would be great if someone could shed some light on this.

Okay, I figured it out by myself, simply by trial and error:
When you normalize your training and test data, you also need to normalize the input you want to have a prediction for. You will also need to denormalize the result, to get a reasonable prediction.
Specifically, in my case, this means:

First I calculated the mean and the standard deviation, and then subtracted the mean and divided by standard deviation, to normalize my training and test data.
So to normalize the input I want to have a prediction for, I also subtract the mean and divide by the standard deviation.
Finally, when I get a result, I multiple by the standard deviation, and add the mean.

This way, I get reasonable results

Maybe it was just accidentally that the results after reverting the transformation made sense? What should I do instead?

Neural network with a single dense layer, with a single node using MSE and SGD. Just a really, really simple example.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Should I be balancing the data before creating the vocab-to-index dictionary?
My question is about when to balance training data for sentiment analysis.
Upon evaluating my training dataset, which has 3 labels (good, bad, neutral), I noticed there were twice as many neutral labels as the other 2 combined, so I used a function to drop neutral labels randomly.
However, I wasn't sure if I should do this before or after creating the vocab2index mappings.
To explain, I am numericizing my text data by creating a vocabulary of words in the training data and linking them to numbers using enumerate. I think to use that dictionary of vocab2index values to numericise the training data. I also use that same dictionary to numericise the testing data, dropping any words that do not exist in the dictionary.
When I took a class on this, they had balanced the training data AFTER creating the vocab2index dictionary. However, when I thought about this in my own implementation, it did not make sense. What if some words from the original vocabulary are gone completely, then we aren't training the machine learning classifier on those words, but they would not be dropping from the testing data either (since words are dropping from X_test based on whether they are in the vocab2index dictionary).
So should I be balancing the data BEFORE creating the vocab2index dictionary?
I linked the code to create X_train and X_test below in case it help.
def create_X_train(training_data='Sentences_75Agree_csv.csv'):
    data_csv = pd.read_csv(filepath_or_buffer=training_data, sep='.@', header=None, names=['sentence','sentiment'], engine='python')
    list_data = []
    for index, row in data_csv.iterrows():
        dictionary_data = {}
        dictionary_data['message_body'] = row['sentence']
        if row['sentiment'] == 'positive':
             dictionary_data['sentiment'] = 2
        elif row['sentiment'] == 'negative':
             dictionary_data['sentiment'] = 0
        else:
             dictionary_data['sentiment'] = 1 # For neutral sentiment
        list_data.append(dictionary_data)
    dictionary_data = {}
    dictionary_data['data'] = list_data
    messages = [sentence['message_body'] for sentence in dictionary_data['data']]
    sentiments = [sentence['sentiment'] for sentence in dictionary_data['data']]

    tokenized = [preprocess(sentence) for sentence in messages]
    bow = Counter([word for sentence in tokenized for word in sentence])
    freqs = {key: value/len(tokenized) for key, value in bow.items()} #keys are the words in the vocab, values are the count of those words

    # Removing 5 most common words from data
    high_cutoff = 5
    K_most_common = [x[0] for x in bow.most_common(high_cutoff)]
    filtered_words = [word for word in freqs if word not in K_most_common]

    # Create vocab2index dictionary:
    vocab = {word: i for i, word in enumerate(filtered_words, 1)}
    id2vocab = {i: word for word, i in vocab.items()}
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized]

    # Balancing training data due to large number of neutral sentences
    balanced = {'messages': [], 'sentiments':[]}
    n_neutral = sum(1 for each in sentiments if each == 1)
    N_examples = len(sentiments)
    # print(n_neutral/N_examples)
    keep_prob = (N_examples - n_neutral)/2/n_neutral
    # print(keep_prob)
    for idx, sentiment in enumerate(sentiments):
        message = filtered[idx]
        if len(message) == 0:
            # skip this sentence because it has length 0
            continue
        elif sentiment != 1 or random.random() < keep_prob:
            balanced['messages'].append(message)
            balanced['sentiments'].append(sentiment)

    token_ids = [[vocab[word] for word in message] for message in balanced['messages']]
    sentiments_balanced = balanced['sentiments']

    # Unit test:
    unique, counts = np.unique(sentiments_balanced, return_counts=True)
    print(np.asarray((unique, counts)).T)
    print(np.mean(sentiments_balanced))
    ##################

    # Left padding and truncating to the same length
    X_train = token_ids
    for i, sentence in enumerate(X_train):
        if len(sentence) <=30:
            X_train[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_train[i] = sentence[:30]
    return vocab, X_train, sentiments_balanced

def create_X_test(test_sentences, vocab):
    tokenized = [preprocess(sentence) for sentence in test_sentences]
    filtered = [[word for word in sentence if word in vocab] for sentence in tokenized] # X_test filtered to only words in training vocab
    # Alternate method with functional programming:
    # filtered = [list(filter(lambda a: a in vocab, sentence)) for sentence in tokenized]
    token_ids = [[vocab[word] for word in sentence] for sentence in filtered] # Numericise data

    # Remove short sentences in X_test
    token_ids_filtered = [sentence for sentence in token_ids if len(sentence)>10]
    X_test = token_ids_filtered
    for i, sentence in enumerate(X_test):
        if len(sentence) <=30:
            X_test[i] = ((30-len(sentence)) * [0] + sentence)
        elif len(sentence) > 30:
            X_test[i] = sentence[:30]
    return X_test

If you look at the words in your dictionary (vocab) before/after pruning, most likely you'd see there isn't a lot of difference, not so much to affect your model performance.
In fact, creating a dictionary and model training are two more or less indpendent processes. To make your life easier, you could find the largest dev set you can find for building your vocab (excluding test set), and freeze it for all subsequent ETL/modeling.  This way you don't have to deal with dictionary versioning, for example, after choosing different subsets of your training data.
Also if you have compute capacity, I'd suggest to upsample positive/negative classes instead, because those neutral samples you're dropping do have signal on the use of language and which borderline ambiguous samples should be treated as neutral.

Thanks for this response! I never thought about just finding a large dev set but that makes a lot of sense. Could you explain what you mean by upsampling though? Thanks!

Sure. So say your class distribution is pos=5, neg=5, neutral=10, you do sampling with replacement from pos and neg until you have 10 samples for all classes. In pandas you can easily do this with df.sample and if you do a bit of search you'll find other ways/libs to help you with that.

Ah okay, so this would create duplicate sentences in the training correct? Have you also heard of the class_weights parameter of most sklearn models? It seems to balance classes by assigning weights. Someone else pointed it out to me and I think it does what I want but much nicer. Wanted to ask if you knew it as well? Thanks!

yes correct. Indeed an alternative is class_weights, but depends if the model you want to use implements it.

Hey I had a further ponder over finding a large dev set but something occurred to me. Wouldn't this encode various words that exist in the dev set and testing set but not in the training set? I'm wondering what the point would be of keeping those words in the test set if the classifier isn't even trained on them. At the moment, when I use BoW I drop the words not in the training set from the test set. What's the intuition behind keeping them using a dev set? Would they just be ignored by the classifier anyway? Or rather, how do trained classifiers deal with those kinds of words? Thanks!

You don't really have to go the extra mile for finding the largest dev set. Using your raw training set (tokenized) would be enough. Your observation is currect, when train corpus doens't contain a word, there will be no signal learned from that single word. In practice this isn't a problem, because you look at more than one word for making a decision (n-grams, conv nets, RNNs, transformers, etc). If a word doesn't appear in the training set, most likely it's not significant enough (rare) to have an effect. In practice a vocab of 30k-50k words is enough to make a decent classifier.

You freeze the vocab mostly for convenience. You shouldn't be doing tokenization and vocab building by hand. Most of the tools out there will take care of picking the most common N words, will tokenize the text properly, and map any unseen (unknown) word to a special <unk> token. That will be the placeholder to represent all rare/out-of-vocabulary (oov) tokens and your model will also learn to play nicely with it (usually treat it as noise).


text:
What does "off-the-shelf" mean?
I encountered the phrase/concept off-the-shelf CNN in this paper in which authors used off-the-shelf CNN representation, OverFeat, with simple classifiers to address different recognition tasks.
As I understand it correctly, it literally means something is ready to be used for a task without alteration.
Can somebody explain in simple words what off-the-shelf CNN technically means in the context of AI and convolutional neural networks?

I've encounter this term in couple of papers, usually they propose concepts where a component can be replaced by any existing model. For example if we think about GANs (generative adversarial nets) they proposed adversarial training where each of their components, the generator and the discriminator, could be off the self CNN. That means that we could take any CNN network which is suitable for the task we are trying to solve, for example AlexNet, and just plug it in.

The dictionary's definition of off-the-shelf is

used to describe a product that is available immediately and does not need to be specially made to suit a particular purpose

The same dictionary provides several examples

You can purchase off-the-shelf software or have it customized to suit your needs.
If you have complex needs, we don't recommend that you buy software off the shelf.
For this, off-the-shelf algorithms included in the robot's programming libraries are used.

So, your intuition is correct! An off-the-shelf model, software, product, etc., is any model, software or, respectively, product that would be easily (or immediately) available or applicable to the specific context, but, at the same time, it may also be applicable to many other contexts or problems.
An off-the-shelf convolutional neural network is thus a typical or standard CNN that can be applied immediately in that context (but that is potentially applicable to many other contexts or problems). Examples of CNNs that could be used as off-the-shelf models are AlexNet or LeNet-5, but the actual choice depends on the context and needs.
An off-the-shelf model can also be a baseline model (i.e. a very simple model that is used in experiments as the model that every other model should outperform), but not necessarily.
Alex Graves uses this term/expression in his paper Practical Variational Inference for Neural Networks.

In the context of the linked paper, it looks like the modern term for what they are doing is transfer learning - taking a pre-trained CNN for large scale image task and using it for something else by swapping out the last layer or two (or simply pre-processing the images by collecting the last feature layer from them before feeding to new ML). That is probably the more popular term, and maybe worth linking to something explaining that for the OP.

@NeilSlater Ok, I will maybe read that paper, but I think that this doesn't change the main meaning of "off-the-shelf". If I understand what you're saying, they are still using an "off-the-shelf" model (although a trained one). I don't think "off-the-shelf" is only used in the context of transfer learning. I am sure I've seen it being used in other contexts too (but I admit I would need to search for the specific papers).


text:
Is there any literature on the design of dialogue systems for interviews and questionnaire administration?
For my master thesis I am working on a dialogue system that should be deployed in hospitals to administer simple questionnaires to patients. I already did literature research and I'm fine with what I found since I don't have to replicate something which has been already done, but I noticed that there are really few papers regarding this specific 'robot interviewer' topic.
Let me explain the task a bit more in details: in a real interview a human interviewer usually starts with greetings and with an explanation of the questionnaire to administer, and then he/she starts asking some more or less structured questions to the person to interview. The idea here is to replace a human interviewer with a dialogue system.
Now, at a first glimpse it seems like a task that can be easily hand coded, and indeed lot of real application simply use systems in which specific questions are stored in memory in association with some already made answers to choose from (here's an example), and the system simply show them (or read in the case of humanoid robots) to the people being interviewed, the system then wait for the answer and then move on with the next questions.
The point is that in real interviews the conversation flow is obviously much more smooth and natural. A human being can detect doubts in the voice of the interviewed person, which can also explicitly ask for explanations, a human being can also understand when an answer comes with emotional implications ('yes I feel sad every day') and we are able to automatically react to these hidden implications with emotional fillers ('I'm sorry to hear that'). All these aspects require to train some natural language understanding module in order to be replicated in an artificial agent (and this is actually what I'm currently working on), so I though I would have found more papers on this.
Now, despite having found a tons of paper related to open domain dialogue systems, affective and attentive systems, even systems able to reply with irony, I did not found many papers about dialogue systems for smooth interviews or questionnaire administration, which in my opinion sounds like an much easier task to tackle (especially if compared to open domain conversations).
The only two papers that I found which truly focused on interviewer systems are:

SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support
A job interview dialogue system with autonomous android ERICA

So my question is: did I miss something, like some specific keywords? Or is there actually a gap in the literature with regard to the design of dialogue systems for interviews and questionnaire (or surveys) administration? I'm interested in any link or hint from anyone working on similar applicants, thank you in advance!

Thank you for the link! I did found already lot of papers about using prosodic and other auditory features for emotion and doubt detection but every new link is useful. Actually the question refers more about papers that tries to develop entire systems created with the purpose of performing interviews or questionnaire administration, I'll edit it to make this point more explicit!

[Disclaimer: I work for a company that provides a platform for developing conversational AI systems]
The platform used by the company I work for has a sentiment analysis component, so you can recognise if the user input expresses certain emotions. The dialogues are encoded in 'flows', which are graphs with an initial trigger consisting of output nodes and transitions that correspond to user input. In order to react to emotions expressed by the user you'd have a trigger that does not react to the user input text, but instead to a setting of the sentiment flags.
For example, in a call centre bot, if the user repeatedly uses angry sentiment in their utterances, the bot could detect this and trigger a flow that says "It seems you are not happy with my responses -- please wait while I transfer you to a human agent".
This all depends on the mechanics of organising the dialogues. If you have a ML model, this will probably not be as easy as if you have a manually designed graph structure (which gives you more control over such matters).
Being a commercial system, there are obviously no academic papers on this. But as it's so easy to do within the platform, it doesn't seem too much of a problem in the first place.

So, I guess that one of the keywords/terms that the OP was missing is "sentiment analysis".

Thanks for the answer! @nbro Actually I used the reference to sentiment analysis and auditory feature analysis just to show that these systems can be improved a lot compared to simple hand coded ones. Oliver: May I ask you if you work in the development of systems in which the robot has the lead of the conversation? That is actually the main key point in interviews from my perspective. Anyway you already gave me an hint: it could be that these type of dialogue systems are mostly researched in companies rather than academia, that might explain the rather low number of papers.

@EdoardoGuerriero Yes, you can design the dialogue so that the bot takes the lead; in fact that is more likely (after the initial user utterance) as the user is typically asking the bot to do something, and the bot then requests required information etc. I'm not developing the system myself, though, but design dialogues.

@OliverMason Thanks for the clarification! I'll wait.a couple of days to see if some else will answer with some reference otherwise I'll except the answer


text:
Can a variational auto-encoder learn images composed of random noise at each pixel (each drawn from the same distribution)?
Can a variational auto-encoder (VAE) learn images whose pixels have been generated from a Gaussian distribution (e.g. $N(0, 1)$), i.e. each pixel is a sample from $N(0, 1)$?
My gut feeling says no, because the VAE adds additional noise $\epsilon$ to the original image in the latent space, and if all images and pixels are random from the same distribution it would be impossible do decode/reconstruct into the particular input image. However, VAEs are a bit of a mystery to me internally. Any help would be appreciated!

Hi and welcome to AI SE! The $\epsilon$ that you're talking about probably refers to the sample drawn from $N(0,1)$ during the forward pass of the encoder. This is the so-called "re-parametrization trick", i.e. you do not sample directly from the latent space (i.e. you do not sample directly $z$, i.e. a latent representation of the image), but you sample $\epsilon$, then you combine it with the mean and variance of the distribution of the latent space to produce $z$. Why? Because this apparently makes training more stable (i.e. it's a variance reduction technique).

@nbro I see, thanks for the explanation! So in this case, would it be that the VAE can actually learn these images composed of random pixels?

It's a bit ambiguous what you mean by "images composed of random noise at each pixel (each drawn from, say N(0, 1))"? Do you mean that 1. you add a sample from N(0, 1) to each pixel of the original image (i.e. you add Gaussian noise to some image, e.g. a face of a person), or 2. every pixel is drawn from $N(0, 1)$ (i.e. you generate the images really by drawing a sample from $N(0, 1)$)? I believe it's option two. Anyway, by "random", one usually means draw from the uniform distribution.

Anyway, I don't think you can learn images whose pixels have been generated from a uniform distribution. In general, in those cases, there will be no pattern to learn (i.e. anything to compress). Have a look at information entropy. However, there are denoising auto-encoders. These can reconstruct the original inputs (i.e. not the noisy ones), but the noisy inputs are "just" noisy  (i.e. there are still patterns in those noisy images), so they are not uniformly randomly generated.

@nbro yes, sorry, it is 2, every pixel is random Gaussian noise. From what you're saying, it can't learn a pattern because there is none? What if we were to use a latent space of size equal to the input space size, would it be able to learn then?

I was talking about the case where pixels are generated from a uniform distribution, not from a normal distribution. Anyway, note that N(0, 1) doesn't restrict the values of the pixels to be in the range [0, 1].

Why did you delete this post and the other? I can delete it again, but I just wanted to understand why.

@nbro Question wasn't well-posed and the other one was absolutely all over the place as you pointed out, as a result I guess it got overlooked by people who may have been able to answer and didn't get much interaction at all. I think I'm going to get my ideas sorted, and post a more precise and thought-out question tomorrow that should be easier to answer and understand for people. I'll see you then!

This question is actually interesting, but feel free to delete it again if you want. Your other post was also interesting, although it needed clarification initially and you were asking too many things. Try to ask more specific questions next time and ask only one question per post. If you have multiple questions, ask each of them in its separate post.  Also, note that you may not receive an answer, but it doesn't mean your question is not interesting. Our community is still small, but, hopefully, we will grow and provide more help.

@nbro sounds good, let's keep this one then.

VAE's try and model the distribution of your data. So it's not going to learn " images composed of random noise at each pixel" per se (though, if overfitting, it could remember them). But it would be very capable of learning the simple noise distribution from which you sampled your random pixels

The question is not well-posed, so you should have waited for clarification before answering.

I think the question is well posed, you seemed not to have understood it the way I did, is all :) the user seems to wonder if there's something to learn from noise and I believe my answer answered just that

Yes, I admit that I missed one part. But it's still not clear whether the user wants to know about 1. images whose pixels have been generated from $N(0, 1)$ or 2. images to which we have added Gaussian noise. These are two different things! In one case, we are just corrupting the image, in the other case, we are really generating "Gaussian images".

I really understand the question as being about 1 => "composed of random noise at each pixel" ; but only they can actually answer that, true

Indeed, I meant 1), images are generated where each pixel is Gaussian noise, before anything to do with the VAE. Sorry @nbro for not being clear!

@ted if it were to learn the simple noise (standard normal) distribution, would it still be able to decompress/decode from the latent space accurately to the original image? Because of the introduction of random noise, reparameterization trick, wouldn't the latent space effectively look like a completely different one of those images (with pixels still from the standard normal)?

I think you're confused by epsilon, it's a sampling procedure not a noise added to the latent representation or anything, it's very different from the generating process of your data

@ted what if the latent space were the same size as the input, would it learn the distribution better than if it had to compress under these circumstances?

@ted Well, the pixels of the "random images" are also sampled, so I understand the confusion of the OP.


text:
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
I am reading the article How Transformers Work where the author writes

Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to process word by word. Not only that but there is no model of long and short-range dependencies.

Why exactly does the transformer do better than RNN and LSTM in long-range context dependencies?

I think it's incorrect to say that LSTMs cannot capture long-range dependencies. Well, it depends on what you mean by "long-range". They certainly can capture certain long-range dependencies. Also, when the author of that article says "there is no model of long and short-range dependencies.", this probably answers the question. The transformer probably has a "model" (whatever the author means by that) that models long-range dependencies explicitly (whatever "explicitly" means). Honestly, I am currently not familiar with the details of the transformer, so I cannot provide a formal answer now.

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them.
Transformers
Transformers (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order to allow parallel computation (to reduce training time) and also to reduce drops in performance due to long dependencies. The main characteristics are:

Non sequential: sentences are processed as a whole rather than word by word.
Self Attention: this is the newly introduced 'unit' used to compute similarity scores between words in a sentence.
Positional embeddings: another innovation introduced to replace recurrence. The idea is to use fixed or learned weights which encode information related to a specific position of a token in a sentence.

The first point is the main reason why transformer do not suffer from long dependency issues. The original transformers do not rely on past hidden states to capture dependencies with previous words. They instead process a sentence as a whole. That is why there is no risk to lose (or "forget") past information. Moreover, multi-head attention and positional embeddings both provide information about the relationship between different words.
RNN / LSTM
Recurrent neural networks and Long-short term memory models, for what concerns this question, are almost identical in their core properties:

Sequential processing: sentences must be processed word by word.
Past information retained through past hidden states: sequence to sequence models follow the Markov property: each state is assumed to be dependent only on the previously seen state.

The first property is the reason why RNN and LSTM can't be trained in parallel. In order to encode the second word in a sentence I need the previously computed hidden states of the first word, therefore I need to compute that first. The second property is a bit more subtle, but not hard to grasp conceptually. Information in RNN and LSTM are retained thanks to previously computed hidden states. The point is that the encoding of a specific word is retained only for the next time step, which means that the encoding of a word strongly affects only the representation of the next word, so its influence is quickly lost after a few time steps. LSTM (and also GruRNN) can boost a bit the dependency range they can learn thanks to a deeper processing of the hidden states through specific units (which comes with an increased number of parameters to train) but nevertheless the problem is inherently related to recursion. Another way in which people mitigated this problem is to use bi-directional models. These encode the same sentence from the start to end, and from the end to the start, allowing words at the end of a sentence to have stronger influence in the creation of the hidden representation. However, this is just a workaround rather than a real solution for very long dependencies.
CNN
Also convolutional neural networks are widely used in nlp since they are quite fast to train and effective with short texts. The way they tackle dependencies is by applying different kernels to the same sentence, and indeed since their first application to text (Convolutional Neural Networks for Sentence Classification) they were implement as multichannel CNN. Why do different kernels allow to learn dependencies? Because a kernel of size 2 for example would learn relationships between pairs of words, a kernel of size 3 would capture relationships between triplets of words and so on. The evident problem here is that the number of different kernels required to capture dependencies among all possible combinations of words in a sentence would be enormous and unpractical because of the exponential growing number of combinations when increasing the maximum length size of input sentences.
To summarize, Transformers are better than all the other architectures because they totally avoid recursion, by processing sentences as a whole and by learning relationships between words thanks to multi-head attention mechanisms and positional embeddings. Nevertheless, it must be pointed out that also transformers can capture only dependencies within the fixed input size used to train them, i.e. if I use as a maximum sentence size 50, the model will not be able to capture dependencies between the first word of a sentence and words that occur more than 50 words later, like in another paragraph. New transformers like Transformer-XL tries to overcome exactly this issue, by kinda re-introducing recursion by storing hidden states of already encoded sentences to leverage them in the subsequent encoding of the next sentences.

It's not clear from this answer what you mean when you say that "transformers process sentences as a whole". Do you mean that first you create word embeddings that represent whole sentences rather than words? I think that you should have mentioned the encoder-decoder architecture with LSTMs, because it seems that this "processing sentences as a whole" already existed in other machine translation architectures with LSTMs (like neural machine translation and stuff like that), so that would not be a novel thing.

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically no effect in the output at the current step.
LSTMs/GRUs mainly try to solve this problem, by including a separate memory (cell) and/or extra gates to learn when to let go of past/current information. Check these series of lectures for more in-depth discussion. Also check the interactive parts of this article for some intuitive understanding of dependency on past elements.
Now, given all this, information from past steps still goes through a sequence of computations and we're relying on these new gate/memory mechanisms to pass information from old steps to the current one.
One major advantage of the transformer architecture, is that at each step we have direct access to all the other steps (self-attention), which practically leaves no room for information loss, as far as message passing is concerned. On top of that, we can look at both future and past elements at the same time, which also brings the benefit of bidirectional RNNs, without the 2x computation needed. And of course, all this happens in parallel (non-recurrent), which makes both training/inference much faster.
The self-attention with every other token in the input means that the processing will be in the order of $\mathcal{O}(N^2)$ (glossing over details), which means that it's going to be costly to apply transformers on long sequences, compared to RNNs. That's probably one area that RNNs still have an advantage over transformers.

Why can you look at both the future and past elements at the same time without 2x computation?

A bidrectional RNN has a forward and a backward RNN, which means we process the whole sequence 2 times. It's basically a hack so that we have some way to look at the future using the backward RNN. In transformers, this is done in "one" operation, which is just calculating the result of softmax(QK) x V

Any benefit of Positional Encoding? Does it make possible Self Attention for longer sentence?

table is on page 6 from arxiv:1706.03762 Attention Is All You Need
It's because of the path length. If you have a sequence of length n. Then a transformer will have access to each element with O(1) sequential operations where a recurrent neural network will need at most O(n) sequential operations to access an element.
Very long sequences gives you problem with exploding and vanishing gradients because of the chain rule in backprop.  Transformers don't have this problem as the distance to each element in the sequence is always O(1) sequential operations away. (as it selects the right element instead of trying to "remember" it)
Yet there is a paper to handle longer input using RNN structure.
(arxiv:2103.05487) UnICORNN: A recurrent model for learning very long time dependencies
  "The design of RNNs that can accurately handle sequential inputs with long-time dependencies is very challenging.
  This is largely on account of the exploding and vanishing
gradient problem (EVGP)."

Is the Sequential Operations for a layer or entire structure for Transformer? My guess is that Sequential Operations for LSTM is entire structure. By the way is the Complexity per Layer sounds like Space Complexity (not Time Complexity)

It's per layer. Yes, a lstm block is considered to be a single layer (as they tend to be stacked). What do you mean with entire structure, do you mean the unrolled architecture? They basically just say what the span is of the architecture, for lstm's you have to wait for the previous token to finish computing before computing the next one so you get a span of O(n), where self attention and convolution (during training) don't have to wait sequentially O(1) span.

For convolution I've to check but the time and space complexity of attention is O(n^2d) and for recurrent O(nd^2).

For the "entire structure", I meant whole network strtucture not one module (ref: https://colah.github.io/posts/2015-08-Understanding-LSTMs/). For the complexity, I still don't get clear picture. Thanks for additional explantion though.

The post you send shows for a single layer the unrolled structure, so repetition of the lstm's module along the sequence length. We look at the layer perspective in this case, so for the lstm that is a single unrolled layer. (this can be different from the whole network structure as you can also layer these unrolled lstm layers on-top of each-other)

(the stacking can be done by feeding the (Ht-n,..,Ht-1,Ht,Ht+1,...,Ht+n as the (entire) Xt sequence for the layer ontop. The O(n) time and memory complexity for the lstm comes from the unrolling over the entire sequence, so it needs to process each and every element in the sequence at-least once (and store it for the computational graph to do backprop). In self attention (from attention is all you need) you have O(n^2) memory and compute since you have every element in the sequence message pass to every other element in the sequence. O(n^2)

in causal (self) attention you just mask away the messages so the structure is causal, but the full n^2 matrix (from the dot product between the key and queries, what when normalized with the softmax will represent the amount of "communication" between each token (element in the sequence)) is still formed it's just masked with a triangular zero's matrix.

so the O(n) for lstm's comes from the fact they form a chain and pass messages along the chain sequentially and self attention forms a fully connected graph where at once all nodes messages pass to all other nodes giving a O(n^2). This is also why the path length is so short in attention as each node (item in the sequence) is at-most 1 edge away from every other node, where in lstm's (or other recurrent networks) it tends to be a chain so at-most you are n-1 edges away  from another node

I hope this sheds a bit more light on it. :)

First: RNN is one part of the Neural Network family for processing sequential data. The way in which RNN is able to store information from the past is to loop in its architecture, which automatically keeps information from the past stored.
Second: sltm / gru is a component of regulating the flow of information referred to as the gate and GRU has 2 gates, namely reset gate and gate update. If we want to make a decision to eat like the analogy above, resetting the gate on the GRU will determine how to combine new inputs with past information, and update the gate, will determine how much past information should be kept.
source : https://link.springer.com/article/10.1007/s00500-019-04281-z

Hi @Lutfizain, welcome to AI stack exchange! Please see the tour and look around to see how this site works. I think you aren't answer the question, OP is asking why transformer (kind of NN architecture) performs better than RNN/LSTM on that context, and you didn't mention it


text:
Why is creating an AI that can code a hard task?
For people who have experience in the field, why is creating AI that has the ability to write programs (that are syntactically correct and useful) a hard task?
What are the barriers/problems we have to solve before we can solve this problem? If you are in the camp that this isn't that hard, why hasn't it become mainstream?

AI has been applied to programming (check out TabNine, my favorite autocomplete engine) although not in as robust a fashion as you describe.
Programming requires a high level of abstract while AI is typically trained to solve a very specific task. Given thousands of examples of insert sort in Python I think a model could be trained (perhaps after autocomplete and syntax correction) figure it out. However at this point the field has not developed a more general intelligence that can apply the ideas of the algorithm to other problems.
Addition based on comments:
Big picture, training an algorithm to solve a general class of problems (say, web dev) requires a huge number of examples or an immense number of trials. Further, as the complexity of the problem grows the number of parameters necessary to build the model grows. Writing code is a very complex problem and would thus require a huge amount of data and a huge number of parameters making it totally infeasible with today's math and (because of how the math is solved) hardware.
Modern AI is has a very simple goal: find the model that solves a problem optimally. If we could quickly search every possible model this would be simple. Fields like machine (deep) learning and reinforcement learning are concerned with finding a good solution in a reasonable amount of time. At this point no such solution exists for a problem of such complexity.

thank you for the response and you have some good points. To add one more question, what is stopping our current techniques in "narrow AI" from achieving this goal? If given enough data, do you think this could be solved with narrow AI? Is this not a problem that can be solved with just "more data"? Is this generally unreachable with current techniques? I'd like to hear your thoughts and again thank you for the response.

As you say, TabNine isn't really an AI that codes, it's just an auto-completion system. So, in my opinion, it has little to do with the question, so it can be misleading. Anyway, it's not a big problem to mention AI applied in software development tasks.

I am not an expert on this specific topic, but I can say a few words. I will use the term "programming" to refer to software development (of any kind).

If you are in the camp that this isn't that hard, why hasn't it become mainstream?

It's definitely hard, otherwise, we would have already some useful artificial programmers.

Why is creating AI that can code a hard task?

Programming is actually a hard task because it often requires creativity and a deep understanding of the context(s), goal(s), programming languages, etc. In other words, it's a very complex task (even for humans), apart from the exceptions where you can copy and paste.
Programming can probably be considered an AI-complete problem, i.e. a problem that probably requires an AGI to be solved. In other words, if an AI was as capable as humans in terms of programming, then that probably means it is an AGI (but this is not guaranteed), i.e. programming is a task that probably requires general intelligence. This is why I say that programming is an AI-complete problem. However, note that being able to program is just a necessary (but not sufficient) ability that an AI needs to possess in order for it to be an AGI (although not all general intelligences, e.g. animals, may be able to develop software, but the definition of general intelligence is also fuzzy).
AFAIK, no AGI has yet been created, and I think we are still very far away from that goal. Currently, most AI systems are only able to tackle a specific problem (i.e. we only have narrow AIs, such as AlphaGo). You could say that programming is a very specific problem too, but this is misleading or wrong, because, unless you just want to develop very specific programs in a very limited context (and there are already machine learning models and approaches, such as neural programmer-interpreters and genetic programming respectively, that can do this to some extent; see the answers to this question for other examples), then you will need to know a lot about other contexts too. For example, consider the task of developing a program that can detect signs of cancer in images. To develop this program, the AI would need to have the knowledge of an AI engineer, doctor, etc.
Furthermore, programming often requires common-sense knowledge. For example, while reading the software specifications, the AI needs to interpret them in the way that they were originally meant to be interpreted. This also suggests that programming requires an AGI (or human-level AI) to be solved.
(Finally, to address a comment, note that writing a 4-line program is not equivalent to writing a 10-line program. Also, the length of the program often doesn't correspond to its difficulty or complexity, so that alone is not a good measure of the ability to program.)

What are the barriers/problems we have to solve before we can solve this problem?

I think that the answer to this question is also the answer to the question "How can we create an AGI?". However, to be more concrete, I think that, in order to be able to create an AI that is able to program as well as humans, we will need to be able to create an AI that is able to think about low- and high-level concepts, compose them and it will probably require common-sense knowledge (so knowledge representation). A typical supervised learning solution will not be enough to solve this task. See the paper Making AI Meaningful Again, which also suggests that ML-based solutions may not be enough to solve many tasks.


text:
What is "natural image domain"?
I see some papers use the term "natural image domain". I googled that but didn't find any explanation of it.
I guess I understand the normal meaning of "natural image", such as the image people take by phones. The images in ImageNet database are all natural images.
Is "natural image domain" a subfield of computer vision?

Hi and welcome to AI SE! Can you please link to the specific papers that use the term "natural image domain"?

An image of a natural domain refers to any kind of image that has some variance in its structure, so it does not always present the same peculiarities.
For example, in a digits classification task, you can train your network on the MNIST dataset, that is a simple black & white set of images that are always well defined: the digit is in darker pixels, and the background is always white, they just have different shapes.
For the same task but in the natural image domain, the digits may vary in color, noise, and even position inside of a single image. As you may imagine this does not apply to digits only, but it includes any image that was not specifically created or modified to be fed into a neural network.
So, to put it simply, a neural network trained for natural images is able to perform classification in any real-life situation or scenario, without the need to edit the image to adapt it to what the network was trained for (for example, changing the color or cropping so the object you want to recognize is exactly in the center).
Here a similar question that may fill some doubts you may have.

I think that this sentence "a neural network trained for natural images is able to perform classification in any real-life situation or scenario" is misleading. What you wanted to say is that you don't have to pre-process the inputs before feeding them into the network. AFAIK, no neural network is able to perform classification in ANY real-life situation or scenario.

Yes, i gave for granted that a neural network can only classify objects that it was trained for


text:
What are some resources for coding some artificial intelligence techniques in the context of games?
I know the most basic rudimentary theory on AI, and I want to delve into actual practical coding with AI and machine learning. I already know a decent bit of coding in C++ and I'm learning Python syntax now.
I think I want to start implementing artificial intelligence techniques for simple games (like snake or maybe chess, which isn't really a simple game, but I know a lot about it), and then move on to more complex methods and algorithms.
So, what are some resources (e.g. tutorials, guides, books, etc.) for coding some artificial intelligence techniques in the context of games?

I don't know if it's enough for an answer, but if you're not already aware of it, Gym is probably the best introductory tool out there to train intelligent agents for games.

One of the simplest games you could solve with an AI technique is tic-tac-toe, which is a very simple game. To solve it, you could use minimax or alpha-beta pruning (an extension of minimax), which are very basic but fundamental search techniques for two-player games, so minimax can be applied not only to tic-tac-toe but any two-player game.
The notes CS 161 Recitation Notes - Minimax with Alpha-Beta Pruning provide a decent overview (with a concrete example) of alpha-beta pruning, which may be a little bit confusing at the beginning, but it is a relatively simple search technique that you can grasp with a few reading iterations (but take this with a grain of salt because the time and effort that it takes to learn something strongly depends on your knowledge and experience). There are several tutorials online that show how to implement minimax and alpha-beta pruning for tic-tac-toe. I could list them all, but I think it's better you look for them and choose your favorite. Maybe have a look at this implementation.
Of course, when it comes to games, one cannot forget about reinforcement learning. I cannot currently list any good tutorials, but there are many resources online. You can even apply RL to tic-tac-toe, so this may be your next step. As someone mentioned in the comments, OpenAI's gym is an RL library for implementing RL agents, which can be used to solve many games, so you should definitely have a look at it. OpenAI's gym comes with a nice introduciton, especially if you are already familiar with RL, which, of course, you probably should before trying to use it. In principle, you can apply RL to many games, including chess, snake, etc.
There's also a book entitled Clever Algorithms: Nature-Inspired Programming Recipes by Jason Brownlee that describes numerous AI techniques, some of them could, in principle, be applied to games too. The book is concise and relatively clear. It also comes with the implementation (in Ruby, which is a language similar to Python) of all of the algorithms presented in it.
To conclude, although this site is not appropriate for recommendations, I recommend you first get familiar with fundamental search techniques, such as minimax and alpha-beta pruning, then you could start learning RL, which can be applied to many games. There are many resources online for both of them.


text:
How do we reach at the formula for UCB action-selection in multi-armed bandit problem?
I came across the formula for Upper Confidence Bound Action Selection (while studying multi-armed bandit problem), which looks like:
$$
A_t \dot{=} \operatorname{argmax}_a \left[ Q_t(a) + c \sqrt{ \frac{\ln t}{N_t(a)} } \right]
$$
Although, I understand what the second term in the summation actually means but I am not able to figure out how and from where the exact expression came from, what is the log doing there? What effect does $c$ have? And, why a square root?

Here is an intuitive description/explanation.
$c$ is there for a trade-off between exploration and exploitation. If $c=0$ then you only consider $Q_t(a)$ (no exploration). If $c \rightarrow \infty$ then you only consider exploration term.
$\frac{\ln t}{N_t(a)}$ is there to balance out exploration term. If you consider a simple case where you only have one action (then it wouldn't make sense to explore you could always pick that action but let's pretend there is) then as $t \rightarrow \infty$, because $\ln t$ has sublinear growth,
\begin{equation}
\frac{\ln t}{N_t} \rightarrow 0
\end{equation}
So, after you picked an action infinitely many times, the exploration term will completely diminish, i.e. you already know a lot about what that action does. If you picked a numerator that doesn't have sublinear growth, then as $t \rightarrow \infty$ exploration term would not diminish, so you would always have a chance to explore and exploration term can "overpower" action value term if $Q_t$ is very small, even after you picked an action infinitely many times, which is not desired.
A similar thing is with multiple actions, $\ln t$ will make sure the exploration term $\rightarrow 0$ if you picked it many times, but it's still better than constant term $K/N_t(a)$, where $K$ is some constant, because it can diminish too fast.
With $\ln t$ you will also not stop exploring completely if you haven't picked some action in a long time, because $\ln t$ will keep growing and $N_t(a)$ will remain the same, so their fraction will increase with time, which is useful in non-stationary environments.
The square root is also there probably to balance out the magnitude of exploration term.
You can also see this answer. It has couple of links to some papers for a more mathematical description.


text:
Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions?
Neural networks are commonly used for classification tasks, in fact from this post it seems like that's where they shine brightest.
However, when we want to classify using neural networks, we often have the output layer to take values in $[0,1]$; typically, by taking the last layer to be the sigmoid function $x \mapsto \frac{e^x}{e^x +1}$.
Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?

@DuttaA That question has some interesting answers. But note that this question is more focused and specific. The other question is about the general use of sigmoids for neural networks and logistic regression.

Why not polyfit?

As far as I know, the sigmoid is often used as the activation function of the output layer mainly because it is a convenient way of producing an output $p \in [0, 1]$, which can be interpreted as a probability, although that can be misleading or even wrong (if you interpret it as an uncertainty too).
You may require the output of the neural network to be a probability, for example, if you use a cross-entropy loss function, although you could in principle produce only $0$s or $1$s. The probability $p$ can then be used to decide the class (or label) of the input. For example, if $p > \alpha$, then you purposedly decide that the input belongs to class $1$, otherwise, it belongs to class $0$. The parameter $\alpha$ is called the classification (or decision) threshold. The choice of this threshold can actually depend on the problem and it is one of the reasons people use the AUC metric, i.e. to avoid choosing this classification threshold.

Can neural networks with a sigmoid as the activation function of the output layer approximate continuous functions? Is there an analogue to the universal approximation theorem for this case?

The most famous universal approximation theorem for neural networks assumes that the activation functions of the units of the only hidden layer are sigmoids, but it does not assume that the output of the network will be squashed to the range $[0, 1]$. To be more precise, the UAT (theorem 2 of Approximation by Superpositions of a Sigmoidal Function, 1989, by G. Cybenko) states

Let $\sigma$ be any continuous sigmoidal function. Then finite sums of the
$$G(x) = \sum_{j=1}^N \alpha_j \sigma (y_j^T x + \theta_j)$$
are dense in $C(I_n)$.
In other words, given any $f \in C(I_n)$ and $\epsilon > 0$, there is a sum, $G(x)$, of the above form, for which
$$|G(x) - f(x)| < \epsilon $$

Here, $f$ is the continuous function that you want to approximate, $G(x)$ is a linear combination of the outputs of $N$ (which should be arbitrarily big) units of the only hidden layer, $I_n$ denotes the $n$-dimensional unit cube, $[0, 1]^n$, $C(I_n)$ denotes the space of continuous functions on $I_n$, $x \in I_n$ (so the assumption is that the input to the neural network is an element of $[0, 1]^n$, i.e. a vector $x \in \mathbb{R}^n$, whose entries are between $0$ and $1$) and $y_j$ and $\theta_j$ are respectively the weights and bias of the $j$ unit. The assumption that $f$ is a real-valued function means that $f$ can take any value on $\mathbb{R}$ (i.e. $f: [0, 1]^n \rightarrow \mathbb{R}$). You should note that $G(x)$ is the output of the neural network, which is a combination (where the coefficients are $\alpha_j$) of the outputs of the units in the only hidden layer, so there's no restriction on the output of $G(x)$, unless you restrict $\alpha_i$ (but, in this theorem, there's no restriction on the values $\alpha_j$ can take).
Of course, if you restrict the output of the neural networks to the range $[0, 1]$, you cannot approximate all continuous functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (because not all of these functions will have the codomain $[0, 1]$)! However, the sigmoid has an inverse function, i.e. the logit, so you can reverse the output of such a neural network. So, in this sense (i.e. by reversing the output of the sigmoid), a neural network with a sigmoid as the activation function of the output layer can potentially approximate any continuous function too.
The UAT above only states the existence of $G(x)$ (i.e. it's an existence theorem). It doesn't tell you how you can find $G(x)$. So, if you use a sigmoid as the activation function of the output layer or not is a little bit orthogonal to the universality of neural networks.

Comments are not for extended discussion; this conversation has been moved to chat.

So basically, people infer it from the universal approximation theorem but there is no actual paper...

@AIM_BLB I don't know if there's a paper or not that formally shows that, if the output of the neural network is a sigmoid, then the NN is able to approximate a class X of functions, but if you have $\sigma(G(x))$, you can retrieve $G(x)$ by applying the inverse of $\sigma$, i.e. $\sigma^{-1}(\sigma(G(x))) = G(x)$. This is more what I meant.

@nbro have you ever come across a book or paper vagely stating this argument.  I've been looking but I cant find anything...

@AnnieTheKatsu Stating what exactly?

I came across this post and saw your above comment.  It seems like your argument is "common knowledge" but I cant find it written down anywhere...

@AnnieTheKatsu If you are talking about the comment above, the point of that comment is that you can reverse the output of a sigmoid, so, in this sense, a net with a sigmoid at the output still follows the UAT.

@AnnieTheKatsu Maybe it helps if you think about it in this other way. Suppose you have a net without a sigmoid at the output. Now, provided the conditions to achieve the universality (e.g. sufficiently number of neurons, etc.), then this network can approximate any continuous function. Now, you have this network that can approximate any continuous function, then you apply the sigmoid to it. This will squash the original output to $[0, 1]$, but you can reverse the output of the sigmoid again to obtain the original output.

I know, but the issue is I'm looking for a source I can site for this (to show traction in the paper Im writing)

@AnnieTheKatsu I think I recently came across a paper that goes into the details of networks with probabilities as output, but I don't remember which paper was that.

If you could find that it would be fantastic... It lookd to me that this is "folklore" which makes it hard to pin down...

@AnnieTheKatsu Actually, I think that the paper that I am referring to is actually the paper by Cybenko that I am citing in my answer, i.e. "Approximation by Superpositions of a Sigmoidal Function". Although I didn't read that part fully, if I recall correctly, it seemed to me that Cybenko goes into the direction of "networks that classify". Have a look at it. I may be wrong, but the part "We now demonstrate the implications of these results in the context of decision regions" could be useful. Let me know.

I looked, the closest is Theorem 3 but still it doesnt use a soft-max/logistic output layer only appeals to Lusin's theorem...

@AnnieTheKatsu I would need to think about it and read that part more carefully. But I think we are not asking the right question! What do we want to show? Again, note that I am not saying in my answer that a neural network with a sigmoid as output can approximate any continuous function. In fact, I am more saying the opposite. It cannot because not all functions have the range $[0, 1]$. But, as I said, this range can be converted to the reals and vice-versa.

Maybe you can extend the UAT to this case by assuming that both $G(x)$ and $f(x)$ can be first converted to a sigmoid, and then maybe you can show that the UAT still applies with this change of variable.


text:
Do all expert trajectories have the same starting state in apprenticeship learning?
In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\, ...\}_{i=1}^m$. However, they also mentioned that $s_0 $ is drawn from distribution D. Do all expert trajectories then have to have the same starting state? Why is it not possible to compute the feature expectation based on a single trajectory?

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because many of the code implementations on github focus on trajectories starting from the same state.
Hope this helps everyone !

Maybe you should say where/how you discovered that $s_0$ is drawn from $D$. Does the paper you mention in your question state this?

Yes the paper mentioned above states this. The confusion came when I was reading a blog post about IRL. I might have misunderstood when the blog said that "The Starting position(state) of the bot is fixed, as according to the IRL algorithm it is necessary that the starting state is same for all the iterations.", from https://jangirrishabh.github.io/2016/07/09/virtual-car-IRL/ leading to some doubts that I had


text:
What are "proxy data sets" in machine learning?
The paper Assessment of Deep Generative Models for High-Resolution Synthetic Retinal Image Generation of Age-Related Macular Degeneration uses the term "proxy data sets" in this way

To develop DL techniques for synthesizing high-resolution realistic fundus images serving as proxy data sets for use by retinal specialists and DL machines.

I googled that term, but didn't find a definition of "proxy data sets". What are "proxy data sets" in machine learning?
The paper Analysis of Manufacturing Process Sequences, Using Machine Learning on Intermediate Product States (as Process Proxy Data) mentions a similar term

The advantage of the product state-based view is the focus on the product itself to structure the information and data involved throughout the process. Using the intermediate product states as proxy data for this purpose

Does "proxy data" mean the same thing as "proxy data sets" does?

In computer science, if you say "A is a proxy for B", then it means that "A replaces B" (temporarily or not), or that "A is used as an intermediary for B".
The term "proxy" usually refers to a server, i.e. there are the so-called proxy servers, which intuitively do the same thing (i.e. they are used as intermediaries). The following picture is worth more than 1000 words.

Let's go back to your context. In the following paragraph

To develop DL techniques for synthesizing high-resolution realistic fundus images serving as proxy data sets for use by retinal specialists and DL machines.

The deep learning techniques will be used to generate data that will be used as a replacement for real data for use by retinal specialists.
In your second paragraph

The advantage of the product state-based view is the focus on the product itself to structure the information and data involved throughout the process. Using the intermediate product states as proxy data for this purpose

The word "intermediate" is even used, so there should not be any need for further explanation, although I admit that I have no idea of what "product states" are in this context, but they will be used as "proxy data" for other data.

Does "proxy data" mean the same thing as "proxy data sets" does, if yes, I will go through that paper.

I guess so, but bear in mind that "data" is more general than "data set".
I have found other slightly different uses of the expression "proxy data", not in the context of computer science, but in the context of paleoclimatology. For example, have a look at this web article What Are Proxy Data?.
Just for completeness, here are some dictionary definitions of proxy.

authority given to a person to act for someone else, such as by voting for them in an election
a person who you choose to do something officially for you
a situation, process, or activity to which another situation, etc. is compared


text:
Why is it called back-propagation?
While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (say $w$) from any of the hidden layers, we're just writing the error function from the final outputs in terms of the inputs and hidden layer weights and then canceling all the terms without $w$ in it as differentiating those terms with respect to $w$ would give zero.
Where is the back-propagation of error while doing this? This way, I can find the partial derivatives of the first hidden layer first and then go towards the other ones if I wanted to. Is there some other method of going about it so that the Back Propagation concept comes into play? Also, I'm looking for a general method/algorithm, not just for 1-2 hidden layers.
I'm fairly new to this and I'm just following what's being taught in class. Nothing I found on the internet seems to have proper notation so I can't understand what they're saying.

i guess it's nothing special, just backward calculation from output loss gradient, to get loss gradients wrt every weight

Have a look at the following article Principles of training multi-layer neural network using backpropagation. It was very useful to me.
You can also see here an example of backpropagation in Matlab. It effectively solves the XOR problem. You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer your question?

I don't see how the method in the link you shared would work if there are activation functions on each node, it seems to give a different result to the one I referred to in the question. It makes sense if there are no activation functions though. Can you explain how the error gets propagated backward the same way the input is propagated forward? Thanks

You need to find the derivative of the function e.g $\text{sigmoid}'(e) = s(e)*(1-s(e))$

Why is it called back-propagation?

I don't think there is anything special here!
It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).
It's also called BP because it is just the application of the chain rule. Why is this interesting?
Let me answer this question with an example. Consider the function $y=e^{\sin(x^{2})}$. This is a composite function, i.e. a function composed of multiple simpler functions, which, in this case, are $e^x$, $\sin(x)$, $x^2$ and $x$. To compute the derivative of $y$ with respect to $x$, let's define the following variables
\begin{align}
y &= f(u) = e^u,\\
u &= g(v) = \sin v = \sin(x^2),\\
v &= h(x) = x^2
\end{align}
The derivative of $y$ with respect to the variable $x$ is (according to the chain rule)
$$
\underset{\color{red}{\LARGE \rightarrow}}{
\frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv}  \color{green}{\cdot}
\frac{dv}{dx}}
$$
If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $y$ to the function $v$). This is the same thing with BP!
Why is it called "chain rule"? Because you are chaining different partial derivatives. More specifically, you are multiplying them.
BP is also known as the reverse mode of automatic differentiation. Why? The automatic differentiation should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this automatically, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $e^x$) to the inner function (which, in the example above, is $x$). The Wikipedia article related to automatic differentiation provides more details.

What exactly are you back-propagating?

The partial derivative of the loss function $\mathcal{L}$ with respect to a parameter $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$, intuitively, represents the "contribution" of the parameter $w_i$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $w_i$ as follows
$$
w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i}
$$
where $\frac{\partial \mathcal{L}}{\partial w_i}$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.
This gradient descent step will hopefully make your network produce a smaller error next time.
The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, but he didn't reference neural networks. This Jürgen Schmidhuber's article goes into the details of the history of BP.

So you're saying the only reason it's called back propagation is because of the chain rule? I was under the impression that there is some kind of propagation of error from the output to the input, like you calculate the error on the final hidden layer, then use that to find the error in the layer before that and so on, kind of like how you propagate the input to hidden layer to output. Thanks for the answer.

@Skawang Yes, you can view the chain rule also in that way, but that's still the chain rule, I just didn't explicitly show the "error" and how it is part of the equations, but you can find this online.

@nbro, when using chain rule for BP in deep learning, do u agree that we can start computing the gradient (i.e. the product of some firsts layer jacobians) even if the forward pass is not terminated ? I mean one can imagine an algorithm computing both forward and backward at almost the same time ?

@RémyHosseinkhanBoucher You can compute the general formulas to compute the partial derivatives
without doing the forward pass (as I did in my answer, for example), but you need the specific values obtained during the forward pass to compute the actual values of the partial derivatives, otherwise, what would be the purpose of the forward pass?

So, no, you need the forward pass to compute the actual gradient. You can do this exercise with a simple neural network. Maybe a 1-hidden layer net. Compute the formulas of the backprop for it. Then you will realize that to compute the actual values you need the values obtained during the forward pass.


text:
Why do we need both the validation set and test set?
I know that this has been asked a hundred times before, however, I was not able to find a question (and an answer) which actually answered what I wanted to know, respectively, which explained it in a way I was able to understand. So, I'm trying to rephrase the question…
When working with neural networks, you typically split your data set into three parts:

Training set
Validation set
Test set

I understand that you use the training set for, well, train the network, and that you use the test set to verify how well it has learned: By measuring how well the network performs on the test set, you know what to expect when actually using it later on. So far, so good.
Now, a model has hyper parameters, which – besides the weights – need to be tuned. If you change these, of course, you get different results. This is where in all explanations the validation set comes into play:

Train using the training set
Validate how well the model performs using the validation set
Repeat this for a number of variants which differ in their hyperparameters (or do it in parallel, right from the start)
Finally, select one and verify its performance using the test set

Now, my question is: Why would I need steps 2 and 3? I could as well train multiple version of my model in parallel, and then run all of them against the test set, to see which performs best, and then use this one.
So, in other words: Why would I use the validation set for comparing the model variants, if I could directly use the test set to do so? I mean, I need to train multiple versions either way. What is the benefit of doing it like this?
Probably, there is some meaning to it, and probably I got something wrong, but I can't figure out what. Any hints?

The difference between the validation and test set in my opinion should be explained in this way:

the validation set is meant to be used multiple times.
the test set is meant to be used only once.

I think that the misunderstanding here arise because machine learning is mostly taught focusing only on a specific part of a large pipeline, which is the model training. In every tutorial standard datasets are used so that you don't have to worry about data collection, data labelling (it's really sad to see that lot of people have not a clue about what the inter annotator agreement is), data pre-processing and especially, all the part about the real application of the model is almost never mentioned.
The importance of having a set of instances that you can use for fine tuning (validation) and a set of instances that your model never encountered neither in training nor during fine tuning (test) becomes particularly clear if you focus on the subsequent deployment of the model you trained. No one expect a model to have the same performance scores in training and when applied to some unknown data. And the crucial point is that the performance of a model on the validation set are not representative either of the behaviour of a model with unknown data, because the same validation data have been used to fine tuned the model! So here's why having a set of data completely new to the model is important, because it gives you a much more unbiased view about the model performance on a real use case scenario.

Comments are not for extended discussion; this conversation has been moved to chat.

Simply stated, you use your validation set to regularize your model for unseen data.
Test data is completely unseen data, on which you evaluate your model.
Various validation strategies are used to improve your model to perform for unseen data. So strategies like k-fold cross-validation are used.
Also, the validation set helps you in tuning your hyperparameters such as learning rate, batch size, hidden units, number of layers, etc.
Train, Validation, Test sets help you in identifying whether you are underfitting or overfitting.
E.g. If human error at a task is 1%, train error is 8%, validation error is 10%, test set error is 12 % then,
Difference between,

Human level and training set error tells you about "Avoidable Bias"
Training set error and Validation set error tells you about "Variance and data mismatch"
Validation set error and Test error tells you about "degree of overfitting" with the validation set.

Based on these metrics, you can apply appropriate strategies for better performance on validation or test sets.


text:
Should I always start from the same start state in reinforcement learning?
In an episodic training of an RL agent, should I always start from the same initial state or I can start from several valid initial states?
For example, in a gym environment, should my env.reset() function always resets me to the same start state or it can start from different states at each training episode?

It depends on the task the agent is trying to learn and of course on the environment constrains.
In an Atari game agents have a pre-fixed starting point because that's part of the games rules, so I would say that this is enough of a justification to make each simulation start from that starting point. Moreover, you have to pay attention to the kind of reward function you're using, for example (a really dumb one just to give a grasp of the concept) if you're rewarding the agent depending on how close it gets to the ending point rather than how far it goes from the starting point, the agent might end up having huge rewards only because it respawned close to end point, which would be an artefact and not a fair reward for a good action choice.
Other situations in which it does not really make sense to select a random starting point might be dialogue systems, in which you know that a conversation start with greetings, therefore it would not be logical to make an agent starting with a random question or sentence (the space in these case would be made of different dialogue acts).
There are anyway situations in which the environment allows a random selection of the start location of the agent. In this paper for example, an agent was trained to escape from a maze, and both, the exit point and the initial location of the agent where randomly selected at each iteration. This was partially due to research reasons, the authors were analysing the possibility to train policies more complex than a simple 'space memorisation' one but anyway, conceptually there's nothing wrong in this situation in selecting a random starting point. Other tasks could be following or escaping from a specific object, again, as long as the reward function is properly designed, even with a random starting location at the end the agent would anyway learn to move faster when close to the target (toward or in the opposite direction depending on the task). Actually, in this situation I think that a random initial location would have potential benefits, like avoiding the agents to learn biases due to external biases of the environment (like parts of space with less random obstacles than others).

It is your choice.
This can even be different between training and target system. The approach called "exploring starts" chooses a random start state (and action if you are assessing a deterministic policy for action values).
In general, if you don't have a reason to pick exploring starts, you should aim for your env.reset() function to put the environment into a state drawn from the distribution of start states that you expect the agent to encounter in production. This will help if you are using function approximation - it will mean that the distribution of training data will better match the distribution seen in production, and approximators can be sensitive to that.
In some cases, such as policy-gradient methods, your cost function will be defined in terms of expected return given a start state distribution, so at least during assessment you will want a env.reset() function that matches that target start distribution.
It is still OK to have different distributions for start states for training and assessment, and might be worth investigating as a hyperparameter for training. For instance if the training start state distribution can pick states that are hard to get to randomly otherwise, it may help the agent to find any of those states that are high value.


text:
Why is there an expectation sign in the Bellman equation?
In chapter 3.5 of Sutton's book, the value function is defined as:

Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy $\pi$, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to this Wikipedia definition of the expected values?

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-

The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a_t$ is distributed as-
$$a_t \sim \pi(s_t)$$
Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a_t$, in general, only provides us with a distribution over the possible next states of the agent. That is,
$$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$
Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.

As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in Monte-Carlo estimation of value functions.
EDIT:
As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.

While your explanation is correct, your conclusion is not. The expectation does indeed correspond to an averaging over a collection of trajectories since you need to consider all the possible combinations of states and actions that lead to the return $G_t$ and weight them by how probable the combination is under the policy $\pi$ and the transition function $T$.

@DiegoGomez Have added an edit, thanks for pointing out.

Thanks. So in the equation https://imgur.com/4g6fjIU the flashed symbols are the ones that denote the probabilities of the expectation formula?

If so, why we only have $E_{\pi}$, shouldn't it be $E_{\pi,p}$? Because I suppose the $p$ of environment transitions is not dependent on policy $\pi$, is it?

@Jack in the linked image, summation symbols have been highlighted. The inner summation is over the possible next-state, reward tuple and the outer one is over a stochastic policy, so yes all of them together are capturing the expectation if that is what you meant.

@Jack yes I agree, perhaps to be rigorous, it should be made explicit that the expectation is over both $\pi$ and the transition distribution $p$.

In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant is the constant itself), so the expectation operator also applies to the deterministic cases. Therefore, the expectation is a general way of expressing the value of a state in all possible cases (both when trajectories are fixed or not).


text:
What is the relationship between the reward function and the value function?
To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses to 'generate' those rewards for it to use in the calculation of how 'good' it is to be in the state?

I think it is pedagogically useful to distinguish between the theory (equations) and the practice (algorithms).
If you're talking about the definition of the value function (the theory)
\begin{align}
v_{\pi}(s)
& \dot{=} \mathbb{E}_{\pi} \left[ G_t \mid S_t = s \right]\\
&= \mathbb{E}_{\pi} \left[ \sum_{k=0}^\infty \gamma^k R_{t+k+1} \bigl\vert S_t = s \right]\\
\end{align}
for all $s \in \mathcal{S}$, where $\dot{=}$ means "is defined as" and $\mathcal{S}$ is the state space, then the value function can be defined in terms of the reward, as it can be clearly seen above. (Note that $R_{t+k+1}$, $G$ and $S_t$ are random variables, and, in fact, expectations are taken with respect to random variables).
The definition above can actually be expanded to be a Bellman equation (i.e. a recursive equation) defined in terms of the reward function $R(s, a)$ of the underlying MDP. However, often, rather than the notation $R(s, a)$, you will see $p(s', r \mid s, a)$ (which represents the combination of the transition probability function and the reward function). Consequently, the value is a function of the reward.
If you're estimating a value function (the practice), e.g. using Q-learning, you don't necessarily use the reward function of the Markov decision process. You can estimate the value function by just observing the rewards that you receive while exploring the environment, without really knowing the reward function. But, by exploring the environment, you can actually estimate the reward function. For example, if every time you're in state $s$ you take action $a$ and you receive reward $r$, then you already know something about the actual underlying reward function. If you explore enough the MDP, you could potentially learn the reward function too (unless it keeps on changing, in that case, it may be more difficult to learn it).
To conclude, yes, value functions are certainly very related to reward functions and rewards, in ways that you immediately see from the equations that define the value functions.

Sutton and Barto give a great description in their book Reinforcement Learning:

Whereas the reward signal indicates what is good in an immediate sense, a value
function specifies what is good in the long run. Roughly speaking, the value of a state is
the total amount of reward an agent can expect to accumulate over the future, starting
from that state. Whereas rewards determine the immediate, intrinsic desirability of
environmental states, values indicate the long-term desirability of states after taking into
account the states that are likely to follow and the rewards available in those states. For
example, a state might always yield a low immediate reward but still have a high value
because it is regularly followed by other states that yield high rewards. Or the reverse
could be true. To make a human analogy, rewards are somewhat like pleasure (if high)
and pain (if low), whereas values correspond to a more refined and farsighted judgment
of how pleased or displeased we are that our environment is in a particular state.

Mathematically, the value function is the expected sum of discounted rewards from a given state or for a particular action in a given state.


text:
What does equation in the "related work" section of the GAN paper mean?
I was going through the paper on GAN by Ian Goodfellow. Under the related work section, there is an equation. I cannot decipher the equation. Can anyone help me understand the meaning of the equation?
$$\lim_{\sigma \to 0} \nabla_{\mathbf  x} \mathbb E_{\epsilon \sim \mathcal N(0, \sigma^2 \mathbf I)} f(\mathbf x+\epsilon) = \nabla_x f(\mathbf x)$$
Also, any guide to understanding mathematical notation for reading research paper is highly appreciated.

In full:

The limit, as standard deviation $\sigma$ tends towards zero, of the gradient with respect to vector $\mathbf{x}$, of the expectation - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of any function $f$ of $\mathbf{x}$ plus $\epsilon$ is equal to the gradient with respect to $x$ of the same function of $\mathbf{x}$.

If we break that down:
$$\lim\limits_{\sigma \rightarrow 0}$$

The limit, as standard deviation $\sigma$ tends towards zero of

$$\nabla_\mathbf{x}$$

the gradient with respect to vector $\mathbf{x}$ of

$$\mathbb{E}$$

the expectation ...

$$\mathbb{E}_{\epsilon \sim \mathcal{N}(0, \sigma^2\mathbf{I})}$$

[the expectation] - where perturbation $\epsilon$ follows the normal distribution with mean 0 and variance $\sigma^2$ times identity vector $[1,1,1,1...]$ * - of

$$f(\mathbf{x} +\epsilon)$$

any function $f$ of $x$ plus $\epsilon$

$$ = \nabla_x f(\mathbf{x})$$

is equal to the gradient with respect to $\mathbf{x}$ of the same function $f(\mathbf{x})$.

Basically it says that taking small perturbations of a vector input to a function and measuring the gradient at those varied points can be used to give you a valid estimate of the true gradient at the point you are making variations of.
In terms of understanding the equations, read introductory texts to the research area, and if like me your maths has been unused for many years before attempting this, expect to spend time and effort. Re-read the equations, memorise and write out the basic ones from the field, apply them to simple problems that might be presented in text books. Reading maths equations is not much different to reading music or reading another language - it takes concentration, practice, time and effort to become fluent enough to read an equation and comprehend it. The different research fields can be quite different too, some might be as similar enough to get by with what you know already, others may require learning all over again.

* I am not 100% certain of the interpretation of $\mathbf{I}$ as an identity vector - a matrix may be more appropriate, which depends on the form of $\mathcal{N}(\mu, \sigma^2)$ when handling vector distributions. A matrix form for the second argument would be more general and allow for covariance, although the use of $\mathbf{I}$ would then explicitly remove covariance and make each component of $\epsilon$ independent, which is required for this result.

@DuttaA: Actually I am not sure, it depends on the form of second argument $\mathcal{N}$, which I am assuming will just take a vector of variances for independent vector components here, but you are right it could also support some forms of covariance in general, and we'd want an identity matrix instead to make the variables independent. Do you have a reference to the common assumptions of that function?