text:
What is the difference between LSTM and RNN?
What is the difference between LSTM and RNN? I know that RNN is a layer used in neural networks, but what exactly is an LSTM? Is it also a layer with the same characteristics?

RNNs have recurrent connections and/or layers
You can describe a recurrent neural network (RNN) or a long short-term memory (LSTM), depending on the context, at different levels of abstraction. For example, you could say that an RNN is any neural network that contains one or more recurrent (or cyclic) connections. Or you could say that layer $l$ of neural network $N$ is a recurrent layer, given that it contains units (or neurons) with recurrent connections, but $N$ may not contain only recurrent layers (for example, it may also be composed of feedforward layers, i.e. layers with units that contain only feedforward connections).
In any case, a recurrent neural network is almost always described as a neural network (NN) and not as a layer (this should also be obvious from the name).
LSTM can refer to a unit, layer or neural network
On the other hand, depending on the context, the term "LSTM" alone can refer to an

LSTM unit (or neuron),
an LSTM layer (many LSTM units), or
an LSTM neural network (a neural network with LSTM units or layers).

People may also refer to neural networks with LSTM units as LSTMs (plural version of LSTM).
LSTMs are RNNs
An LSTM unit is a recurrent unit, that is, a unit (or neuron) that contains cyclic connections, so an LSTM neural network is a recurrent neural network (RNN).
LSTM units/neurons
The main difference between an LSTM unit and a standard RNN unit is that the LSTM unit is more sophisticated. More precisely, it is composed of the so-called gates that supposedly regulate better the flow of information through the unit.
Here's a typical representation (or diagram) of an LSTM (more precisely, an LSTM with a so-called peephole connection).

This can actually represent both an LSTM unit (and, in that case, the variables are scalars) or an LSTM layer (and, in that case, the variables are vectors or matrices).
You can see from this diagram that an LSTM unit (or layer) is composed of gates, denoted by

$i_t$ (the input gate: the gate that regulates the input into the unit/layer),
$o_t$ (the output gate: the gate that regulates the output from the unit)
$f_t$ (the forget gate: the gate that regulates what the cell should forget)

and recurrent connections (e.g. the connection from the cell into the forget gate and vice-versa).
It's also composed of a cell, which is the only thing that a neuron of a "vanilla" RNN contains.
To understand the details (i.e. the purpose of all these components, such as the gates), you could read the paper that originally proposed the LSTM by S. Hochreiter and J. Schmidhuber. However, there may be other more accessible and understandable papers, articles or video lessons on the topic, which you can find on the web.
LSTMs also have recurrent connections!
Given the presence of cyclic connections, any recurrent neural network (either an LSTM or not) may be represented as a graph that contains one or more cyclic connections. For example, the following diagram may represent both a standard/vanilla RNN or an LSTM neural network (or maybe a variant of it, e.g. the GRU).

When should you use RNNs and LSTMs?
RNNs are particularly suited for tasks that involve sequences (thanks to the recurrent connections). For example, they are often used for machine translation, where the sequences are sentences or words. In practice, an LSTM is often used, as opposed to a vanilla (or standard) RNN, because it is more computationally effective. In fact, the LSTM was introduced to solve a problem that standard RNNs suffer from, i.e. the vanishing gradient problem. (Now, for these tasks, there are also the transformers, but the question was not about them).


text:
In the Markov chain, how are the directions to each successive state defined?
I'm watching the David Silver series on YT which has raised a couple of questions:
In the Markov process (or chain), how are the directions to each successive state defined? For example, how are the arrow directions defined for the MP below? What's stopping our sample episodes from choosing A -> C -> D -> F?

Also, how is the probability transition matrix populated? From David's example, the probabilities seem to have already been set. For example:

It's not possible, as in the chain illustrated there are no transitions between A and C, C and D, and D and F. Only sequences where transitions exist are possible. The choice of transitions is arbitrary; it depends on what you want to model with it.

As DuttaA says in his comment, you can imagine that all nodes are linked with all other nodes, but those links have a transition probability of 0.0; so the probability of observing the sequence ACDF is actually 0.0 as well. In order to make the diagram more easy to understand, zero-probability transitions are generally not shown.

There is no prescribed method of populating the transition probabilities. You can define the probabilities manually, randomly, or however you like. You could derive them from observable training data.

Re updating: this depends on the application. If you are modelling a process with known probabilities, you would not update the probabilities; if you are modelling a dynamic process which changes over time, then this is something you might want to consider. However, from my own experience (HMMs in speech recognition), once they have been assigned, they are generally kept as they are.
A Markov chain is really a fairly basic model; it gets more complex with a Hidden Markov Model, where you would generally use a learning algorithm to assign the transition and emission probabilities.

Yes, DuttaA, thanks. So I guess the transitions are dictated in the Probability Transition Matrix? If so, then I presume the sequences are set as Oliver Mason says in point 2. So, if they’re set randomly, how are the probabilities updated to their ‘real’ probabilities (or does that not happen?).

@DuttaA Thanks for your comments; I have extended my answer.

Thanks for updating, Oliver. If, for example, the environment is a board game and its states are the various position the game pieces may be in, how would the PTM be initialised? How would it be (if it is?) updated?

It's probably best to put that in a new question; and give a concrete example of a game.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What are the variables used in a Gaussian radial basis kernel in the context of SVMs?
If I have the Gaussian kernel
$$
k(x, x') = \operatorname{exp}\left( -\| x - x' \|^2 / 2\sigma^2 \right)
$$
What is $x$ and $x'$ in the context of training an SVM?

$\mathbf{x} \in \mathbb{R}^p$ and $\mathbf{x}' \in \mathbb{R}^p$ are two inputs (or feature vectors).
In the context of classification with an SVM, you are given a dataset $D = \{(\mathbf{x}_i, y_i) \}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^p$ is an input (or point) and $y_i$ the corresponding label. The goal is to find a hyperplane that classifies the points $\mathbf{x}_i$. The hyperplane actually corresponds to a binary classifier that splits the plane into two, so the assumption is that there are two labels. However, these points $ \mathbf{x}_i$ may not be linearly separable in $\mathbb{R}^p$, i.e. there may not be a hyperplane (in 2d, i.e. when $p=2$, a hyperplane is a line) that separates them. The kernel trick, i.e. the use of kernels (such as the Gaussian radial basis), allows an SVM to perform non-linear classification by transforming the inputs to a space where they are linearly separable.

Just to solidify my understanding,
if I take the MNIST dataset for example and reduce it to just working with 2 classes (instead of the actual 10), with images of size 28x28 and say there are 500 examples in this example dataset. I would then have a 500x28x28 dataset with corresponding y labels of 500x2.
If I then wanted to apply this gaussian kernel to this dataset, what are the actual values I would plug in?

In your example, $\mathbf{x}$ (or  $\mathbf{x}'$) would be a MNIST image, which can be represented as a vector in $\mathbb{R}^{784}$ (where $784 = 28*28$). In your example, you would have $500$ examples of the form $\mathbf{x}$. The labels don't have to be a vector, they can simply be a scalar, so your labels could simply be a vector of the form $\mathbb{R}^{500}$. Regarding how you apply the kernel, this is more a detail of how the SVM works, but, to give you the intuition, essentially, you will be replacing the dot products in the SVM computations with $k(\mathbf{x}, \mathbf{x}')$.

so is x and x' two of the datapoints(image vectors) in my dataset? sorry but its just not clicking.

@FeedMeInformation Yes. (But the application of a kernel function $k$ is not just restricted to the data points (image vectors), but, given that it is a general function, it can be applied to any pair of vectors. You can ignore this, if you don't want to know the details of SVMs. But, if you read the Wikipedia entry on SVMs, you will get a better feeling of the whole picture (even though Wikipedia may not be the most reliable source ever)).

@FeedMeInformation In the SVM formulations, you will see dot products, which, in general, are often used to calculate a notion of "distance" between vectors. The kernel function (e.g. your Gaussian radial basis) has the same purpose (in this and other contexts), i.e. it's still a way of calculating distances between vectors, but you do it in a more "sophisticated way". So, you can think of a kernel function as a function that computes a notion of "distance".

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What is teacher forcing?
In the paper Neural Programmer-Interpreters, the authors use the teacher forcing technique, but what exactly is it?

Consider the task of sequence prediction, so you want to predict the next element of a sequence $e_t$ given the previous elements of this sequence $e_{t-1}, e_{t-2}, \dots, e_{1} = e_{t-1:1}$. Teacher forcing is about forcing the predictions to be based on correct histories (i.e. the correct sequence of past elements) rather than predicted history (which may not be correct). To be more concrete, let $\hat{e}_{i}$ denote the $i$th predicted element of the sequence and let $e_{i}$ be the corresponding ground-truth. Then, if you use teacher forcing, to predict $e_{t}$, rather than using $\hat{e}_{t-1:1}$, you would use $e_{t-1:1}$.
Recall that supervised learning can also be thought of as learning with a teacher. Hence the expression "teacher forcing", i.e. you force the predictions to be based on correct histories (the teacher's labels).
Of course, intuitively, teacher forcing should help to stabilize training, given that the predictions are not based on noisy or wrong histories.
See also the blog post What is Teacher Forcing for Recurrent Neural Networks? by Jason Brownlee.


text:
How can I train a RL agent to play board games successfully without human play?
How would you go about training an RL Tic Tac Toe (well, any board game, really) application to learn how to play successfully (win the game), without a human having to play against the RL?
Obviously, it would take a lot longer to train the AI if I have to sit and play "real" games against it. Is there a way for me to automate the training?
I guess creating "a human player" to train the AI who just selects random positions on the board won't help the AI to learn properly, as it won't be up against something that's not using a strategy to try to beat it.

The approach will vary depending on some features of the game:

How many players (two for tic tac toe and many classic games).
Whether it is a "perfect information" game (yes for chess and tic tac toe), or whether there is significant hidden information (true for most forms of poker)
Whether the game is zero sum. Any game with simple win/loss/draw results can be considered zero sum.
Branching complexity of the game. Tic tac toe has relatively low branching factor, at worst 9, and reduces by one each turn. The game of Go has branching factoraround 250.
Action complexity of the game. A card game like Magic the Gathering has huge complexity even if the number of possible actions at each turn is quite small.

The above traits and others make a large impact on the details of which algorithms and approaches will work best (or work at all). As that is very broad, I will not go into that, and I would suggest you ask separate questions about specific games if you take this further and try to implement some self-playing learning agents.
It is possible to outline a general approach that would work for many games:
1. Implement Game Rules
You need to have a programmable API for running the game, which allows for code representing a player (called an "agent") to receive current state of the game, to pass in a chosen action, and to return results of that action. The results should include whether any of the players has won or lost, and to update internal state of the game ready for next player.
2. Choose a Learning Agent Approach
There are several kinds of learning algorithms that are suitable for controlling agents and learning through experience. One popular choice would be Reinforcement Learning (RL), but also Genetic Algorithms (GA) can be used for example.
Key here is how different types of agent solve the issues of self-play:

GA does this very naturally. You create a population of agents, and have them play each other, selecting winners to create the next generation. The main issue with GA approaches is how well they scale with complexity - in general not as well as RL.
With RL you can have the current agent play best agent(s) you have created so far - which is most general approach. Or for some games you may be able to have it play both sides at once using the same predictive model - this works because predicting moves by the opposition is a significant part of game play.

2a. How self-play works in practice
Without going into lines of code, what you need for self-play is:

The game API for automation and scoring
One or more agents that can use the API
A system that takes the results of games between agents and feeds them back so that learning can occur:

In a GA, with tournament selection, this could simply be saving a reference - the id of the winner - into a list, until the list has grown large enough to be the parents for the next generation. Other selection methods are also possible, but the general approach is the same - the games are played to help populate the next generation of agents.
In RL, typically each game state is saved to memory along with next resulting state or the result win/draw/lose (converted into a reward value e.g. +1/0/-1). This memory is used to calculate and update estimates of future results from any given state (or in some variants used directly to decide best action to take from any given state). Over time, this link between current and next states provides a way for the agent to learn better early game moves that eventually lead to a win for each player.
An important "trick" in RL is to figure out a way for the model and update rules to reflect opposing goals of players. This is not a consideration in agents that solve optimal control in single agent environments. You either need to make the prediction algorithm predict future reward as seen from perspective of the current agent, or use a global scoring system and re-define one of the agents as trying to minimise the total reward - this latter approach works nicely for two player zero sum games, as both players can then direcly share the same estimator, just using it to different ends.

A lot of repetition in a loop. GA's unit of repetition is usually a generation - one complete assessment of all existing agents (although again there are variants). RL's unit of repetition is often smaller, individual episodes, and the learning part of the loop can be called on every single turn if desired. The basic iteration in all cases is:

Play anything from one move to multiple games, with automated agents taking all roles in the game and storing results.
Use results to update learned parameters for the agents.
Use the updated agents for the next stages of self-play learning.

3. Planning
A purely reactive learning agent can do well in simple games, but typically you also want the agent to look ahead to predict future results more directly. You can combine the outputs from a learning model like RL with a forward-planning approach to get the best of both worlds.
Forward search methods include minimax/negamax and Monte Carlo Tree Search. These can be combined with learning agents in multiple ways - for instance as well as in planning stages using the RL model, they can also be used to help train a RL model (this is how it is used in Alpha Zero).

One solution would be to simply play the AI against itself (which yields good results for the tic tac toe example, I've tried it), but a much more interesting approach is to train two networks at the same time and have them play against each other.
It's called a Generative adversarial network (GAN) and it's very promising. It is able to make realistic images and much more.
It's quite easy to implement both solutions so I'd recommend doing both and seeing for yourself since each problem is different.

Maybe you should explain how GANs can specifically help in this situation.

Thank you for your response. Do you have any non-NN examples of how I can train?


text:
Is there a possibility that there is no relationship between some inputs and outputs?
I'm doing machine learning projects. I took a look at many datasets I worked with, mostly there are already famous datasets that everyone uses.
Let's say I decided to make my own dataset. Is there a possibility that my data are so random so that no relationship exists between my inputs and outputs? This is interesting because if this is possible, then no machine learning model will achieve to find an inputs outputs relationship in the data and will fail to solve the regression or classification problem.
Moreover, is it mathematically possible that some values have absolutely no relationship between them? In other words, there is no function (linear or nonlinear) that can map those inputs to the outputs.
Now, I thought about this problem and concluded that, if there is a possibility for this, then it will likely happen in regression because maybe the target outputs are in the same range and the same features values can correspond to the same output values and that will confuse the machine learning model.
Have you ever encountered this or a similar issue?

Well, it depends on what do you mean under the "relationship". For any two set of inputs and outputs I may build a polynomial $f$ of many variables such that $outputs=f(inputs).$ Is it a relationship?

Of course, it's possible to define a problem where there is no relationship between input $x$ and output $y$. In general, if the mutual information between $x$ and $y$ is zero (i.e. $x$ and $y$ are statistically independent) then the best prediction you can do is independent of $x$. The task of machine learning is to learn a distribution $q(y|x)$ that is as close as possible to the real data generating distribution $p(y|x)$.
For example, looking at the common cross-entropy loss, we have
$$
\begin{align}
H(p,q) = -\mathbb{E}_{y,x \sim p}\left[\log q(y|x)\right] & = \mathbb{E}_{x\sim p}\left[\text{H}(p(y|x)) + \text{D}_{\text{KL}}(p(y|x)\|q(y|x))\right]  \\
& = \text{H}(p(y)) + \mathbb{E}_{x \sim p}\left[\text{D}_{\text{KL}}(p(y)\|q(y|x))\right],
\end{align}
$$
where we have used the fact that $p(y|x)=p(y)$ since $y$ and $x$ are independent. From this, we can see that the optimal predicted distribution $q(y|x)$ is equal to $p(y)$, and actually independent of $x$. Also, the best loss you can get is equal to the entropy of $y$.

Not sure if I can answer the question as a whole, but a pure random input/output pair doesn't quite have "no relationship" at all. At the very least, for any fixed training set input/output pair, you can do an if...then mapping to construct a 1-to-1 function, such that you can classify the training set with 100% accuracy (assuming no duplicates of input).
In any case, I assume you mean uniform random, because if you have something like gaussian random, you can still learn some latent structure from how the random numbers are generated.
But even if you assume uniform random, and your algorithm is only guessing, your algorithm is technically still operating optimally per the data generating distribution, which basically means its as optimal as it gets.
The only such case that I can imagine which would satisfy your question, would be if you had a separate training/validation set, where the only element of the training input/output is [1,1], but the validation set only has elements of [1,-1], or something along those lines.
From reading your comments, I suspect that your intention with the question was: "Can there be a relationship of data such that no method can learn it?". To the extent that the data-generating distribution exists, then by the universal approximation theorem of neural networks, then it is reasonable that you can at least partially learn it.
However it is important to note that the universal approximation theorem doesn't mean that such a data generating distribution can be learned by a neural net, it only means that you can get "non-zero as close as you want" to the data generating distribution. More explicitly: there is a setting of weights that gives you results as good as you want, but gradient descent doesn't necessarily learn it.

thanks for the answer, my goal is not to make an if. if I used machine learning model with a validation and test set the model would overfitt if i made if else all the time!! by relationship I mean a pattern that map such data to an output. precisely, is there a dataset where i can look and say no sorry there is no model here that can find relationship in this data, hence, machine learning can do nothing for me in this case

@basilisk strictly speaking, an if....then.. algorithm is as valid of a classification algorithm as any other, the main issue being that it is not differentiable, and thus you can't run things like gradient descent on it, and what i said applies for an overtrained model anyhow. >"is there a dataset [...] no model here that can find relationship in this data" i have already answered your question, but to spell it out for you: strictly speaking, likely no, unless your training/validation set pair was poorly picked, and you can always overfit your set if you don't have a training/validation pair.

a relationship between data doesnt mean memorizing the values with an if then statements! a relationship is a function that can be created to map input x to output y. Furthermore, of course you can use gradient descent in classification, you need to choose the right differentiable loss function for it (at least in a certrain range). Your answer does not answer my question. as you used in the first paragraph, using if then will give you 100% accuracy on training data but who validate on trainingdata?for capturing a pattern you need to test on test data, in your case the model giv0% on test data

"a relationship between data doesn't mean [...]" yes it does. both a "usual ML function" and an "if then statement" are functions that map from input space to output space. if you don't acknowledge this, then you lack understanding. your initial question asks for a theoretical result regarding a /fixed dataset/, and hence i was able to cheese an answer by suggesting you can always achieve your intended result by overfitting. now, your intention seems be about learning the underlying "data generating distribution", which is very different from how you initially posed the question

and in any case, in the last paragraph i already answered your question regarding separate validation/training sets

my question clearly shows that I'm asking from a machine learning aspects and not just mathematics. Please read the question again, I stated whether mL model would fail sometimes to find pattern in data because of randomness. you think I lack understanding, I would say who doesnt? . Moreover, I think everyone knows that data can be memorized with if then.. but I'm asking for a relationship, a pattern, a mapping function.. I don't know what to say more. also I didn't understand the last paragraph you wrote, can you clear more

...machine learning is literally math/stats......... you asked if something might happen in theory, so i answered your question "as in theory", which is why i gave you an answer this is "technically correct"

last paragraph tl;dr: if your validation/training set are "obviously unrelated" then you might argue that you cannot learn any relationship between them. but the term "obviously unrelated" is loaded with tricky epistemic (as in: philosophical) issues, which makes it non-trivial to unpack, which is why i gave the example as above.

you might be interested in this: https://stats.stackexchange.com/questions/320375/what-does-it-mean-for-the-training-data-to-be-generated-by-a-probability-distrib    if you've read that, then the "formal interpretation" of what i meant was that the training/validation set comes from a different "data-generating distribution". if you've read this far you might come to the suspicion that i am again being pedantic by providing an answer like this. but i'll add something to my answer that might address your original intent of the question.

actually i've came to realize that you've asked the same question here: https://datascience.stackexchange.com/questions/69083/is-there-a-possibility-that-there-is-no-relationship-between-some-inputs-and-out , which is a better place to ask that question anyhow. @Dave 's answer there pretty much superseeds mine and is as clear as it gets


text:
What are the differences between transfer learning and meta learning?
What are the differences between meta-learning and transfer learning?
I have read 2 articles on Quora and TowardDataScience.

Meta learning is a part of machine learning theory in which some
  algorithms are applied on meta data about the case to improve a
  machine learning process. The meta data includes properties about the
  algorithm used, learning task itself etc. Using the meta data, one can
  make a better decision of chosen learning algorithm(s) to solve the
  problem more efficiently.

and

Transfer learning aims at improving the process of learning new tasks
  using the experience gained by solving predecessor problems which are
  somewhat similar. In practice, most of the time, machine learning
  models are designed to accomplish a single task. However, as humans,
  we make use of our past experience for not only repeating the same
  task in the future but learning completely new tasks, too. That is, if
  the new problem that we try to solve is similar to a few of our past
  experiences, it becomes easier for us. Thus, for the purpose of using
  the same learning approach in Machine Learning, transfer learning
  comprises methods to transfer past experience of one or more source
  tasks and makes use of it to boost learning in a related target task.

The comparisons still confuse me as both seem to share a lot of similarities in terms of reusability. Meta-learning is said to be "model agnostic", yet it uses metadata (hyperparameters or weights) from previously learned tasks. It goes the same with transfer learning, as it may reuse partially a trained network to solve related tasks. I understand that there is a lot more to discuss, but, broadly speaking, I do not see so much difference between the two.
People also use terms like "meta-transfer learning", which makes me think both types of learning have a strong connection with each other.
I also found a similar question, but the answers seem not to agree with each other. For example, some may say that multi-task learning is a sub-category of transfer learning, others may not think so.

@nbro: I would like to hear from you as you are knowledgable of this. The two differences discussed in the answer section, in my opinion, do not clearly distinguish the two types of learning. For example, is it really true that meta-learning always optimizes a new network only, or transfer-learning does not consider the distribution of tasks (and if not, then what)?
This paper even  identifies meta-learning as transfer learning https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf

I tried to provide an answer (with a slightly different perspective that hopefully will help you).

First of all, I would like to say that it is possible that these terms are used inconsistently, given that at least transfer learning, AFAIK, is a relatively new expression, so, the general trick is to take terminology, notation and definitions with a grain of salt. However, in this case, although it may sound confusing to you, all of the current descriptions on this page (in your question and the other answers) don't seem inconsistent with my knowledge. In fact, I think I had already roughly read some of the cited research papers (e.g. the MAML paper).
Roughly speaking, although you can have formal definitions (e.g. the one in the MAML paper and also described in this answer), which may not be completely consistent across sources, meta-learning is about learning to learn or learning something that you usually don't directly learn (e.g. the hyperparameters), where learning is roughly a synonym for optimization. In fact, the meaning of the word "meta" in meta-learning is

denoting something of a higher or second-order kind

For example, in the context of training a neural network, you want to find a neural network that approximates a certain function (which is represented by the dataset). To do that, usually, you manually specify the optimizer, its parameters (e.g. the learning rate), the number of layers, etc. So, in this usual case, you will train a network (learn), but you will not know that the hyperparameters that you set are the most appropriate ones. So, in this case, training the neural network is the task of "learning". If you also want to learn the hyperparameters, then you will, in this sense, learn how to learn.
The concept of meta-learning is also common in reinforcement learning. For example, in the paper Metacontrol for Adaptive Imagination-Based Optimization, they even formalize the concept of a meta-Markov decision process. If you read the paper, which I did a long time ago, you will understand that they are talking about a higher-order MDP.
To conclude, in the context of machine learning, meta-learning usually refers to learning something that you usually don't learn in the standard problem or, as the definition of meta above suggests, to perform "higher-order" learning.
Transfer learning is often used as a synonym for fine-tuning, although that's not always the case. For example, in this TensorFlow tutorial, transfer learning is used to refer to the scenario where you freeze (i.e. make the parameters non-trainable) the convolution layers of a model $M$ pre-trained on a dataset $A$, replace the pre-trained dense layers of model $M$ on dataset $A$ with new dense layers for the new tasks/dataset $B$, then retrain the new model, by adjusting the parameters of this new dense layer, on the new dataset $B$. There are also papers that differentiate the two (although I don't remember which ones now). If you use transfer learning as a synonym for fine-tuning, then, roughly speaking, transfer learning is to use a pre-trained model and then slightly retrain it (e.g. with a smaller learning rate) on a new but related task (to the task the pre-trained model was originally trained for), but you don't necessarily freeze any layers. So, in this case, fine-tuning (or transfer learning) means to tune the pre-trained model to the new dataset (or task).
How is transfer learning (as fine-tuning) and meta-learning different?
Meta-learning is, in a way, about fine-tuning, but not exactly in the sense of transfer learning, but in the sense of hyperparameter optimization. Remember that I said that meta-learning can be about learning the parameters that you usually don't learn, i.e. the hyper-parameters? When you perform hyper-parameters optimization, people sometimes refer to it as fine-tuning. So, meta-learning is a way of performing hyperparameter optimization and thus fine-tuning, but not in the sense of transfer learning, which can be roughly thought of as retraining a pre-trained model but on a different task with a different dataset (with e.g. a smaller learning rate).
To conclude, take terminology, notation, and definitions with a grain of salt, even the ones in this answer.

I should also note that transfer learning can generally refer to any learning technique that attempts to transfer the knowledge acquired in some domain to another domain. See transfer learning in the context of reinforcement learning, domain adaptation and continual/lifelong learning.

Meta-learning is more about speeding up and optimizing hyperparameters for networks that are not trained at all, whereas transfer learning uses a net that has already been trained for some task and reusing part or all of that network to train on a new task which is relatively similar. So, although they can both be used from task to task to a certain degree, they are completely different from one another in practice and application, one tries to optimize configurations for a model and the other simply reuses an already optimized model, or part of it at least.

Thanks for pointing out that in meta-learning the parameters for optimization are not trained at all. I would be really appreciate if you can provide additional reading on this from reputable resource. We have 2 days left for the bounty so I would like a more comprehensive answer.

okay ill try to get to making an edit, in the mean time you can give this a read in regards to transfer learning https://arxiv.org/abs/1911.02685 and this in regards to meta learning https://arxiv.org/pdf/1703.03400.pdf

The difference really comes down to the fact that in meta-learning, there is a population of tasks $\tau$ which have distribution $p(\tau)$. The goal is to perform well on a task drawn from $p(\tau)$. Generally 'perform well' means that with only a few training steps or data points, the model can give good classification accuracy, achieve high reward in an RL setting, etc.
A concrete example is given in the original MAML paper 1, where the task is to perform regression on data given by a sinusoidal distribution with parameters $p(\theta)$. The meta-learning goal is to get high regression accuracy on tasks where the data is drawn from distributions coming from $p(\theta)$.
In contrast, transfer learning is a bit more general since there's not necessarily a notion of a distribution of tasks. There is generally just one (although there can be more) source problem $S$, and the goal is to do well on a target problem $T$. You know both of these explicitly, unlike in MAML where the goal is to do well amongst any unknown problem drawn from a certain distribution. Very often, this is performed by taking a model that performs well on $S$ and adapting it to work on $T$, perhaps by using extracted features from the model for $S$.
The extent to which this will succeed obviously depends on the similarity of the two tasks. This is also known in the literature as domain adaptation, and has some theoretical results 2, although the bounds are not really applicable to modern high-dimensional datasets.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (Finn et al) 2017.
A Theory of Learning from Different Domains (Ben-David et al) 2010.

I gave one thumb up for this. The notion of distribution is interesting. I just realized it was proposed in the paper from Marcin Andrychowicz1 et. al. (Google DeepMind), 2016 too! It would be great if you can give some discussion about previous work prior to MAML, for example: from Yoshua et. al., (University of Toronto) and Sachin Ravi (Twitter).
Thanks a lot.
I'll give it one more day before selecting answer for this question.

@Chris Cundy For me the difference still is not clear. If we have a distribution over a single task, doesn't this reduced to transfer learning? In that sense, isn't meta-learning more general? For me, both try to take advantage of some prior knowledge and their differences are very nuanced.

The way nbro describes meta-learning, it sounds identical to hyper-parameter optimization. Here I would like to clarify possible differences:
According to Dataset2Vec: learning dataset meta-features

Meta-learning, or learning to learn, refers to any learning approach that systematically makes use of prior learning experiences to accelerate training on unseen tasks or datasets. For example, after having chosen hyperparameters for dozens of different learning tasks, one would like to learn how to choose them for the
next task at hand.

So, while hyper-parameter optimization often uses a single dataset, while meta-learning considers multiple datasets, where one (or multiple) are used to optimize hyper-parameters and then use those parameters for later datasets (without re-optimizing those hyper-parameters).
While this might feel a bit like "invent your own category" (a technique in marketing), meta-learning might be considerably more difficult. The reason is that (classical) hyper-parameter optimization only needs to build models that generalize over a single datasets, while meta-learning needs to generalize over multiple datasets - without seeing the later dataset(s) (at least for the hyper-parameter optimization).


text:
Are there any rules of thumb for having some idea of what capacity a neural network needs to have for a given problem?
To give an example. Let's just consider the MNIST dataset of handwritten digits. Here are some things which might have an impact on the optimum model capacity:

There are 10 output classes
The inputs are 28x28 grayscale pixels (I think this indirectly affects the model capacity. eg: if the inputs were 5x5 pixels, there wouldn't be much room for varying the way an 8 looks)

So, is there any way of knowing what the model capacity ought to be? Even if it's not exact? Even if it's a qualitative understanding of the type "if X goes up, then Y goes down"?
Just to accentuate what I mean when I say "not exact": I can already tell that a 100 variable model won't solve MNIST, so at least I have a lower bound. I'm also pretty sure that a 1,000,000,000 variable model is way more than needed. Of course, knowing a smaller range than that would be much more useful!

Theoretical results
Rather than providing a rule of thumb (which can be misleading, so I am not a big fan of them), I will provide some theoretical results (the first one is also reported in paper How many hidden layers and nodes?), from which you may be able to derive your rules of thumb, depending on your problem, etc.
Result 1
The paper Learning capability and storage capacity of two-hidden-layer feedforward networks proves that a 2-hidden layer feedforward
network ($F$) with $$2 \sqrt{(m + 2)N} \ll N$$ hidden neurons can learn any $N$ distinct samples $D= \{ (x_i, t_i) \}_{i=1}^N$ with an arbitrarily small error, where $m$ is the required number of output neurons. Conversely, a $F$ with $Q$ hidden neurons can store at least $\frac{Q^2}{4(m+2)}$ any distinct data $(x_i, t_i)$ with
any desired precision.
They suggest that a sufficient number of neurons in the first layer should be $\sqrt{(m + 2)N} + 2\sqrt{\frac{N}{m + 2}}$ and in the second layer should be $m\sqrt{\frac{N}{m + 2}}$. So, for example, if your dataset has size $N=10$ and you have $m=2$ output neurons, then you should have the first hidden layer with roughly 10 neurons and the second layer with roughly 4 neurons. (I haven't actually tried this!)
However, these bounds are suited for fitting the training data (i.e. for overfitting), which isn't usually the goal, i.e. you want the network to generalize to unseen data.
This result is strictly related to the universal approximation theorems, i.e. a network with a single hidden layer can, in theory, approximate any continuous function.
Model selection, complexity control, and regularisation
There are also the concepts of model selection and complexity control, and there are multiple related techniques that take into account the complexity of the model. The paper Model complexity control and statistical learning theory (2002) may be useful. It is also important to note regularisation techniques can be thought of as controlling the complexity of the model [1].
Further reading
You may also want to take a look at these related questions

How to choose the number of hidden layers and nodes in a feedforward neural network?

How to estimate the capacity of a neural network?

(I will be updating this answer, as I find more theoretical results or other useful info)

This may sound counter intuitive but one of the biggest rules of thumb for model capacity in deep learning:
IT SHOULD OVERFIT.
Once you get a model to overfit, its easier to experiment with regularizations, module replacements, etc. But in general, it gives you a good starting ground.

Although I understand your point, this sounds like a bad suggestion for several reasons. Ideally, you want to design the neural network before training it and not design by trial-and-error (which can be computationally prohibitive, if your task is difficult, you have a big dataset, big net, etc). What if the training data isn't really similar to the test data, even though this assumption is usually implicitly made? This rule of thumb could have been just a comment.

Maybe you can rephrase this answer by saying that "In theory, if the model overfits the training data, it means that it has a sufficient capacity to model the training data, but this will require you to first train the model and try different configurations".

Thanks for the tip. And @nbro thanks for the clarity.

Personally, when I begin designing a machine learning model, I consider the following points:

My data: if I have simple images, like MNIST ones, or in general images with very low resolution, a very deep network is not required.

If my problem statement needs to learn a lot of features from each image, such as for the human face, I may need to learn eyes, nose, lips, expressions through their combinations, then I need a deep network with convolutional layers.

If I have time-series data, LSTM or GRU makes sense, but, I also consider recurrent setup when my data has high resolution, low count data points.

The upper limit however may get decided by resources available on the computing device you are using for training.
Hope this helps.


text:
Why do we regularize the variational autoencoder with a normal distribution?
When we define the loss function of a variational autoencoder (VAE), we add the Kullback-Leibler divergence between the sample taken according to a normal distribution of parameters:
$$ N(\mu,\sigma) $$
and we compare it with a normal distribution of parameters
$$ N(0,1) $$
My intuition is that it is clever having samples taken from a distribution centered around zero, but I don't understand why we want that examples are taken with a normal distribution.

If you are mathematically inclined, here is an article that discusses the reasoning.
What I get as a take away is that the VAE forces the learned latent space to be Gaussian due to the KL divergence term in the loss function. So, now we have a known distribution to sample from to create input vectors to feed to the decoder, to produce say images of dogs, if the VAE was trained on images of dogs. As you sample from the distribution, you will produce images of different types of dog images.
I assume a different type of distribution could be selected if one uses the proper loss function for that type of distribution, that is, the loss function which would measure the difference in the distribution of the latent space and the desired distribution.
KL divergence is the loss function that forces the latent space distribution to be Gaussian. If you do not "restrict" the latent space, as is the case with a regular autoencoder, you have no idea what kind of vector to select as an input to the decoder to produce a dog image. Without restriction, there are $2^n$ (where $n$ is the dimensions of the latent space) possible vectors you could choose from. Chances of selecting one that produces a dog image would be minuscule.
Well, I hope this helps. I no longer am mathematically proficient (age 75), so I hope my interpretation of the math is correct.
A VAE tends to produce blurry images because there are two terms in the loss function. One term is trying to make the output look like the input while the KL loss term is trying to restrict the latent space distribution. GANs ( generative adversarial networks) don't have this conflict, so they produce much high-quality images.

great answer! please also take a look into this


text:
What does "the expectation is taken across different possible inputs, drawn from the distribution of inputs we expect the system to encounter" mean?
I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter 5.2 Capacity, Overfitting and Underfitting, the authors say the following:

Typically, when training a machine learning model, we have access to a training set; we can compute some error measure on the training set, called the training error; and we reduce this training error. So far, what we have described is simply an optimization problem. What separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. The generalization error is defined as the expected value of the error on a new input. Here the expectation is taken across different possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice.

I found this part unclear:

Here the expectation is taken across different possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice.

The language used here is confusing me, because it is discussing a "distribution", as in a "probability distribution", but then refers to inputs, which are data gathered from outside of any probability distribution. Based on the limited information my studying of machine learning has taught me so far, my understanding is that the machine learning algorithm (or, rather, some machine learning algorithms) uses training data to implicitly construct some probability distribution, right? So is this what it is referring to here? Is the "distribution of inputs we expect the system to encounter in practice" the so called "test set"? I would greatly appreciate it if people would please take the time to clarify this.

The language used here is confusing me, because it is discussing a "distribution", as in a "probability distribution", but then refers to inputs, which are data gathered from outside of any probability distribution. Based on the limited information my studying of machine learning has taught me so far, my understanding is that the machine learning algorithm (or, rather, some machine learning algorithms) uses training data to implicitly construct some probability distribution, right? So is this what it is referring to here?

They're not referring to probability distributions of training data that ML algorithms (implicitly) construct here. The main point of confusion seems to be where you state this:

but then refers to inputs, which are data gathered from outside of any probability distribution

Any data / inputs ever collected always originate from some distribution. We will typically not exactly know what that distribution is, we are often not able to provide a clean expression for it, and it might not even be a nice "smooth" distribution, but that doesn't mean it doesn't exist.
If I collect a large number of photographs of $H \times W$ pixels of streets for the purpose of training a self-driving car, then this collection of training data was collected from some distribution. For each of the pixels in the $H \times W$ plane, there exists some probability distribution that tells us how likely it is for such a pixel to have a certain colour under the data collection procedure that was used to generate our data. This is a largely unknown distribution, for which we don't have a nice mathematical expression, but it does exist. I assume that, in this distribution, it's relatively likely for pixels in the centre to be gray (because streets tend to be gray and we collected data by taking photographs of streets). I also guess it's relatively likely for pixels at the top of the images to be blue, because of the sky. Other than that, we can't say much about the distribution, but it does exist.

Is the "distribution of inputs we expect the system to encounter in practice" the so called "test set"?

Kind of, yeah. Although I suppose the "test set" is mostly a thing in academic settings, where we use a test set to evaluate how well an approach performs on data that it did not observe during training. In the "real world", the distribution of inputs we expect the system to encounter in practice refers to the distribution that generates samples we encounter after "deployment" of the model. For example, this could be the distribution over all images that a self-driving car may encounter when driving anywhere in the world.
Continuing with the self-driving car example, we may get a large generalization error if we only train it on images of streets in one particular city or country, but then afterwards have it drive in many different cities or countries around the world (which may look very different).

For illustration, I use the dog/cat classification task. Suppose, the training data of cat and dog follows the Gaussian distribution(for simplicity) and we trained a model which gives an accuracy as below.

train - 98.2%
val - 97.7%
test - 97.2%

The model is neither overfitting nor underfitting but we want the classifier to achieve an accuracy of 100%  in all the three sets theoretically. You are right that the model learns the distribution of training data to classify the classes. Due to the overlapping of fat-tail in the distribution of cat and dog, it is highly impossible for the model to get 100% accuracy practically. There will be infinite edge cases that we encounter in reality so we can only improve the model by an iterative approach.


text:
What is relation between gradient descent and regularization in deep learning?
Gradient descent is used to reduce the loss and regularization is used to fight over-fitting.
Is there any relation between gradient descent and regularization, or both are independent of each other?

Usually, when talking about regularization for neural networks there are 3 main types:
L1, L2 and dropout. All affect the gradient descent procedure.
L1 and L2 regularization is implemented in the loss function, and therefore are part of gradient descent directly by altering the derivatives of the loss function thereby altering the weight update rules of the network during gradient descent.
For L1 you add a penalty based on the $\mathcal L^1$ norm of the weight vector, while for L2 you add a penalty based on the $\mathcal L^2$  norm.
For dropout, there is no direct impact on the loss function, but you are still interfering in the gradient descent procedure indirectly by masking nodes to alter the forward and backward propagation.


text:
How do I turn this formula of the average degree of a graph into Python code?
I am working through the textbook "Graph-Based Natural Language Processing and Information Retrieval", where I've got a question on implementation of this first Latex looking formula/algorithm.
Can you help me turn the formula under 1.2 Graph Properties into python code? Yes, I know there are many other languages, but python is more user-friendly so I'm starting there, and will eventually rewrite it into C.
As I read the above node example, sorry the D and E nodes were cut off.
Node A has two outflowing arrows notating it as their head node, and it is the one tail node.
This first sentence references the Graphs:
To traverse from A to B, If A to B value is sufficient (above Nx), go to B. If A to B value is below Nx, go A to C to D to A to B total cost is 5.2+7+1+8 = 21.20 traverse cost, this makes sense.

This sentence refers to the Latex formula in the book.
Then to start the formula calculation, the average degree of a graph "a" is equal to the sum of, one over N, times the sum of the in-degree of vertices? Asserting that the sum is a non zero integer between 1 and N?
Ok, I only loaded one page and hope that's not a TOS violation or causes issue, it's a challenge to find people who understand graph theory.
Let me know what questions you have, but I'm just wanting to get clarification if my understanding is what this page is saying.

Please, can you be more specific about the sentences you're referring to? Don't just post the screenshot. Only copy and paste the relevant sentences and remove as many unnecessary details as possible. This question has been flagged as unclear, so please try to clarify it further!

Sure, I'm just trying to understand what the a=1/N algorithm means in the screenshot. For example, a sentence would be "a equals one over N times the sum of degree Vi, where Vi is a non zero integer from one to n"?

Each node is a position in the arrays

values = value of the node
conn = indexes of connected nodes

If its an undirected graph, each node must have all the nodes to which they are attached.
Instead, in directed graphs, only the start node has the index.
For your image:
values = ['A','B','C','D','E']
conn = [[1,2,3],[0,4],[0,3,4],[0,2,3],[1,3]]

Example = 'A' -> 1 ('B') ,2 ('C') ,3 ('D')
def averageGraph(conn):
    if conn.all() != None:
        average = 0
        for node in conn:
            average = average + len(node) #len(node) = nº nodes connected = degree
        return average / len(conn)
    else:
        return None

Thank you Alejandro, that is very helpful! What is your background if you don’t mind me asking as I have a few hundred pages to go through similar to this one.

The Maths are knowledge that you learn at university, but any PDF of introduction to graph theory can explain you very well, as soon as you understand a little they are very simple concepts! Good luck


text:
Can neurons in MLP and filters in CNN be compared?
I know they are not the same in working, but an input layer sends the input to $n$ neurons with a set of weights, based on these weights and the activation layer, it produces an output that can be fed to the next layer.
Aren't the filters the same, in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layer uses this new "image"?

tl;dr The equivalent to a neuron in a Fully-Connected (FC) layer is the kernel (or filter) of a Convolution layer
Differences
The neurons of these two types of layers have two key differences. These are that the convolution layers implement:

Sparse connectivity, i.e. each neuron is connected only to an area of the input, not the whole.
Weight sharing, i.e. similar connections end up having the same weights. This is usually visualized as the same filter traversing the image.

Besides these two key differences, there are some other technical details, e.g. how the biases are implemented. Other than that they perform the same operation.
What causes some confusion is that the input of a CNN is usually 2 or 3-dimensional, while a FC is usually 1-dimensional. These aren't mandatory however. To better help visualize the differences between the two I made a couple of figures illustrating the differences between a conv-layer and a FC one, both in 1D.
Sparse connectivity
On the left are two FC neural networks. On the right, are layers with sparse connections.

Weight sharing
On the left is a sparsely connected network. The colors represent the different values of the weights. On the right is the same network with weight-sharing. Note that similar weights (i.e. arrows with the same direction in each layer) have the same value.

To answer your other questions:

Are filters not the same in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layers use these new "images"?

Yes, if the input of a convolution layer is an image, so will the output. The next layer will also operate on an image.
However, I'd like to note that not all convolution layers accept images as their inputs. There are 1D and 3D convolutional layers as well.

The other answer gives a good overview of the differences between MLPs and CNNs, and it includes 2 diagrams that attempt to illustrate the main differences between MLPs and CNNs, i.e. sparse connectivity and weight sharing. However, these diagrams do not clarify what a neuron in a CNN could be. A better diagram, which illustrates what a neuron is in a CNN, from a CNN and MLP perspective, is the following (taken from the famous article on CNNs).

Here, there are 2 main blocks (aka volumes): the orange block on the left (the input) and the blue/cyan volume on the right (the feature maps, i.e. the outputs of the convolutional layer, i.e. after the application of the convolutions with different kernels).
The circles in the visible stack of the cyan block represent the neurons (or, more precisely, their activations or outputs). We only see $k=5$ neurons stacked: this corresponds to the application of $k=5$ different kernels (i.e. weights) to that specific subset of the input (aka receptive field), hence the sparse connectivity of CNNs. So, these neurons, in the same stack, are looking at the same small subset of the input, but with different weights (i.e. kernels). The neurons, which are not shown in this diagram, that are on the same (vertical) 2d plane (known as feature map) of the same neuron (e.g. the first that we see from left to right) in the cyan volume are the neurons that share the same weights, i.e. we use the same kernel to produce their outputs.
So, in this biological/neuroscientific view of the CNN, when you apply the convolution (or cross-correlation) operation with 1 specific filter (or kernel), you are computing the activation (not to be confused with the activation function, which is used to compute the activation!) i.e. the output of multiple neurons, all of them share the same weights. You stack all these activations on the same 2d plane (known as feature map) of the output volume: note that this operation is just the convolution operation! When you compute the convolution with another kernel, you are again computing the activation of other multiple neurons, which share another different weight matrix, and so on and so forth.
Some authors prefer to use the term convolutional networks, i.e. without the term neural, probably because of this issue, i.e. it's not clear, especially to newcomers, what a neuron would be in a CNN, so the neuroscientific/biological view of CNNs is not always clear, although it's important to emphasize that CNNs were inspired by the visual cortext, so this biological interpretation could (and should) be more widely known or less confusing/misunderstood.
Now, let's address your question more directly.

Aren't the filters the same, in the way that they convert an "image" to a new "image" based on the weights that are in that filter? And that the next layer uses this new "image"?

The filters in a CNN correspond to the weights of an MLP.
A neuron in a CNN can be viewed as performing exactly the same operation as a neuron in an MLP. The big differences between a CNN and an MLP (as explained also in the other answer) are

Weight sharing: Some neurons (not all!) in the same convolutional layer share the same weights. The convolution (or cross-correlation) is the operation that implements this partial forward pass with the same weights for different neurons.

Neurons in a CNN only look at a subset of the input and not all inputs (i.e. receptive field), which leads to some notion of sparse connectivity

A convolutional layer, in a CNN, is composed of neurons in a 3d dimensional volume (or, more precisely, their activations are organized in a 3d volume), rather than a 1-dimensional one, as in an MLP.

CNNs may use subsampling (aka pooling)


text:
Why does the variational auto-encoder use the reconstruction loss?
VAE is trained to reduce the following two losses.

KL divergence between inferred latent distribution and Gaussian.

the reconstruction loss

I understand that the first one regularizes VAE to get structured latent space. But why and how does the second loss help VAE to work?
During the training of the VAE, we first feed an image to the encoder. Then, the encoder infers mean and variance. After that, we sample $z$ from the inferred distribution. Finally, the decoder gets the sampled $z$ and generates an image. So, in this way, the VAE is trained to make the generated image to be equal to the original input image.
Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.
But, as you know, VAE works well. So I think I miss something important or understand it in a totally wrong way.

The VAE uses the ELBO loss, which is composed of the KL term and the likelihood term. The ELBO loss is a lower bound on the evidence of your data, so if you maximize the ELBO you also maximize the evidence of the given data, which is what you indirectly want to do, i.e. you want the probability of your given data (i.e. the data in your dataset) to be high (because you want to use the VAE for the generation of inputs similar to the ones in your dataset). So, the idea is that you optimize both the KL term and the reconstruction (or likelihood) term jointly (i.e. the ELBO). Why? Because, as I just said, the ELBO is the Evidence Lower BOund on the given data, so, by maximizing it, you are also maximizing the evidence of your data. In other words, if you maximize the ELBO, you are finding a decoder that will have a high probability of reconstructing your inputs (i.e. the likelihood term), but, at the same time, you want your encoder to be constrained (i.e. KL term). Please, read this answer for further details.

Here, I cannot understand why the sampled $z$ should make the original image, since the $z$ is sampled, it seems that the $z$ does not have any relationship between the original image.

The relationship is that you will be maximizing the ELBO, which implies (and you can see this implication only if you are familiar with the ELBO loss) you will be minimizing the KL divergence between your posterior and the prior to generate the samples $z$ (i.e. minimizing because there will be a minus in front of the KL term in the ELBO loss) and maximizing the probability of the reconstructed input. More precisely, $z$ is used to reconstruct the input (i.e. the decoder does this), which is then used to calculate the reconstruction loss.
In the mathematical formulations, you will see that the likelihood term of the ELBO is $p(x \mid z)$, i.e. the likelihood of the input $x$ given $z$. The $z$ is the input to the decoder, which produces a reconstruction of $x$. In practice, people will e.g. use the cross-entropy to then calculate the "reconstruction loss" (e.g. see this PyTorch implementation), which should correspond to this likelihood term $p(x \mid z)$. Why does the cross-entropy correspond to a likelihood? Because you can actually prove that the cross-entropy is equivalent to the negative log-likelihood. (Also, note that, in the ELBO loss, $p(x \mid z)$ does not appear, but the logarithm of $p(x \mid z)$ appears, but, for simplicity, I have used $p(x \mid z)$ rather than $\log p(x \mid z)$ above.)

This answer could potentially make you more confused. The concepts aren't really straightforward and it takes several iterations until you get them, so, please, try to read my answer and other sources related to the topic several times.


text:
What are some online courses for deep reinforcement learning?
What are some (good) online courses for deep reinforcement learning?
I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from 2015. It doesn't really teach deep Q-learning at this time.

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer will point the reader to potentially useful courses on RL (also because there aren't many free courses completely dedicated to deep RL), which have at least one lesson on deep RL or function approximation. I have only followed the course by Isbell and Littman and partially the course by David Silver, so I can't ensure you that the other courses are good, but I found these two useful, although not perfect.

Title
Instructor(s)
Focus on deep RL?
Topics
Free

Reinforcement Learning
Charles Isbell, Michael Littman
No
TD learning, convergence, function approximation, POMDP, options, game theory
Yes

Introduction to Reinforcement Learning with David Silver
David Silver
No
MDPs, planning, dynamic programming, model-free prediction and control, function approximation, policy gradients, exploration and exploitation
Yes

CS234: Reinforcement Learning Winter 2020
Emma Brunskill
No
See the course schedule; lesson 6 is about DRL
Yes

Reinforcement Learning
NPTEL
No
Bandits, MDPs, policy gradients, dynamic programming, TD learning, function approximation, hierarchical RL, POMDP
Yes

Reinforcement Learning in the Open AI Gym
Phil Tabor
?
SARSA, double Q-learning, Monte Carlo methods, Q-learning
Yes

Advanced Deep Learning & Reinforcement Learning
DeepMind
No
video 14 discusses DRL topics
Yes

Advanced AI: Deep Reinforcement Learning in Python
Udemy
Yes, it seems
?
No

Machine Learning: Beginner Reinforcement Learning in Python
Udemy
?
?
No

Deep Reinforcement Learning 2.0
Udemy
?
?
No

Modern Reinforcement Learning: Deep Q Learning in PyTorch
Phil Tabor (Udemy)
?
?
No

Modern Reinforcement Learning: Actor-Critic Methods
Phil Tabor (Udemy)
?
?
No

In any case, if you are familiar with RL and deep learning topics, I encourage you to directly read the DQN papers (both by DeepMind folks)

Playing Atari with Deep Reinforcement Learning (2013)
Human-level control through deep reinforcement learning (2015)

Of course, deep RL isn't just DQN, but these are two very important papers that you should read. Other key papers on deep RL can be found here.
Note that, depending on your experience with and knowledge of RL and DL, you may require a few iterations to fully understand these papers, but this applies every time you need to read a research paper.

For the programming part I suggest this YouTube channel by Phil Tabor (he also has a website: neuralnet.ai. I found his videos really useful while I was attending reinforcement learning classes at the uni. He covers basic algorithms like value iteration and policy iteration and also more advanced like deep q learning, covering all main python libraries (Keras, tensorflow, pytorch). Hope it will help you as well!


text:
What does the notation ${s'\sim T(s,a,\cdot)}$ mean?
I have been seeing notations on Expectations with their respective subscripts such as $E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]$. This equation is taken from https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf and $Q^\pi(s,a,R) = R(s) + \gamma E_{s'\sim T(s,a,\cdot)}[V^\pi(s',R)]$ ,in the case of the Bayesian IRL paper.(https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-416.pdf)
I understand that $s_0 \sim D$ means that the starting state $s_0$ is drawn from a distribution of starting states $D$. But how do we understand the latter with subscript ${s'\sim T(s,a,\cdot)}$ ? How is $s'$ drawn from a distribution of transition probabilities?

The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, $T(s,a,.)$ is a probability distribution.


text:
How to calculate the number of parameters of a convolutional layer?
I was recently asked at an interview to calculate the number of parameters for a convolutional layer. I am deeply ashamed to admit I didn't know how to do that, even though I've been working and using CNN for years now.
Given a convolutional layer with ten $3 \times 3$ filters and an input of shape $24 \times 24 \times 3$, what is the total number of parameters of this convolutional layer?

What are the parameters in a convolutional layer?
The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case.
2d convolutions
Example
If the 2d convolutional layer has $10$ filters of $3 \times 3$ shape and the input to the convolutional layer is $24 \times 24 \times 3$, then this actually means that the filters will have shape $3 \times 3 \times 3$, i.e. each filter will have the 3rd dimension that is equal to the 3rd dimension of the input. So, the 3rd dimension of the kernel is not given because it can be determined from the 3rd dimension of the input.
2d convolutions are performed along only 2 axes (x and y), hence the name. Here's a picture of a typical 2d convolutional layer where the depth of the kernel (in orange) is equal to the depth of the input volume (in cyan).

Each kernel can optionally have an associated scalar bias.
At this point, you should already be able to calculate the number of parameters of a standard convolutional layer.  In your case, the number of parameters is $10 * (3*3*3) + 10 = 280$.
A TensorFlow proof
The following simple TensorFlow (version 2) program can confirm this.
import tensorflow as tf

def get_model(input_shape, num_classes=10):
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Input(shape=input_shape))
    model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Dense(num_classes))

    model.summary()

    return model

if __name__ == '__main__':
    input_shape = (24, 24, 3)
    get_model(input_shape)

You should try setting use_bias to False to understand how the number of parameters changes.
General case
So, in general, given $M$ filters of shape $K \times K$ and an input of shape $H \times W \times D$, then the number of parameters of the standard 2d convolutional layer, with scalar biases, is $M * (K * K * D) + M$ and, without biases, is $M * (K * K * D)$.
See also these related questions How is the depth of filters of hidden layers determined? and In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?.
1d and 3d convolutions
There are also 1d and 3d convolutions.
For example, in the case of 3d convolutions, the kernels may not have the same dimension as the depth of the input, so the number of parameters is calculated differently for 3d convolutional layers. Here's a diagram of 3d convolutional layer, where the kernel has a depth different than the depth of the input volume.

See e.g. Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks.

The diagrams were taken from https://ai.stackexchange.com/a/13786/2444.

For a standard convolution layer, the weight matrix will have a shape of (out_channels, in_channels, kernel_sizes). In addition, you will need a vector of shape [out_channels] for biases. For your specific case, 2d, your weight matrix will have a shape of (out_channels, in_channels, kernel_size[0], kernel_size[1]).
Now, if we plugin the numbers:

out_channels = 10, you're having 10 filters
in_channels = 3, the picture is RGB in this case, so there are 3 channels (the last dimension of the input)
kernel_size[0] = kernel_size[1] = 3

In total you're gonna have 10*3*3*3 + 10 = 280 parameters.


text:
What is the current state-of-the-art in unsupervised cross-lingual representation learning?
What is the current state-of-the-art in unsupervised cross-lingual representation learning?

The blog post Unsupervised Cross-lingual Representation Learning (2019), the related paper and slides by Sebastian Ruder (a researcher currently at DeepMind) summarize what you are looking for. In fact, the authors write

We will introduce researchers to state-of-the-art methods for constructing resource-light cross-lingual word representations and discuss their applicability in a broad range of downstream NLP applications, covering bilingual lexicon induction, machine translation (both neural and phrase-based), dialogue, and information retrieval tasks. We will deliver a detailed survey of the current cutting-edge methods, discuss best training and evaluation practices and usecases, and provide links to publicly available implementations, datasets, and pretrained models and word embedding collections.

Maybe later I will update this answer with a list of the SOTA methods/approaches.


text:
How to prove that gradient descent doesn't necessarily find the global optimum?
How can I prove that gradient descent doesn't necessarily find the global optimum?
For example, consider the following function
$$f(x_1, x_2, x_3, x_4) = (x_1 + 10x_2)^2 + 5x_2^3 + (x_2 + 2x_3)^4 + 3x_1x_4^2$$
Assume also that we can't find the optimal value for the learning rate because of time constraints.

Here is a related question.

Well, GD terminates once the gradients are 0, right? Now, in a non-convex function, there could be some points, which do not belong to the global minima, and yet, have 0 gradients. For example, such points can belong to saddle points and local minima.
Consider this picture and say you start GD at the x label.

GD will bring you the flat area and will stop making progress there as gradients are 0. However, as you can see, global minima is to the left of this flat region.
By the same token, you have to show, for your own function, that there exists at least a single point whose gradients are 0 and yet, it is not the global minima.
In addition to that, the guarantee on converge for convex functions depends on annealing the learning rate appropriately. For example, if your LR is too high, GD can just keep overshooting the minima. The visualization from this page might help you to understand more regarding the behavior of GD.

In convex function with an unknown learning rate value (which is not optimal value of that as mentioned) how to ensure that it converges or not?

If the function is convex, GD converges with a decaying learning rate. I guess if you don't make any assumptions on the learning rate, it could keep overshooting the global minima. Say you're 1 units left of minima but due to big LR, you could go like 3 units right of minima, then back to 1 units left of minima and so on.

how to find out whether LR has decaying rate or not? Is there any way to find out the oscillation?

well in your question, are you allowed to set the lr? if not, you have to look for auxiliary information. for example, if your training loss/function value jumps back and forth, this probably indicates that GD is overshooting its target due to big lr.

I'm allowed but can't find the optimal value for lr due to time limit.

okay then you have to monitor the function value and decrease the learning rate when it starts to overshoot. also look into lr scheduling methods such as line search in general.

please add it to your answer to confirm it. gimme upvote in case you want. thanks

Sure. I've added a link that visualizes how GD behaves on loss surfaces. You might find it useful.

You can find by yourself a counterexample that, in general, GD is not guaranteed to find the global optimum!
I first advise you to choose a simpler function (than the one you are showing), with 2-3 optima, where one is the global and the other(s) are local. You don't need neural networks or any other ML concept to show this, but only basic calculus (derivatives) and numerical methods (i.e. gradient descent). Just choose a very simple function with more than one optimum and apply the basic gradient descent algorithm. Then you can see that, if you start gradient descent close to one local optimum (i.e. you choose an initial value for $x$ or $\theta$, depending on your notation for the variable of the function) and then you apply gradient descent (for some iterations), you will end up in that close local optimum, from which you cannot escape, after having applied the gradient descent steps.
See also the question Does gradient descent always converge to an optimum? and For convex problems, does gradient in Stochastic Gradient Descent (SGD) always point at the global extreme value?

There is no way you can be sure you have reached a global minimum. Steepest descent will converge toward where the gradient approaches zero. Depending on the initial conditions ( ie the initial values of the weights) you can and will converge on some minimum. Notice if you run your model several times with random weight initialization you will get slightly different results.  What I do find interesting is that it seems that in general the local minima have roughly the same  value. The cost function is some kind of surface in N space where N is the number of trainable parameters. We do not know what that surface is like and how many local minimums  exist.

You are not even guaranteed to converge to a local minima. You could converge to a saddle point with plain GD.


text:
How can I detect the frame from video streaming that contains a graffiti on city wall?
I am working on a graffiti detection project. I need to analyze data stream from a camera mounted sideways on a vehicle to identify graffiti on city walls and notify authorities with the single best capture of graffiti and its geolocation, etc.
I am trying to use a ResNet50 model pre-trained on ImageNet using transfer learning for my graffiti image dataset. The classification will be done on an edge device as network connectivity may not be reliable.
Suppose I have a series of frames that have been detected to contain graffiti, as the vehicle goes past it, but I only need to report one image (so not all frames containing graffiti in the series). How can I do that?
Ideally, I would like to report the frame where the camera is perpendicular to the wall. Why perpendicular? I think that images containing the graffiti when the camera is perpendicular to the wall will more clearly show the graffiti.

Welcome to AI Stack Exchange. It is not clear to me where you are stuck. You say about ResNet50 "I'm trying as of this post" - so that seems to be something you can already do (and I would guess you know at least something about neural networks, CNNs and image processing). Please give more detalis - in what way are you stuck with the object detection?

Thank you. I am not able to understand the design of how to wrap around a Faster RCNN system around my ResNet-50 model for object detection.  In my knowledge, ResNet-50 is equivalent to a feature extraction technique for image classification. How do I detect the best sample of a series of images with Graffiti to save for further action. Edited post as well.

I have edited your post to further clarify your problem. Make sure it's still consistent with your intent. Furthermore, I remove the second question, because otherwise this post becomes a little too broad. Please, ask your second question in a different and separate post and make sure to clarify your second question better.

Actually, there's still something that's not fully clear. You are apparently asking two different questions: 1) how to identify graffiti in real-time (and that's already a big problem) and 2) how to report the graffiti perpendicular to the wall, once the detection has been made. Is 1) your main question? Please, edit your post to clarify this.

I saw this job on Upwork. Like 30 people applied. What's your secret?

Not through upwork, this is through a not-for-profit org I work for.

ResNet is an architecture for object recognition and you may use it to do your classification task.  Fast RCNN may improve your results but is a more difficult architecture to implement.  If you want to go in this direction the best place to start is the arxiv paper of the Fast R-CNN (arxiv.org/abs/1504.08083).  If I am not wrong, there is an implementation of fast-rcnn in pytorch if you really want that.
You seem to be new to deep learning.  I would strongly recommend starting simple. ResNet-50 will probably be more than enough for your application. Moreover, I suggest using a lib like fast.ai.


text:
What is the difference between hill-climbing and greedy best-first search algorithms?
While watching MIT's lectures about search, 4. Search: Depth-First, Hill Climbing, Beam, the professor explains the hill-climbing search in a way that is similar to the best-first search. At around the 35 mins mark, the professor enqueues the paths in a way similar to greedy best-first search in which they are sorted, and the closer nodes expanded first.
However, I have read elsewhere that hill climbing is different from the best first search. What's the difference between the two then?

Let's see their definition first:

Best First Search (BFS):

Best-first search is a search algorithm that explores a graph by expanding the most promising node chosen according to a specified rule.
estimating the promise of node n by a "heuristic evaluation function ${\displaystyle f(n)}$ which, in general, may depend on the description of n, the description of the goal, the information gathered by the search up to that point, and most importantly, on any extra knowledge about the problem domain."

Hill Climbing (HC):

In numerical analysis, hill climbing is a mathematical optimization technique that belongs to the family of local search. It is an iterative algorithm that starts with an arbitrary solution to a problem, then attempts to find a better solution by making an incremental change to the solution. If the change produces a better solution, another incremental change is made to the new solution, and so on until no further improvements can be found.

Base on the definition, we can find the following differences:

The aim of BFS is reaching to a specified goal by using a heuristic function (it might be greedy) vs. HC is a local search algorithm
BFS is mostly used in the graph search (in a wide state space) to find a path. vs. HC is using for the optimization task.

It's not true that BFS algorithms don't look at history. A* is a BFS algorithm and it looks at the history (in fact, its evaluation function is $f(n) = g(n) + h(n)$, where $h$ is an admissible heuristic and $g$ is the history (i.e. cost of the path from the start node to $n$), but it also looks at the estimates to the goal. That's the idea.

@nbro Many thanks for the comment. You're right. I wanted to explain about the locality of HC as I've explained in the first difference! So, I've removed that statement. Do you suggest any more differences?

Yes, actually, I think you should explain how BFS algorithms (e.g. A*) choose to "move" (i.e. just explain a little bit about the evaluation function I talked about in my comment above) and how HC also "moves" in order to solve the optimization problem.

Also, note that the OP was specifically asking about "greedy" BFS. I think it's important to distinguish a greedy from a non-greedy BFS. A* is non-greedy. (At least, in the title). But I guess he was also confused ;)


text:
What are episodic and non-episodic domains in reinforcement learning?
I was reading about the temporal difference (TD) learning and I read that:

TD handles continuing, non-episodic domains

Assuming that continuing means non-terminating, what does non-episodic or episodic domain mean?

Here's another relevant post https://stats.stackexchange.com/q/271356/82135.

Assuming that continuing means non terminating, what does non-episodic or episodic domain mean ?

Non-episodic means the same as continuing. The quote you found is not listing two separate domains, the word "continuing" is slightly redundant. I expect the author put it in there to emphasise the meaning, or to cover two common ways of describing such environments.
Episodic domain problems are ones that terminate, or otherwise naturally split into groups of time steps that can be considered separately.