text:
Does a second-order fully-connected layer have any uses?
I was thinking about implementing second-order regression via a fully-connected layer, and I came up with this:

$X$ is the input data, shaped $(features, batch\_number)$.
$w0$ is the bias, shaped $(output\_dims,)$.
$w1$ and $w2$ are the weights, each shaped $(output\_dims, features)$.
$Y$ is the target data, shaped $(output\_dims, batch\_number)$.
$\hat{Y}$ is the output of the layer: $\hat{Y} = w2 \cdot X^2 + w1 \cdot X + w0$

$X^2$ is the element-wise square of $X$ taken over its first axis. (I.e., what you would expect intuitively.)

We will then minimize the MSE between $Y$ and $\hat{Y}$ using gradient descent.
Is this a valid implementation of second-order regression?
Is this layer used in any deep learning architectures? What is it useful for?
Is this layer (exactly) equivalent to some MLP?

Is this a valid implementation of second-order regression?

No, but it is not far off.
To perform a full second-order regression, you will need all terms for $x_{i,j}x_{i,k}$ where the first index is the example and the second index the feature. This includes every combination of two input variables. In your element-wise squaring you only produce terms $x_{i,j}x_{i,j}$, only using each feature squared, so are missing a lot of potential second-order terms.

Is this layer used in any deep learning architectures?

In general no. One of the goals of deep learning is to automate the discovery of useful non-linearities - including feature interaction terms - in the hidden layers. These won't be exactly the same as your idea, but can work well enough to get good performance from the neural network on a given dataset.
Pre-calculating a specific set of interactions might help with some data sets, but comes at a high cost of squaring the number of features represented in any given layer, and thus multiplying the number of weights required between two layers by the size of the input layer again. Your original idea would only double the size of the input layer, but would be less general.

What is it useful for?

It is useful in linear regression as a way to find interactions between features. Or you can think of it as a way to automatically search through a subset of possible nonlinear interactions to see if any of those have a strong correlation to the target variable.
There is a danger though that by adding a large number of terms (and associated parameters), that you may find spurious correlations and thus overfit the training data. This is more likely, the larger the number of interactions that you add to the input. Usually this kind of feature generation is paired with regularisation, to reduce the chance of overfitting.

Is this layer (exactly) equivalent to some MLP?

No, although because MLPs can approximate any function, it should be possible to train an MLP such that it performs very similarly.
It may not hurt to insert a layer like the one you describe into some relatively small layer of an MLP, and use your idea in pracice. I expect in some circumstances it may help, but it will be a little bit of an edge case, mostly not making much difference or real advantage. However, no harm in trying it on a few sample datasets. This applies to both your idea as written, using just the squared feature terms, and to the full second order expansion - both may have their niche where they could be useful.


text:
How to actually teach the ANN the resulting weights of different training inputs?
I thought I have implemented the code (from scratch, no library) for an artificial neural network (first endeavour in the field). But I feel like I miss something very basic or obvious.
To make it short: code works for a single pair of in-/out-values but fails for sets of value pairs. I do not really understand the training process. So I want to get this issue out of the way first. The following is my improvised training (aka all that I can think of) in pseudocode.
trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)

while(trainingData not empty) {
  for(0<iterations) {
     set = trainingData.pop()

     updateInput(network, set.in)

     forwardPropagate(network, links)

     linkUpdate = backPropagate(network, links, set.out)

     updateLinks(linkUpdate, links)}
}

Is this how it is supposed to work? Do you feed in your training data set by set (while-loop)?
Edit 1: because my final comment did distract from the issue at hand.
Edit 2: less wordy, more code-y

Removed my comment for clarity.

Do you mean in terms of what I have set up so far or do you mean in terms of describing my problem? I am afraid you mean the latter which could be an issue. Would pseudo code be acceptable? Hold on... Edited question.

Your network must have something which persist like weights add bias
Your new implementation would be like this :
trainingData = [{in: [0,0], out:[0]}, {in: [0,1], out:[0]}, ...];
iterations = 10000

network = graphNodesToNetwork()
links = graphLinksToNetwork()
randomiseLinkWeights(links)
weights = []

while(trainingData not empty) {
    for(0<iterations) {
       set = trainingData.pop()

       weights = updateInput(network, set.in)

       forwardPropagate(network, links)

       linkUpdate = backPropagate(network, links, set.out, weights)

       updateLinks(linkUpdate, links)}
}

Inshort retain weights, and backprapogate.

It looks like you are training you model 10,000 times on one piece of data, and then dropping that piece and moving to the next.
This will not work: the model will become extremely good at learning one piece of data, but will then forget about it when optimizing for the next piece.
Instead, either pick one example at random in each iteration, or compute the gradient for all 4 examples and just update in that direction instead.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
LLMs as "fuzzy JPEGs"
We should

conceptualize LLMs as very quirky and very experimental if advanced
information retrieval systems (fuzzy JPEGs)

according to point 7 in the screenshot below (grabbed from pmarca's Twitter feed).
Could someone please help me understand how "fuzzy" and "JPEGs" are relevant here?

JPEGs are not a great analogy, but the idea is lossy compression. If you use arithmetic coding on the indices of correct output from the LLM, then minimizing cross-entropy loss minimizes your coding scheme. If you were to always use the most likely output (or only sample from the top 10% of tokens), you'd have a lossy compression.

@programjames here, does "fuzzy" mean "fuzzy logic" or a "not sharp image"?

The latter. JPEGs are already lossy, so it wouldn't make sense to add more noise when reconstructing them. The hidden layers of LLMs do happen to implement fuzzy logic though, so that is confusing.

For reproducibility, please, provide the link to the tweet

@nbro done. https://twitter.com/pmarca/status/1774547180713631885

That sentence makes little sense to me. Possibly, there's a typo. Specifically, the "if" seems to be a typo. I suppose they meant to write "as". I wouldn't spend much time on deciphering tweets, which were maybe written by people that don't know much about the topic. I've never heard of "fuzzy JPEGs". JPEGs are lossy (I think), which means you can lose info. Maybe that's what the "fuzzy" refers to. I'm unaware of any algorithm that has that name, but I know little about those things.

As a psychologist, Alison Gopnik's analogy of a "fuzzy jpeg" refers to human brain development. Young children's brains might be like a "fuzzy jpeg," but they can still demonstrate reasonable learning and knowledge. She extended the analogy to artificial intelligence/machine learning. While a neural network is still fairly "fuzzy," it can still produce reasonable and sometimes surprising results.
Of course, at the risk of overfitting, the more we train a neural network the less fuzzy it becomes.
But I think it is a poor analogy, given that a JPEG is a static image and a neural network is proven to be Turing complete (computationally universal).

I think some mixup on "Turing complete". The universal approximation theorem says nothing about that. I believe that recurrent neural networks can be viewed as trainable Turing machines, but not sure if there's proof of universal computation? If you have a reference for the proof you mention at the end, I would appreciate a link

The Universal approximation theorem only addresses continuous functions, not sufficient to be Turing complete.
I have several references to papers that address neural networks as Turing complete, therefore also computational universal. I have a list of attributions to papers that prove certain classes and configurations of ANNs as Turing complete. I will share the links to the online versions I can find.

Here are a few relevant papers addressing the topic of Turing completeness of ANNs, there are many more. "How Powerful are Decoder-Only Transformer Neural Models?"  https://arxiv.org/pdf/2305.17026.pdf "On the Turing Completeness of Modern Neural Network Architecture" https://arxiv.org/abs/1901.03429 "Attention is Turing Complete" https://jmlr.org/papers/volume22/20-302/20-302.pdf

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Distinguishing between the fundamental structures of the convolutional neural network and the recurrent neural network: hierarchical vs sequential
I'm trying to distinguish between the fundamental structures of the convolutional neural network and the recurrent neural network. Convolutional neural networks build a hierarchical model from the data using the convolutional layers. Recurrent neural networks build a sequential model from the data using sequential passing of information (are "recurrent layers" (to contrast with the aforementioned convolutional layers) even a thing? This website seems to have a tag for recurrent-layers, so I guess so?). The differences here aren't clear to me; specifically, it isn't clear to me how one is "hierarchical" whereas the other is "sequential", precisely how the "hierarchical" structure differs from the "sequential" structures, and how this "hierarchical" vs "sequential" distinction / inductive bias affects the suitability (effectiveness) of the models for different tasks (images vs language is the obvious case, but I'm also interested in this in a more general/abstract sense).
P.S. With regards to the "recurrent layers" question, I'm familiar with RNN architecture and memory cells (LSTM, GRU), but I've never seen the "recurrent layer" terminology, and it isn't clear to me how any part of the RNN architecture is a "recurrent layer", as opposed to the convolutional layers of CNNs, which is clearer to me as having distinct "layers" for distinct features (for instance, in an image).
P.P.S. Given the "sequential" structure, are recurrent neural networks not a hierarchical model? Do they not model things hierarchically? Or do they also model things hierarchically (although, it seems to me that this clearly isn't as clear as with convolutional neural networks)?

My answer is based on these 2 papers. https://arxiv.org/pdf/2110.13985 and https://arxiv.org/pdf/2111.00396 and here is the tutorial video https://www.youtube.com/watch?v=EvQ3ncuriCM  that explains these two paeprs.
As they have shown, you can  re-write the sequential operations of RNN in terms of convolutions. Therefore mathematically speaking CNNs can have an equiavalent sequential version that takes one pixel at a time and updates the hidden state and then get the next pixel and so on, until all the pixels of the image is fed to the RNN model.


text:
What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?
My understanding is that normal recurrent neural networks (RNNs) are not good at keeping past information from different time scales. Furthermore, my understanding is that Gated RNNs, such as Long Short-Term Memory, model the keeping and forgetting mechanisms explicitly with sigmoid activations, namely gates. What is it about sigmoid activations in particular that allows for the keeping and forgetting of past information from different time scales?

It is not the sigmoid in particular.
LSTMs and other memory-based recurrent networks are based on the idea of keeping an internal state that acts as a "canvas" in which the model can decide what to write (and thus keep in memory) and what to erase (and thus what to forget).
Observe the top horizontal line in the image below. The line represents the "internal state" $C$. Such state is not used as output for the outside world; it is only passed from the cell to itself at different time steps. This is where the old information is "stored".
The sigmoid is a convenient means to write information in $C$, as the output of a sigmoid ranges between $[0,1]$. For example: think of computing point-wise multiplication between $C$ and some other generic vector $v$. If all values of $v$ are in range $[0,1]$, we are essentially masking $C$, i.e. selectively erasing information from it.
LSTMs try to learn the values of such vector $v$. To make sure that is it in the desired range, sigmoid activations are used.

For more information, I recommend consulting Goodfellow's book Deep Learning, that has a section dedicated to recurrent nets and sequence modeling in general.

Thanks for the answer. Pictured is the Gated Recurrent Unit, correct? So it is this unit in particular that is responsible for the keeping and forgetting of past information from different time scales, rather than just the sigmoid activation itself?

The picture is not a Gated Recurrent Unit. It is a Long Short Term Memory (LSTM) unit, which is similar to a GRU, but not exactly the same. Apart from that, you are correct: it is not the sigmoid that is responsible for keeping and forgetting information, but rather the cell architecture as a whole.


text:
What are pros and cons of using a multi-head neural network versus a single neural network for multi-label classification?
I haven't been able to find a good discussion specifically comparing the two (only one describing a classification and regression problem). I am training a classifier to learn both age and gender based on genomic data. Every sample has a known age and known gender (20 classes in total).
Currently, I am using a single neural network with a sigmoid activation in the last layer with a binary_crossentropy loss. This works fine. However, I also see people using multi-head neural networks where, for example, a set of shared layers would split in to two either additional dense layers or in to two final layers for classification – each with an independent loss (in my case likely a categorical_ce).
What I am unsure of, though, are the advantages and disadvantages between the two (maybe advantages and disadvantages are not the right words to use – actual differences between the two might be more appropriate and when one might use one of those over the other)?
I want to be able to calculate the usual metrics – TP, FP, etc. after training – presumably it would be easier with two heads at the end of the network, as you can work with two independent sets of predictions to calculate these?

If I understood things correctly: You have a task which you need to estimate two values, gender and age. Your question revolves about the difference between networks which share layers for both inputs, whether the shared layers should be followed independent linear layers.
Firstly, using shared layers in the networks of two related tasks may be useful to create more general latent representations in the network hidden layers. It also can speed up training, the shared layer will learn useful features more quickly than if there were two separate networks for each task.
Some examples which demonstrate the potential benefit of shared networks can be found in the papers for two RL algorithms, A3C and PPG (PPG adds some extra tricks to the shared layers it):
http://proceedings.mlr.press/v48/mniha16.html
https://arxiv.org/abs/2009.04416v1
Whether the shared layers should be followed by many separate linear layers or a single one, at least for me, isn't something easy to deduce. Intuitively, having a single linear mapping after the shared layers will help prevent over-fitting because the shared layers induce more general features in those layers. While having many separate layers may be useful if there is some complex non linear mapping between the final output and the latent features from the shared layers.
I think the best way to find out is just to experiment with it and see which gives the best results.
A little bit anecdotal, but, an example from my experience:
Shared layers are commonly used in RL for actor critic algorithms. A network takes an image as input and outputs an action and a value (the output for the actor and the critic, receptively). Generally a single linear mapping from the shared layers works just fine, even better than more complex networks.
=========== Edit
In pseudo-code, this is the networks that came to my mind:
# Network 1
 shared_layer = Linear(input_dim, latent_dim)
 output_layer = Linear(latent_dim, m + n)

# Network 2
 shared_layer = Linear(input_dim, latent_dim)
 output1_layer = Linear(latent_dim, m)
 output2_layer = Linear(latent_dim, n)

And, in the question you mention about using BCE with Network 1 and changing to CE for one of the outputs of the  Network 2.
The networks themselves are equal to one another. One implementation might be more practical than the other, but they are the same. Depending on the framework you use, you can use either BCE or CE loss in Network 1 and 2. In Network 1 this would mean taking the output  of the last layer and slicing the outputs for age and gender into two variables and applying a loss function in each of them.
That said, I would expect to see a difference between using CE or BCE for age classification. When training with BCE it's possible that one estimator turn to be more 'optimist' or 'pessimist' and gives overall high/low probabilities (this will depend on factors such as whether there is class unbalance, if those are taken into account in the training procedure, etc...). And, that will mean that when you take the maximum probability of the age outputs there will be some bias.
CE seems to me to be a more appropriate choice for age classification. Using CE will not prevent bias if there is class unbalance in your data, but with it is more straightforward to handle these issues.

Thank you, my question more more about the difference between shared layers and a 2 separate (as in the case with A2/a3c and one shared layers with a final sense layer of number of outputs (not multi head) and each label being treated as binary, so two neurons would fire in the last layer, one for organ and one for gender. Intuitively I am struggling to see the true difference between the two..  especially if layers are shared upstream which is definitely the case for me. I do have an architecture of shared layers for the reasons you mentioned.

@user9317212 Made an edit to the answer!

thanks! for network1 why would you need to apply two losses when the bce loss handles each class as a binary? and therefore can assign to each sample both an organ and an age classification?

You can apply BCE loss for the task. If you have the exact age of your samples you may also treat the problem as a regression problem, and postprocess the estimated value if needed. All these alternatives should work. Which option is better? BCE loss may be a worse alternative to CE for the reason given in the answer (also, a regression loss can be an even better alternative because it provided a richer feedback). Practically speaking, it is difficult to say whether there will be a significant difference in performance. You may test it to find out!

Thank you! Will try all methods (it is strictly a classification though and I don't need to do a regression for the purposes of this) in also using interpretation tools to find out why the classification decisions are as they are.


text:
How do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, is $1$?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:

The choice of activation function is a critical part of neural network design. In the case of the perceptron, the choice of the sign activation function is motivated by the fact that a binary class label needs to be predicted. However, it is possible to have other types of situations where different target variables may be predicted. For example, if the target variable to be predicted is real, then it makes sense to use the identity activation function, and the resulting algorithm is the same as least-squares regression. If it is desirable to predict a probability of a binary class, it makes sense to use a sigmoid function for activating the output node, so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$.

I've read about sigmoid functions, but it isn't clear to me how they make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$. So how do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$?
EDIT: I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this).

Not clear whether you are asking about the target value $1$, or how sigmoid can represent probabilities at all, or how using sigmoid will result in a meaningful probability measure in practice.

Hi @NeilSlater , I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this). I have edited my question for clarity.

I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this).

They don't in general. In the quoted text, there is an explicit constraint that means this can be the case:

If it is desirable to predict a probability of a binary class

(emphasis mine). This means that the target value $y \in \{0,1\}$. Which in turn means that training data labels will all be $0$ or $1$.
At this point, if you pass in some calculated weighted sum of features to a sigmoid function, it will output a real value between $0.0$ and $1.0$ that can be interpreted as a probability. But without more going on, it is not linked in any way to the probability of the label being $1$.
To complete things, there is also an implicit constraint (not referred in your quoted text) that you will use a training approach that will cause this weighted sum of features to drive the sigmoid to represent a meaningful probability based on the source data. The most important part of this is to use a cost function which is minimised when $\hat{y} = \mathbb{P} \{y = 1 \mid \mathbf{x}\}$ where $\mathbf{x}$ is the input features associated with a label. The most usual cost function here would be based on binary cross-entropy loss:
$$C = -\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1 -\hat{y}_i)$$
You can show using calculus that this term is minimised when $\hat{y}$ is directly related to the frequency of $y = 1$ in the training data. Adding features $\mathbf{x}$ makes this more complex, but as a first pass to understanding what this is doing, you can simply use a list of $N$ times $0$ or $1$ outputs with no input data, and show that the sum above is minimised by a fixed $\hat{y}$ equal to the proportion of $y = 1$ – the rationale is that you would use a fixed $\hat{y}$ as your guess for each value if you had no input data about any example to go on.
One important detail of the related maths is that the activation function does not affect the target of the convergence. However, it may still affect the speed and stability of the convergence. The combination of binary cross-entropy loss and sigmoid activation is popular because the gradient calculation is simple and relatively stable.

Here is the maths worked through for a population of $N$ examples $y_i$ with no feature data (so no $\mathbf{x}_i$), $M$ of which have $y_i = 1$ and $N - M$ of which have $y_i=0$. We will guess a fixed value of $\hat{y}$ so that it minimises the mean cross-entropy loss:
$$\text{argmin}_{\hat{y}} C  = \text{argmin}_{\hat{y}}-\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}) + (1-y_i)\text{log}(1 -\hat{y})$$
The factor of $\frac{1}{N}$ can be removed as it doesn't affect what value of $\hat{y}$ causes the minimum. In addition, we have $M$ values where $y = 1$ and $N - M$ values where $y = 0$, so we can simplify the sum.
$$= \text{argmin}_{\hat{y}}-M\text{log}(\hat{y}) - (N-M)\text{log}(1 -\hat{y})$$
To find the stationary point, take the derivative and set equal to zero (I reversed signs here since $-0 = 0$).
$$\frac{d}{d\hat{y}}[ M\text{log}(\hat{y}) + (N-M)\text{log}(1 -\hat{y})] = 0$$
$$\frac{M}{\hat{y}} - \frac{N-M}{1-\hat{y}} = 0$$
$$(1-\hat{y})M - \hat{y}(N-M) = 0$$
$$M = \hat{y}N$$
$$\hat{y} = \frac{M}{N}$$
This of course is the same as the probability that a randomly selected $i \in (1,N)$ will have $y_i = 1$.

Wow, such an amazing, illuminating answer! Thank you very for taking the time to write this.


text:
How to prove the second form of Bellman's equation?
I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid S_{t} = s]$ where the return $G_{t}$ is defined as follows: $G_{t} = \sum_{k=0}^{\infty}{\gamma^{k}R_{t+k+1}}$.
I tried to use the linearity of the expectation as follows: $v(s) = \mathbb{E}[R_{t+1} \mid S_{t} = s] + \mathbb{E}[\sum_{k = 1}^{\infty}{\gamma^{k}R_{t+k+1}} \mid S_{t} = s]$
Which gives us: $v(s) = \mathbb{E}[R_{t+1} \mid S_{t} = s] + \gamma\mathbb{E}[\sum_{k = 0}^{\infty}{\gamma^{k}R_{(t + 1) + k + 1}} \mid S_{t} = s] = \mathbb{E}[R_{t+1} \mid S_{t} = s] + \gamma\mathbb{E}[G_{t + 1} \mid S_{t} = s]$
I also tried to develop the second formula: $v(s) = \mathbb{E}[R_{t+1} \mid S_{t} = s] + \gamma\mathbb{E}[v(S_{t+1}) \mid S_{t} = s]$ and I'm tempted to say that $\mathbb{E}[G_{t+1} \mid S_{t} = s] = \mathbb{E}[v(S_{t+1}) \mid S_{t} = s]$ but that would only be right in the case that both follow conditions are verified:

We have the value function of a particular state $s^\prime$ inside the expectation of the second term (something like $\mathbb{E}[v(s^\prime) \mid S_{t} = s]$ which would directly give $v(s^\prime)$ since it's a scalar) and not $v(S_{t+1})$.
We have $\mathbb{E}[G_{t+1} \mid S_{\textbf{t+1}} = s^\prime]$ in the second term.

I'm probably not understanding something correctly especially what $v(S_{t+1})$ would mean (that wasn't covered in the material I'm following but for me it would be just a function that maps the possible states at time step $t+1$ to the expected return starting from that step at that time step).

This seems like it would be more appropriate for cs.stackexchange.

@ThePointer thank you for your comment. I'm sorry I didn't see it earlier, I've already solved my question and posted it as an answer. I wonder if I should just delete the question if it's irrelevant to this forum.

It might not be irrelevant (there's a tag for the Bellman equations in the context of reinforcement learning), but cs.stackexchange might have been the better place for this question. Nonetheless, I suggest you leave it, so that others may benefit from it in the future. And close your question by accepting your answer when the system allows.

@ThePointer thank you! I'll keep that in mind for future similar questions.

@ThePointer I disagree, in as much as questions about the maths behind reinforcement learning - or any AI-related topic - are very much on-topic here. It is more often the coding problems that get marked as off-topic here. When the coding issue is regardling theory comprehension (as opposed to more obvious bug or programming language issue) then I believe that should be on topic here too. If you'd like to clarify or debate further, then you could take it to meta.

@NeilSlater Ok, thanks for the clarification.

Yes, I just want to further confirm what Neil already said: this type of question is perfectly fine for our site. Actually, I would like to see more and more questions of this type here. I will soon delete these comments.

Since my question arose from my incomprehension of $v(S_{t + 1})$ and since I got clarifications on it by Neil Slater, I thought I'd go back to this question and try to answer it again.
So I'm assuming that $v(S_{t + 1})$ is a random variable made by the composition of the state-value function $v$ and the random variable $S_{t + 1}$.
Since $v(s) = \mathbb{E}[G_{t + 1} \mid S_{t + 1} = s]$, the random variable $v(S_{t + 1})$ is $\mathbb{E}[G_{t + 1} \mid S_{t + 1}]$. Using a corollary of the total expectation theorem we get that $\mathbb{E}[v(S_{t + 1}) \mid S_{t} = s] = \mathbb{E}[G_{t+1} \mid S_{t} = s]$. This proved, we can conclude using first developments of the question.


text:
Where does the so-called 'loss' / 'loss function' fit into the idea of a perceptron / artificial neuron (as presented in the figure)?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions presents the following figure:

$\overline{X}$ is the features, $\overline{W}$ is the weights, and $\phi$ is the activation function.
So this is a perceptron (which is a form of artificial neuron).
But where does the so-called 'loss' / 'loss function' fit into this? This is something that I've been unable to reconcile.

EDIT
The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to hanugm's answer, it is external and instead used to update the weights of the neuron. So it seems that I misunderstood what was written in the textbook.
In my question above, I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure.
Is the loss / loss function part of the architecture of a perceptron / artificial neuron? I cannot see any indication of a loss / loss function in figure 1.7, so I'm confused about this. If not, then how does the loss / loss function relate to the architecture of the perceptron / artificial neuron?

there is no loss function on this figure. a loss function is a measure of an error between a predicted value and the ground truth. it can be a mean squared error (for regression), or cross-entropy for a classification problem. take a look at this list. for a simple linear perceptron, the problem becomes a linear equation: $y=b+ax$, e.g. what is above the line ($y>0$) is class one, otherwise class two. in this case, you just need to fit a line between samples of 2 classes

It's not clear to me what the question is here. Are you asking what is the loss function used to train the perceptron?

@nbro The way the loss function was introduced in the textbook seemed to imply that it was part of the architecture of the perceptron / artificial neuron, but, according to hanugm's answer, it is external and instead used to update the weights of the neuron. So it seems that I misunderstood what was written in the textbook.

So, is your question: is the loss function of the perceptron somehow part of its architecture? If that was the question, it may be a good idea to edit your post to clarify that.

@nbro I pretty much assumed that the loss function was part of the architecture of the perceptron / artificial neuron, and then asked how it fit into the architecture, since I couldn't see any indication of it in the figure. I'll edit for clarity.

Loss function is a function used to measure the loss. It is not used in any component of a neuron. It is used in updating the weights of the neuron i.e., in order to train the neuron.
The contribution of a loss function is in the updation of $\bar{W}$.
For a given $\bar{X}$ and $\bar{W}$, the neuron gives a post-action value $h$. But the desired output may not be exactly $h$. The distance measure between the desired value (say $h^\prime$) and post-activation value $h$ is called the loss of the perceptron for $\bar{X}$.
We want to decrease the loss of the perception i.e., $\left\vert h - h^\prime \right\vert$. The only task we can do is changing $\bar{W}$, the parameters of the perceptron model given in the question. In order to update the weights, we have to calculate the derivative of loss function wrt the weights of the model and then update weights using some updation rule.
Thus, loss function will come into play during the training phase of the neuron.
To be concise: currently you are dealing with the neuron and its components. Loss functions for training the neuron will come later.

The loss function is simply a way to measure how wrong a neural network is, it doesn't affect the output of the neuron.
Say we have a neural network with 3 output neurons that attempts to classify images of cats, dogs, and humans. The output it gives is the confidence of the neural network's classification. For example if the output is [0, 0.2, 0.8] (0 being the output of the 1st neuron, 0.2 of the 2nd, and 0.8 of the 3rd), this means that the neural network thinks that the image has 0% probability of being a cat, 20% of being a dog and 80% of being a human.
Imagine that the image that was shown to the network is a human, we can say that the target values are [0, 0, 1] because we want it to output that the image is a human with 100% confidence. Now we must measure how wrong the prediction actually was using a loss function. There are many loss functions, but for simplicity I'll use the squared error. In this case the loss will be equal to (1-0.8)^2=0.04 (expected value - output)^2.
The closer the output is to 1, the result inside of the brackets will be closer to 0, so the loss will be smaller. The objective is to minimize this loss function. For example, if the output was 1 instead of 0.8, the network's loss will be (1-1)^2 = 0. If the output was 0.2 instead, the loss would be (1-0.2)^2 = 0.64, which is larger than the two previous, as it is 'more wrong'.
To train the network we use this instead of accuracy for the following reason. With both these outputs [0, 0.1, 0.9], [0.2, 0.3, 0.5] the network predicts 'human', the largest value, but in the first case it is 90% sure whereas in the second it is only 50% sure. We can say that the first network is better, but if we only used accuracy, as both predict the same, they would appear to be just as good.
The same happens when they make a mistake. If the expected values are [0, 1, 0] and one model predicts [0.5, 0.4, 0.1] and the other predicts [0.9, 0, 0.1], they both got it wrong, but the first one was less wrong. The first loss would be (1-0.4)^2 = 0.36 and the second would be (1-0)^2 = 1, which is much higher

@The Pointer, if I were you I would check Sentdex's Neural Networks From Scratch series. Here is the link: https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3

Assume we have a binary classification problem, which we want to solve with a simple single-layer perceptron. For a 2d space, a perceptron will have 2 inputs $x_1$ and $x_2$, and a bias denoted $x_0$ which is always $x_0=1$. It also has corresponding learnable weights $w_0$, $w_1$ and $w_2$.
This can be vectorized:
$$
\overline{x} = \begin{bmatrix} 1 \\ x_1 \\ x_2 \end{bmatrix}; \
\overline{w} = \begin{bmatrix} w_0 \\ w_1 \\ w_2 \end{bmatrix}
$$

Then the pre-activation function $a = \overline{w}\cdot\overline{x}$ nothing more than a linear equation and can be unfolded to $a = w_0 + w_1x_1+w_2x_2$.
In the simplest case $h$ can be defined as follows:
$$h  = \begin{cases}
      \text{class 1} & \text{if $a>0$}\\
      \text{class 2} & \text{otherwise}
    \end{cases}
$$
or it can also be defined as $h=\text{sign}(\overline{w}\cdot\overline{x})$, which means that samples above the line belong to class 1.

Since our $h$ is not continuous and not differentiable, we cannot apply gradient descent optimization. Instead, the weights can be optimized iteratively:

for the details, you can refer to Wikipedia
So answering your question, for this particular case, there is no loss function. Optimization of a loss function usually involves SGD which requires the function to be continuous and differentiable. For instance, if $h$ is defined as a sigmoid function, you can use binary cross-entropy as a loss function.
There are also other optimization approaches, such as genetic algorithm, expectation-maximization and hill-climbing that do not require a loss function.


text:
An explanation involving the sign activation, its affect on the loss function, and the perceptron and perceptron criterion: what is this saying?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:

The classical activation functions that were used early in the development of neural networks were the sign, sigmoid, and the hyperbolic tangent functions:
$$\Phi(v) = \text{sign}(v) \ \ \text{(sign function)} \\ \Phi(v) = \dfrac{1}{1 + e^{-v}} \ \ \text{(sigmoid function)} \\ \Phi(v) = \dfrac{e^{2v} - 1}{e^{2v} + 1} \ \ \text{(tanh function)}$$
While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I am having trouble understanding this part:

While the sign activation can be used to map to binary outputs at prediction time, its non-differentiability prevents its use for creating the loss function at training time. For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation.

I've read over this a number of times, but I still don't have a good idea of what it is saying (or at least the point it is trying to make). What is this actually saying? What is the point this is trying to make? Perhaps a more detailed explanation of what this is saying will clarify it for me.

sign is not continuous and not differentiable, so we cannot get a derivative of it and hence apply gradient descent optimization. Have you looked at this answer. Does it answer your question?

@ArayKarjauv Your answer partially clarifies things, but it doesn't explain how this all relates to the perceptron criterion.

Since the perceptron becomes nondifferentiable, we cannot compute the error and propagate it back through all the nodes. Instead, there are iterative algorithms to optimize the perceptron (see the last part of the answer).

@ArayKarjauv Eh, I don't really see how this addresses the part about the perceptron criterion.

What do you mean by the "perceptron criterion"?

@ArayKarjauv This part: "For example, while the perceptron uses the sign function for prediction, the perceptron criterion in training only requires linear activation."

sign is not continuous and not differentiable. Let's say it is defined as follows:
$$
\text{sing}(a) = \begin{cases}
       +1 & \text{if $a>0$}\\
      -1 & \text{otherwise}
    \end{cases}
$$
where $a = \textbf{w}^T\textbf{x}$ is a linear pre-activation function.
The graph will look as follows:

It is not continuous and, therefore, is not differentiable everywhere. Even if we take the derivative of such a function, it will be equal to zero, since the function is flat. That is, sign does not tell us how good our predictions are, it only says whether they were correct or not.
More formally, it does not return a gradient that represents the direction in which we should update the $a$ wrt $\textbf{w}$. Instead, we can only move the line (in 2d) trying to correctly classify as many data points as possible by manually changing each weight. If $a$ were non-linear, it would be impossible to fit a non-linear function by manually changing the weights, since changes in an individual weight would lead to unpredictable changes in the function.


text:
What would the "state space" and its Python implementation be for my simulation?
Context
I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. The remaining nodes will consist of one red agent, one blue agent, and some number of grey agents. The red agent and blue agent will be the only intelligent agents in the simulation, and their relationship is adversarial.
I'm representing the node data structure with the following Node class:
class Node():

    def __init__(self, id, team):
        self.id = id
        self.team = team
        self.agent = None
        self.edges = []

        if self.team == "green":
            self.agent = agents.GreenAgent()
        elif self.team == "blue":
            self.agent = agents.BlueAgent()
        elif self.team == "red":
            self.agent = agents.RedAgent()

The graph/network is instantiated to consist only of green agents. When the graph/network is instantiated, edges are added between the nodes based on some probability. So these edges represent the connections between green agents / nodes containing green agents, and they allow for interaction. These edges are undirected, and there are no loops (meaning nodes do not have edges to themselves).
I'm representing the graph/network data structure with the following Network class:
class Network():

    def __init__(self, number_of_green_nodes, probability_of_an_edge):
        self.number_of_green_nodes = number_of_green_nodes
        self.number_of_nodes_overall = number_of_green_nodes
        self.probability_of_an_edge = probability_of_an_edge
        self.green_network = [Node(id = i, team = "green") for i in range(1, number_of_green_nodes + 1, 1)]
        self.total_number_of_grey_nodes_inserted = 0

        all_pairs_of_nodes = combinations(self.green_network, 2)
        for node1, node2 in all_pairs_of_nodes:
            if random.random() < self.probability_of_an_edge:
                node1.create_edge(node2)

The green agents represent ordinary people. They will have two attributes: voting, which can be true or false, and uncertainty, which is some number that represents how uncertain the individual is about whether they are voting (say, -1 to +1, or 0 to 1). The green agents are not intelligent, and, in each round of the simulation, when it is their turn, they simply generically interact with each other. If it is the green agents' turn to interact, then all green agents with an edge/connection interact with each other, so there is no selectivity. But, with that said, they themselves do not initiate interaction with the red, blue, or grey agents. When the green agents interact with each other, their opinion on whether or not to vote, and their uncertainty, changes based on some calculation.
My green agent class is represented partially as follows:
class GreenAgent():

    def __init__(self):
        self.voting = random.random() < 0.5
        self.uncertainty = scipy.stats.truncnorm.rvs(0, 1)

    def interact_with_green_agent(self, other_agent_opinion, other_agent_uncertainty):

        if other_agent_opinion != self.voting:
            if other_agent_uncertainty < self.uncertainty:
                self.voting = not self.voting
                self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty/self.uncertainty)*(1 - self.uncertainty)
        else:
            if other_agent_uncertainty < self.uncertainty:
                self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty/self.uncertainty)*(self.uncertainty - other_agent_uncertainty)

The red agent and blue agent are added to the network after the network is initialised with the green agents and their edges. The red agent and blue agent do not have an edge to each other (they do not interact with each other), but they do have an edge to every single green agent/node (so, unlike between the green agents/nodes, it is not based on some probability), so the red agent and blue agent are initially able to interact with all green nodes/agents.
The red agent and blue agent are adversaries. The red agent's goal is to convince a majority of the green agents not to vote, and to have these green agents be more certain of not voting than uncertain (based on the green agents' uncertainty attribute). On the other hand, the blue agent's goal is to convince a majority of the green agents to vote, and to have these green agents be more certain of voting than uncertain (again, based on the green agents' uncertainty attribute. The red/blue agents cannot have their "opinion" changed with regards to voting, so the red agent will always have attribute voting as false, and the blue agent will always have attribute voting as true.
When it is the red agent's turn in the simulation, the red agent can, initially, interact with all of the green agents/nodes. And, just like the case of green-green interaction, the red agent will interact with all green agents that it has an edge/connection to – there is no selectivity. When it interacts with the green agents, it can select from 5 levels of propaganda, with each level of propaganda becoming increasingly radical/extreme. The trade-off here is that, the more radical/extreme the propaganda from the red agent, the greater its uncertainty level becomes when disseminating the propaganda, and the more likely they are to permanently alienate green agents (represented by me as the removal of an edge).
My red agent class is represented as follows:
class RedAgent():

    def __init__(self):
        self.voting = False
        self.uncertainty = None

    def messaging(self, potency):
        if potency == 1:
            self.uncertainty = random.uniform(0.0, 0.2)
        elif potency == 2:
            self.uncertainty = random.uniform(0.2, 0.4)
        elif potency == 3:
            self.uncertainty = random.uniform(0.4, 0.6)
        elif potency == 4:
            self.uncertainty = random.uniform(0.6, 0.8)
        elif potency == 5:
            self.uncertainty = random.uniform(0.8, 1.0)

And my part of the green agent class that deals with the red agent is as follows:
def interact_with_red_agent(self, other_agent_uncertainty, this_green_agent_node, red_agent_node):

    if other_agent_uncertainty < 0.2:
        if self.voting == False and other_agent_uncertainty < self.uncertainty:
            self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                    self.uncertainty - other_agent_uncertainty)
        elif self.voting == True and other_agent_uncertainty < self.uncertainty:
            self.voting = False
            self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty / self.uncertainty) * (
                        self.uncertainty - other_agent_uncertainty)
        if random.random() < other_agent_uncertainty/2:
            this_green_agent_node.remove_edge(red_agent_node)

    elif 0.2 < other_agent_uncertainty < 0.4:
        if self.voting == False and other_agent_uncertainty < self.uncertainty:
            self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                    self.uncertainty - other_agent_uncertainty)
        elif self.voting == True and other_agent_uncertainty < self.uncertainty:
            self.voting = False
            self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty / self.uncertainty) * (
                        self.uncertainty - other_agent_uncertainty)
        if random.random() < other_agent_uncertainty/2:
            this_green_agent_node.remove_edge(red_agent_node)

    elif 0.4 < other_agent_uncertainty < 0.6:
        if self.voting == False and other_agent_uncertainty < self.uncertainty:
            self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                    self.uncertainty - other_agent_uncertainty)
        elif self.voting == True and other_agent_uncertainty < self.uncertainty:
            self.voting = False
            self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty / self.uncertainty) * (
                        self.uncertainty - other_agent_uncertainty)
        if random.random() < other_agent_uncertainty/2:
            this_green_agent_node.remove_edge(red_agent_node)

    elif 0.6 < other_agent_uncertainty < 0.8:
        if self.voting == False and other_agent_uncertainty < self.uncertainty:
            self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                    self.uncertainty - other_agent_uncertainty)
        elif self.voting == True and other_agent_uncertainty < self.uncertainty:
            self.voting = False
            self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty / self.uncertainty) * (
                        self.uncertainty - other_agent_uncertainty)
        if random.random() < other_agent_uncertainty/2:
            this_green_agent_node.remove_edge(red_agent_node)

    elif 0.8 < other_agent_uncertainty < 1:
        if self.voting == False and other_agent_uncertainty < self.uncertainty:
            self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                    self.uncertainty - other_agent_uncertainty)
        elif self.voting == True and other_agent_uncertainty < self.uncertainty:
            self.voting = False
            self.uncertainty = self.uncertainty + (1 - other_agent_uncertainty / self.uncertainty) * (
                        self.uncertainty - other_agent_uncertainty)
        if random.random() < other_agent_uncertainty/2:
            this_green_agent_node.remove_edge(red_agent_node)

When it is the blue agent's turn in the simulation, the blue agent can interact with all of the green agents/nodes. And, again, the blue agent interacts with all green agents/nodes that it has an edge/connection to – so, again, there's no selectivity. Similar to the red agent, the blue agent can select from 5 levels of potency of messaging. The trade-off, however, is that the blue agent has an "energy level", and the more potent the message the blue agent chooses during its turn, the more likely that it loses energy. And, if the blue agent loses all of its energy, then it loses the simulation/game.
Furthermore, another option that the blue agent has during its turn is to insert a grey agent into the network. The grey agent, based on some probability, can either be an ally of the blue agent, helping to convince the green agents to vote, or can actually work against the blue agent, working for the red agent in further radicalising the green agents and convincing them not to vote. Depending on whether it is on the side of the blue agent or the red agent, the grey agent's interaction with the green agents can either mimic the red agent’s interaction abilities or the blue agent’s interaction abilities. However, the difference is that the grey agent does not suffer any of the consequences of taking action that the red/blue agent does during interaction: if the grey agent is an ally of the blue agent, then it can do the same move as the blue agent, without the blue agent losing any energy, but if the grey agent ends up working for the red agent, it can do the same move as the red agent, without the red agent having the chance of alienating any green agents. The choice by the blue agent to insert a grey agent into the network, instead of disseminating messaging, will take up the turn of the blue agent, regardless of whether the grey agent proves to work for the blue agent or against it.
My blue agent class is as follows:
class BlueAgent():

    def __init__(self):
        self.voting = True
        self.uncertainty = None
        self.energy_level = 10

    def messaging(self, potency):
        if potency == 1:
            self.uncertainty = random.uniform(0.8, 1.0)
            if random.random() > self.uncertainty:
                self.energy_level -= 1
        elif potency == 2:
            self.uncertainty = random.uniform(0.6, 0.8)
            if random.random() > self.uncertainty:
                self.energy_level -= 1
        elif potency == 3:
            self.uncertainty = random.uniform(0.4, 0.6)
            if random.random() > self.uncertainty:
                self.energy_level -= 1
        elif potency == 4:
            self.uncertainty = random.uniform(0.2, 0.4)
            if random.random() > self.uncertainty:
                self.energy_level -= 1
        elif potency == 5:
            self.uncertainty = random.uniform(0.0, 0.2)
            if random.random() > self.uncertainty:
                self.energy_level -= 1

And my part of the green agent class that deals with the blue agent is as follows:
def interact_with_blue_agent(self, other_agent_uncertainty):

    if other_agent_uncertainty < self.uncertainty:
        if self.voting != True:
            self.voting = True
        self.uncertainty = self.uncertainty - (1 - other_agent_uncertainty / self.uncertainty) * (
                self.uncertainty - other_agent_uncertainty)

My grey agent class is as follows:
class GreyAgent():

    def __init__(self):
        self.spy = random.random() < 0.5
        self.uncertainty = None

    def lifeline(self, potency):
        if potency == 1:
            self.uncertainty = random.uniform(0.8, 1.0)
        elif potency == 2:
            self.uncertainty = random.uniform(0.6, 0.8)
        elif potency == 3:
            self.uncertainty = random.uniform(0.4, 0.6)
        elif potency == 4:
            self.uncertainty = random.uniform(0.2, 0.4)
        elif potency == 5:
            self.uncertainty = random.uniform(0.0, 0.2)

    def misinformation(self, potency):
        if potency == 1:
            self.uncertainty = random.uniform(0.0, 0.2)
        elif potency == 2:
            self.uncertainty = random.uniform(0.2, 0.4)
        elif potency == 3:
            self.uncertainty = random.uniform(0.4, 0.6)
        elif potency == 4:
            self.uncertainty = random.uniform(0.6, 0.8)
        elif potency == 5:
            self.uncertainty = random.uniform(0.8, 1.0)

The red and blue agents have access to the number of green agents that are voting / not voting, but they do not have access to the green agents' uncertainty levels. Furthermore, the red and blue agents have access to what action the other (red/blue) agent took during their turn. And both agents also have access to the action the grey agent took.

Question
I am now trying to make the red agent and blue agent intelligent. My thought was to use reinforcement learning, whereby each agent learns from previous rounds of the simulation. In particular, I'm currently looking at applying Q-Learning. The problem is that I'm not familiar with reinforcement learning, and I'm trying to learn it by reading tutorials and going through the textbook Reinforcement Learning: An Introduction, second edition, by Sutton and Barto.
So my idea is to apply Q-Learning to train the red agent and blue agent individually. My understanding is that, in order to do this, I need to define the "environment," where the "environment" consists of the (1) "state space," (2) "actions," and (3) "rewards." The current problem is that, despite studying a ton of tutorials, I still can't figure out what my "state space" is supposed to be here (for the red agent and blue agent individually), and nor can I figure out what its Python implementation would be. What is the "state space" here, and what would its Python implementation be?

You may want to work on brevity to get answers. This is 2k words, according to wc.

I'll try to answer the general, conceptual question. For simplicity I assume that you just optimize one agent at a time.
Reinforcement Learning
Sutton and Barto mostly talks about small, fully-observable, discrete state-spaces. (Plus all the basic RL concepts.)
Most practical RL problems are not fully observable (in the Markov sense), meaning there is some decision-relevant hidden state that the agents can only guess from multiple past observations.
To stay within the RL concept, you want to design your "state" such that it summarizes everything the agent needs to know about the past interactions, in a "good enough" way to make optimal decisions. Then the agent itself doesn't need any memory, and the Markov property holds, and all the RL theory is applicable.
I think what this means in your case is that you need to summarize the past observable behaviour of the other agents somehow, and add that into the agent's state.
Black-Box Optimization
Alternatively, you could try framing this as black-box optimization problem. (In RL-terms: You optimize the cumulative return of each episode, not using the per-step rewards.)
The way to approach this is to ask yourself, "can I create a parametrizable model of a solution?", and then throw the parameters at one of the well-known optimization algorithms, like CMA-ES (for continuous parameters), or the Cross-Entropy Method (mostly for discrete parameters), or (if you are willing to do lots of parameter tuning) even a genetic algorithm.
If you can come up with a model that has only very few (say ~50) parameters, IMO just throw it at CMA-ES.
The question here is again how to represent the agent's inputs. Now you have the advantage that the agent can have any state/memory it wants, for example for tracking the other agent's history. (In the MDP setup, only the environment can have state/memory.) You could use a neural network, RNN, or even add a few to-be-optimized parameters that decide what goes into the memory, and what is taken out.
You may also be able to use a policy gradient (reinforcement learning) method, especially if you use a NN, and thus avoid black-box optimization and instead do gradient descent (usually faster, if applicable).


text:
Can we apply transfer learning between any two different CNN architectures?
There are many types of CNN architectures: LeNet, AlexNet, VGG, GoogLeNet, ResNet, etc. Can we apply transfer learning between any two different CNN architectures? For instance, can we apply transfer learning from AlexNet to GoogLeNet, etc.? Or even just from a "conventional" CNN to one of these other architectures, or the other way around? Is this possible in general?
EDIT: My understanding is that all machine learning models have the ability to perform transfer learning. If this is true, then I guess the question is, as I said, whether we can transfer between two different CNN architectures – for instance, what was learned by a conventional CNN to a different CNN architecture.

Do you know what "transfer learning" usually refers to in machine learning?

@nbro I’m studying it now. Apparently there are four categories of “transfer learning”, each involving a transfer of some different aspect of the neural network. Is that correct?

I only know one form of learning that people usually refer to as "transfer learning", but maybe there are others. Can you please link us to the article or resource you are reading that tells you that there are 4 different forms of transfer learning?

@nbro This isn’t the resource I am studying, but you can find the categories in this https://medium.com/georgian-impact-blog/transfer-learning-part-1-ed0c174ad6e7 article.

@nbro actually, I think that article lists 5 categories. Regardless, you’ll see what I mean.

No, transfer learning cannot be applied "between" different architectures, as transfer learning is the practice of taking a neural network that has already been trained on one task and retraining it on another task with the same input modality, which means that only the weights (and other trainable parameters) of the network change during transfer learning but not the architecture.
In my understanding, transfer learning is also only really effective in deep learning, but I could be wrong, considering that this Google search seems to yield some results.
You might otherwise be thinking of knowledge distillation, which is a related but different concept, where an already trained network acts as a teacher and teaches another network (a student network) with possibly a different architecture (or a machine learning model not based on neural networks at all) the correct outputs for a bunch of input examples.

"Transfer learning and machine learning are closely related. On one hand, the aim of transfer learning encompasses that of machine learning in that its key in- gredient is “generalization.” In other words, it explores how to develop general and robust machine learning models that can apply to not only the training data, but also unanticipated future data. Therefore, all machine learning models should have the ability to conduct transfer learning." Chapter 1.3 Relationship to Existing Machine Learning Paradigms of Transfer Learning by Yang, Zhang, Dai, and Pan.

I think the answer is yes, this problem is known as "Where To transfer", please refer to this paper Learning What and Where to Transfer to know what I am talking about, and correct me if I understand something wrongly.


text:
Can most of the basic machine learning models be easily represented as simple neural network architectures?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. In chapter 1.2.1 Single Computational Layer: The Perceptron, the author says the following:

Different choices of activation functions can be used to simulate different types of models used in machine learning, like least-squares regression with numeric targets, the support vector machine, or a logistic regression classifier. Most of the basic machine learning models can be easily represented as simple neural network architectures.

I remember reading something about it being mathematically proven that neural networks can approximate any function, and therefore any machine learning method, or something along these lines. Am I remembering this correctly? Would someone please clarify my thoughts?

See https://en.wikipedia.org/wiki/Universal_approximation_theorem

@pasabaporaqui Thanks for that. Chapter 1.5 The Secrets to the Power of Function Composition seems to say the following: "This basic idea is the essence of the universal approximation theorem of neural networks. In fact, the proof of the ability of squashing functions to approximate any function is conceptually similar to that of kernels at least at an intuitive level. However, the number of base functions required to reach a high level of approximation can be extremely large in both cases, potentially increasing the data-centric requirements to an unmanageable level. [...]

[...] For this reason, shallow networks face the persistent problem of overfitting. The universal approximation theorem asserts the ability to well-approximate the function implicit in the training data, but makes no guarantee about whether the function can be generalized to unseen test data."

I think the author refers to both different choices of activation function and loss. It is explained in more detail in chapter 2. In particular 2.3 is ilustrative of this point.
I don't think there is a relation between this argument and universal approximation theorems, which state that certain classes of neural networks can approximate any function in certain domains, rather than any learning algorithm.

Perhaps I am indeed mixing up two different things. Nonetheless, if a neural network is able to approximate any function, then, given sufficient data (however much "sufficient" may be), it should be able to accomplish anything any other learning algorithm, including the more "conventional" machine learning algorithms, can, right?

"should be able to accomplish anything any other learning algorithm [...] can": there is a subtlety though. Universal approximation only state existence of neural networks that approximate functions, but say nothing about algorithms for finding them.

But doesn't this comment agree with what I said? Since neural networks are able to approximate any function, then it should (assuming sufficient data) be able to approximate the same functions that any other learning algorithms do? Or am I still misunderstanding?

Let me put this another way: the functions defined by neural networks with specific weights and architectures set are different from the algorithms used to find them. Universal approximation theorems say: given a target function, a neural network that does not differ from it to a given degree of tolerance exists. However, a classical algorithm might find an acceptable function, whereas you might not be able to find such neural network. About the "sufficient data" question: UA theorems are valid for certain classes of NNs with infinite VC dimension, so infinite sample complexity.

Oh, I see what you're saying: The UA theorems state that, for any target function, there exists a neural network architecture that can approximate it; but that isn't necessarily the architecture that you're working with, and you might not be able to find a neural network architecture that can approximate said target function. Is that correct?

An architecture and specific weights for that architecture. Even if your architecture was correct, you still need to find the specific weights.


text:
Why doesn't the set $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ include $0$?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following:

The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$
Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes
particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ for misclassified points, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.

Equation 1.7 is as follows:

$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$

And figure 1.6 is as follows:

It is said that we are dealing with the case of binary classification, where $y \in \{ -1, +1 \}$. But the author claims that $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$, which doesn't include the case of $0$. So shouldn't the $\{ -2, +2 \}$ in $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ be $\{ -2, 0, +2 \}$?

The author is saying that $E(X)$ can never be zero for misclassified points, so I am not sure I understand your question/concerns. (Btw, I didn't really read that excerpt carefully).

@nbro But why can it never be zero for misclassified points? As I said, $y \in { -1, +1 }$, so I don't see why $E(X) = (y − \text{sign}{\overline{W} \cdot \overline{X} })$ cannot be zero.

It is important to note that the exact statement is the eqation given below can never be 0 for misclassified points in $ S^+$
$$ E(X) = (y - \text{sign}\{\overline{W} \cdot \overline{X}\}) $$
And $S+$ is defined as the set of all misclassified training points $X \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X})<0 $ which means that $y$ and $ (\overline{W} \cdot \overline{X}) $ will have different signs i.e if $y < 0$ then $(\overline{W} \cdot \overline{X}) > 0$ and vice versa.
Finally the condition on $S^+$ constraints the the $ E(X)$ to these values when $ y < 0$ and $(\overline{W} \cdot \overline{X}) > 0$
$$ E(X) = (-1 - \text{sign}(+ve)) = -2 $$
or when $ y > 0$ and $(\overline{W} \cdot \overline{X}) < 0$
$$ E(X) = (1 - \text{sign}(-ve)) = 2 $$


text:
Backpropogation rule for the output layer of a multi-layer network - What does the rule do in ambiguous cases?
This is the back-propogation rule for the output layer of a multi-layer network:
$$W_{jk} := W_{jk} - C \dfrac{\delta E}{\delta W_{jk}}$$
What does this rule do in the more ambiguous cases such as:
(1) The output of a hidden node is near the middle of a sigmoid curve?
(2) The graph of error with respect to weight is near a maximum or minimum?

Please format any mathematics using MathJax, rather than using images.

I assume you are considering a network where the activation function of the last layer is a sigmoid, so the output of your network is $$\tilde{y}=\sigma(W^{L}\cdot f(X, W^1, \dots, W^{L-1})),$$
where $X$ is the input vector, and $f$ is obtained by feeding the input to the network up to the layer $L-1$. Let's also call $Z:= W^{L}\cdot f(X, W^1, \dots, W^{L-1})$.
The error term is computed as $$E(y, \tilde{y})=E(y, \sigma(Z)),$$ where $y$ is the actual output. Let's get the derivative of the error with respect to the output of the last node (the input of the sigmoid)
$$\frac{\partial E}{z_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\tilde{y}}{\partial z_i}=\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z_i}.$$
The update rule is $$z_i =z_i - C\frac{\partial E}{z_i}= z_i - C\frac{\partial E}{\partial \tilde{y}}\frac{\partial\sigma}{\partial z_i}.$$
Now we can analyse your questions.

To be close to the middle of the sigmoid means that $z_i$ is close to $0$; moreover the derivative $\frac{\partial\sigma}{\partial z_i}$ reaches its maximum value when it is evaluated in $0$. This means that the term $\frac{\partial\sigma}{\partial z_i}$ gets "large" as $z_i$ approaches the center of the sigmoid, contributing more to the update of the weight. Of course it is difficult to say what happens in general, as the term $\frac{\partial E}{\partial \tilde{y}}$ is also in the expression and it is possible that this term gets really small (or big) for $z_i$ close to $0$. You just know that the term $\frac{\partial\sigma}{\partial z_i}$ is trying to push $z_i$ further away from $0$, so the idea should be that the closer to the center the larger the update.
To be close to an extremum point of the loss means that the derivative of the loss with respect to $\tilde{y}$ is close to $0$. Since $\frac{\partial E}{\partial \tilde{y}}$ is backpropagated in a multiplicative fashion, the rule of thumb is that the closer you are to an extremum the smaller the updates get. Though, as above, is kind of difficult to say what will happen when updating a general node, as some of the term in the multiplication can be very large, making the update large even if $\frac{\partial E}{\partial \tilde{y}}$ is small.

For instance, what happens if you are close to the center of the sigmoid but also close to an extremum of the loss? You will have a multiplication of $2$ terms, one trying to make the update small and the other trying to make the update large, and what matter are the orders of magnitude involved.
In conclusion, the rules of thumb are as in the points 1. and 2., but they are no guarantee that you won't find any special cases.


text:
What is emperical distribution in MLE?
I was reading the book Deep Learning by Ian Goodfellow. I had a doubt in the Maximum likelihood estimation section (Pg 131). I understand till the Eq 5.58 which describes what is being maximized in the problem.
$$
 \theta_{\text{ML}} = \text{argmax}_{\theta} \sum_1^m \log(p_{\text{model}}(x^{(i)};\theta))
$$
However the next equation 5.59 restates this equation as:
$$
\theta_{\text{ML}} = \text{argmax}_{\theta}  E_{x \sim \hat{p}_{\text{data}}}(\log(p_{\text{model}}(x;\theta))
$$
where $$\hat{p}_{\text{data}}$$ is described as the empirical distribution defined by the training data.
Could someone explain what is meant by this empirical distribution? It seems to be different from the distribution parametrized by theta as that is described by $$ p_{\text{model}} $$

The idea behind this kind of reasoning is that there is a "true" distribution (unknown to us, mere mortals) and that the data is generated following this distribution. But what we don't really know the shape of the distribution, all we know is the distribution of the data that we have. This is called the empirical distribution.   Let's see a simple example to illustrate the point.
Let's consider a die. Each number is equally likely to show up if we throw the die, so the true underlying distribution is uniform over the set $\{1, 2,\dots6\}$. Now let's say you ask your friend to throw the die 60 times, what you will see is likely something close to uniform over the set $\{1, 2,\dots6\}$ (not really uniform though, as that would be highly unlikely). This distribution is the empirical one, and as you collect more and more samples it will converge to the actual underlying distribution.
In your case what happened is the following:
$x_1,\dots, x_m$ is your sample (in the example above, the $60$ numbers that you see as your friend throws the die). This sample defines a distribution, $\hat{p}_{data}$. In the example above, $\hat{p}_{data}$ would likely be close to the uniform distribution over $\{1, \dots, 6\}$. Now you can think about the sum over $x_1\dots, x_m$ of $\log(p_{model}(x^{(i)};\theta))$ as the average of $\log(p_{model}(x;\theta))$, where $x$ is drawn according to $\hat{p}_{data}$.
Let me make another example with some actual numbers. Let's say you toss a fair coin $7$ times and you see $$\{H, H, H, T, H, T, H\}.$$
The empirical distribution is $\mathbb{P}(H)=5/7$, $\mathbb{P}(T)=2/7$. So if you compute the expectation $$\mathbb{E}_{x \sim \hat{p}_{data}}[\log(p_{model}(x;\theta)]$$ you get
$$\log(p_{model}(H;\theta)\cdot\mathbb{P}(H) +\log(p_{model}(T;\theta)\cdot\mathbb{P}(T) = \\ \frac{5}{7}\log(p_{model}(H;\theta) +\frac{2}{7}\log(p_{model}(T;\theta)),$$
which is what you will get if you compute the first sum you wrote.


text:
$\frac{P(x_1 \mid y, s = 1) \dots P(x_n \mid y, s = 1) P(y \mid s = 1)}{P(x \mid s = 1)}$ indicates that naive Bayes learners are global learners?
I am currently studying the paper Learning and Evaluating Classifiers under Sample Selection Bias by Bianca Zadrozny. In section 3. Learning under sample selection bias, the author says the following:

We can separate classifier learners into two categories:

local: the output of the learner depends asymptotically only on $P(y \mid x)$
global: the output of the learner depends asymptotically both on $P(x)$ and on $P(y \mid x)$.

The term "asymptotically" refers to the behavior of the learner as the number of training examples grows. The names "local" and "global" were chosen because $P(x)$ is a global distribution over the entire input space, while $P(y \mid x)$ refers to many local distributions, one for each value of $x$. Local learners are not affected by sample selection bias because, by definition $P(y \mid x, s = 1) = P(y \mid x)$ while global learners are affected because the bias changes $P(x)$.

Then, in section 3.1.1. Naive Bayes, the author says the following:

In practical Bayesian learning, we often make the assumption that the features are independent given the label $y$, that is, we assume that
$$P(x_1, x_2, \dots, x_n \mid y) = P(x_1 \mid y) P(x_2 \mid y) \dots P(x_n \mid y).$$
This is the so-called naive Bayes assumption.
With naive Bayes, unfortunately, the estimates of $P(y \mid x)$ obtained from the biased sample are incorrect. The posterior probability $P(y \mid x)$ is estimated as
$$\dfrac{P(x_1 \mid y, s = 1) \dots P(x_n \mid y, s = 1) P(y \mid s = 1)}{P(x \mid s = 1)} ,$$
which is different (even asymptotically) from the estimate of $P(y \mid x)$ obtained with naive Bayes without sample selection bias. We cannot simplify this further because there are no independence relationships between each $x_i$, $y$, and $s$. Therefore, naive Bayes learners are global learners.

Since it is said that, for global learners, the output of the learner depends asymptotically both on $P(x)$ and on $P(y \mid x)$, what is it about $\dfrac{P(x_1 \mid y, s = 1) \dots P(x_n \mid y, s = 1) P(y \mid s = 1)}{P(x \mid s = 1)}$ that indicates that naive Bayes learners are global learners?

EDIT: To be clear, if we take the example given for the local learner case (section 3.1. Bayesian classifiers), then it is evident:

Bayesian classifiers compute posterior probabilities $P(y \mid x)$ using Bayes' rule:
$$P(y \mid x) = \dfrac{P(x \mid y)P(y)}{P(x)}$$
where $P(x \mid y)$, $P(y)$ and $P(x)$ are estimated from the training data. An example $x$ is classified by choosing the label $y$ with the highest posterior $P(y \mid x)$.
We can easily show that bayesian classifiers are not affected by sample selection bias. By using the biased sample as training data, we are effectively estimating $P(x \mid y, s = 1)$, $P(x \mid s = 1)$ and $P(y \mid s = 1)$ instead of estimating $P(x \mid y)$, $P(y)$ and $P(x)$. However, when we substitute these estimates into the equation above and apply Bayes' rule again, we see that we still obtain the desired posterior probability $P(y \mid x)$:
$$\dfrac{P(x \mid y, s = 1) P(y \mid s = 1)}{P(x \mid s = 1)} = P(y \mid x, s = 1) = P(y \mid x)$$
since we are assuming that $y$ and $s$ are independent given $x$. Note that even though the estimates of $P(x \mid y, s = 1)$, $P(x \mid s = 1)$ and $P(y \mid s = 1)$ are different from the estimates of $P(x \mid y)$, $P(x)$ and $P(y)$, the differences cancel out. Therefore, bayesian learners are local learners.

Note that we get $P(y \mid x)$. However, in the global case, it is not clear how we get $P(x)$ and $P(y \mid x)$ (as is required for global leaners) from $\dfrac{P(x_1 \mid y, s = 1) \dots P(x_n \mid y, s = 1) P(y \mid s = 1)}{P(x \mid s = 1)}$.


text:
How should we interpret this figure that relates the perceptron criterion and the hinge loss?
I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.2 Relationship with Support Vector Machines says the following:

The perceptron criterion is a shifted version of the hinge-loss used in support vector machines (see Chapter 2). The hinge loss looks even more similar to the zero-one loss criterion of Equation 1.7, and is defined as follows:
$$L_i^{svm} = \max\{ 1 - y_i(\overline{W} \cdot \overline{X}_i), 0 \} \tag{1.9}$$
Note that the perceptron does not keep the constant term of $1$ on the right-hand side of Equation 1.7, whereas the hinge loss keeps this constant within the maximization function. This change does not affect the algebraic expression for the gradient, but it does change which points are lossless and should not cause an update. The relationship between the
perceptron criterion and the hinge loss is shown in Figure 1.6. This similarity becomes particularly evident when the perceptron updates of Equation 1.6 are rewritten as follows:
$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{(\overline{X}, y) \in S^+} y \overline{X} \tag{1.10}$$
Here, $S^+$ is defined as the set of all misclassified training points $\overline{X} \in S$ that satisfy the condition $y(\overline{W} \cdot \overline{X}) < 0$. This update seems to look somewhat different from the perceptron, because the perceptron uses the error $E(\overline{X})$ for the update, which is replaced with $y$ in the update above. A key point is that the (integer) error value $E(X) = (y − \text{sign}\{\overline{W} \cdot \overline{X} \}) \in \{ −2, +2 \}$ can never be $0$ for misclassified points in $S^+$. Therefore, we have $E(\overline{X}) = 2y$ for misclassified points, and $E(X)$ can be replaced with $y$ in the updates after absorbing the factor of $2$ within the learning rate.

Equation 1.6 is as follows:

$$\overline{W} \Leftarrow \overline{W} + \alpha \sum_{\overline{X} \in S} E(\overline{X})\overline{X}, \tag{1.6}$$
where $S$ is a randomly chosen subset of training points, $\overline{X} = [x_1, \dots, x_d]$ is a data instance (vector of $d$ feature variables), $\overline{W} = [w_1, \dots, w_d]$ are the weights, $\alpha$ is the learning rate, and $E(\overline{X}) = (y - \hat{y})$ is an error value, where $\hat{y} = \text{sign}\{ \overline{W} \cdot \overline{X} \}$ is the prediction and $y$ is the observed value of the binary class variable.

Equation 1.7 is as follows:

$$L_i^{(0/1)} = \dfrac{1}{2} (y_i - \text{sign}\{ \overline{W} \cdot \overline{X_i} \})^2 = 1 - y_i \cdot \text{sign} \{ \overline{W} \cdot \overline{X_i} \} \tag{1.7}$$

And figure 1.6 is as follows:

Figure 1.6 looks unclear to me. What is figure 1.6 showing, and how is it relevant to the point that the author is trying to make?

I was also confused about this specific part of the book and came accross the following video explaining hinge loss. Hope it might help: https://www.youtube.com/watch?v=PM2MSAYmzXM

Figure 1.6 is depicting that the hinge-loss used in SVMs (Eqn 1.9):
$$L_i^{svm} = \text{max}\{1 - y_i (\overline{W} \cdot \overline{X}_i, 0\} \tag{1.9}$$
is a shifted version (+1 to the right) of the loss function (Eqn 1.8) used to optimize the perceptron criterion (see footnote (1)):
$$L_i = \text{max}\{-y_i (\overline{W} \cdot \overline{X}_i), 0\} \tag{1.8}$$
Using Figure 1.6 and the accompanying discussion in Section 1.2.1.2, the author (Charu C. Aggarwal) is trying to show that if you make the implicit training process of the perceptron explicit (with an objective function, in his case, a loss function defined by Eqn 1.8), then one realises (in his words):

that the perceptron is fundamentally not very different from well-known machine learning algorithms like the support vector machine in spite of its different origins.

Footnotes
(1) The perceptron criterion was introduced in Section 3.5.1 of Neural Networks for Pattern Recognition by Christopher Bishop as a continuous, piecewise linear error function given by:
$$E^{\text{perc}}(\bf{w}) = - \sum_{\phi^n \in \mathcal{m}} w^T (\phi^n t^n)$$
Bishop suggested that this objective function is minimised when training a perceptron. Aggarwal references this work as an example of how the perceptron training process implicitly minimises this error function, using it as a segue into his own derivation of Eqn 1.8 before encouraging the reader to check that the gradient of Eqn 1.8 leads to the perceptron update (i.e. $\overline{W} \Leftarrow \overline{W} - \alpha \nabla_{W} L_i$).

I can't fully explain the part because I forgot what it talks about.
However, regarding the hinge loss, it is basically allowing your SVM to tolerate misclassifications without increasing the cost function.
For example, you give someone 1 dollar or 1 euro. You can forgive them, you tolerate it. Your hinge loss is 0 for lending someone 1d dollar. However, if you give them 10 dollars or 100, you will ask them to refund you ASAP because you can't tolerate that much loss!

Hi. Thanks for trying to contribute, but you should at least try to address the actual question.

Your information regarding the hinge loss is interesting, but, as @nbro said, it doesn't answer my question.


text:
How would we get a good estimation of the asymptotic performance of machine learning algorithms?
The following question is from the webbook Neural Networks and Deep Learning by Michael Nielson:

How do our machine learning algorithms perform in the limit of very large data sets? For any given algorithm it's natural to attempt to define a notion of asymptotic performance in the limit of truly big data. A quick-and-dirty approach to this problem is to simply try fitting curves to graphs like those shown above, and then to extrapolate the fitted curves out to infinity. An objection to this approach is that different approaches to curve fitting will give different notions of asymptotic performance. Can you find a principled justification for fitting to some particular class of curves? If so, compare the asymptotic performance of several different machine learning algorithms.

The ability to mimic complex curves and fit to the data points comes due to the non-linearity used, since, had we only used a linear combination of weights and biases, we would not have been able to mimic these. Now the output depends a lot on our choice of non-linearity. Suppose we have a model. It overfits and we get an order 5 polynomial, while in another case it underfits and we get a linear model. So how would we get a good estimation of the asymptotic performance, as questioned by the author?

To be honest, I still don't understand your question. What do you mean by "this" in "How can I think about this..."? Moreover, it's not clear to me if by "time complexity" you're referring to the time complexity of the training algorithm (e.g. gradient descent) or something else. Can you clarify that?

@nbro So I come from a CS background and recently started DL. I've been accustomed to using time and space complexity to analyze efficiency of algorithms. I'm not really clear on how the author wants us to think of asymptotic performance so a bit of that vagueness reflects in the question

To me, it seems that you're mixing the idea of a model (e.g. a neural network) with the idea of the time complexity of a training algorithm (e.g. gradient descent), so it would be better that you first understand that there's a difference between a neural network and the algorithm that trains it using the data. Once you understand that, I think you will have a clearer idea of what's going on and that you can still compute the time complexity of an algorithm.  So, before proceeding, I would suggest that you read about "how gradient decent is used to train a neural network"

Just to give you an example that the notion of time complexity of an algorithm applies to the context of machine learning as well, take a look at this answer that I wrote, where I show how you can compute the time complexity of the forward pass of a neural network, i.e. the algorithm that produces the output of the neural network given the input, which is known as "forward pass".

Once you have gone through that, I would suggest that you edit this post to clarify what your ultimate specific question is. I think your doubt is interesting, but your question just need to be clarified. Maybe your ultimate question is: "How do we measure the time complexity of a machine learning algorithm?". If that's the case, please, just edit your post to leave that question. If not, again, please, tell us what your specific question is.

Thanks a lot @nbro , I'll read the reference and update the question in a while :)


text:
What do RNN, LSTM, and GRU layers do in Tensorflow?
I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, etc.
Then I am trying to implement them with tensorflow, and even after reading the documentation, I am unable to connect the layers' API with my theoretical understanding of the operations. For example, take the following simple code:
import tensorflow as tf # tensorflow 2.5.0
inputs=tf.random.normal(shape=(32, 10, 8))
lstm = tf.keras.layers.LSTM(units=4, return_sequences=True, return_state=True)
outputs=lstm(inputs) # Call the layer, gives a list of three tensors
lstm.trainable_weights # Gives a list of three tensors

So what exactly is the layer doing here based on the input it receives and the weights that were initialised randomly?
If I am to implement the layer's operation myself, how do I do that?
The Google and Keras documentation contain a lot of example code, but not really explanations of the internal mathematical operations. So any help in this area, or any reference that explains the mathematical operations (not in general, but what's happening in the Tensorflow layer) would be greatly appreciated.
I have the exact same question regarding RNN and GRU layers too.

I think what you're asking for here is too much to expect in an answer. I'd also be curious to know the details of the mathematical implementation in Tensorflow; and this is the type of stuff I'd expect documentation for. Tensorflow is open-source software, so you can find all of the details here https://github.com/tensorflow/tensorflow ; are you sure there isn't documentation here for what you're looking for? At worst, since it is open-source, you can look through the code yourself (which is likely correct and of high quality, since it is a popular and well-regarded open-source project).

I have to agree with @ThePointer .  In fact, to find out the details of what these layers "do", you have two accessible options:  The documentation and the source code.  The code that implements the LSTM class is completely open as part of the tensorflow package.  Perhaps the right course is to look at those two things and then bring specific additional questions back here.

Thanks. The documentation as I see it (https://keras.io/api/layers/recurrent_layers/simple_rnn/) does not give much hint about it. So the last option is to go through the tensorflow source code. I was not expecting a complete answer here, but was wondering whether there are any other resources on the web apart from going through the source code myself.


text:
What does 'downsampling' and 'upsampling' mean in coarse-to-fine segmentation?
The paper here in section 2.1 Coarse-to-fine prediction:

To increase the field of view presented to the CNN and reduce the
redundancy among neighboring voxels, each image is downsampled by a factor of 2. The resulting prediction maps are then resampled back to the original resolution using nearest-neighbor interpolation.

What does it actually mean to downsample by a factor of 2?
If I have an image size of $256 \times 256 \times 170$, and if I downsample it by a factor of 2, then will it result in an image of size $128 \times 128 \times 85$?
Similarly, would upsampling/resampling be the opposite interpolation method, getting back to the original size of $256 \times 256 \times 170$?

What is actually the downsampling of 2 mean?
If I have an image size of 256x256x170 and if I downsample it by a factor of 2, it will result in an image of size 128x128x85?

Yes, that is correct.

Similarly, upsampling or resampling is the opposite interpolation method to original size 256x256x170?

Yes, correct result again. Resampling is a more general term and describes resizing an image (or other grid-based signal) using an automatic rule which could be enlarging or reducing the size.
What resampling does not mean in digital signal processing is going back to the original source and taking a second sample. That might be a reasonable interpretation of the word when coming across it for the first time, but it is not correct here.
Just saying that a signal is resampled does not give enough information to say exactly what was done numerically to the signal, because it does not specify the rule of what happens with the lost (in case of downsampling) or missing (in case of upsampling) information. There are a few different ways to perform the resize, and different kinds of interpolation. It looks like the paper does give more information here, but you may need to read more or inspect the authors' code to be certain - for instance "nearest neighbour interpolation" probably involves looking at values of the six adjacent voxels to each new one you create, and taking the mean value of the ones that exist in the original grid (some will not, they will be other new cells), but there are a few different ways to manage that process.

@Neuk Slater, they use:
dx = 2  dy=2  dz=2   L = L[::dx, ::dy, ::dz]  I = I[::dx, ::dy, ::dz]
to downsample the Image(I) and Label(L). But resampling back changes the label values. I am finding it hard to print the codes here.

Moreover, in section 2.2 second paragraph, "...the 3D FCN is trained on images of the lowest resolution in order to capture the largest amount of context, downsampled with a factor of ds1 = 2S and optimized using the Dice loss L1...
In the next level, we use the predicted segmentation maps as a second input channel to the 3D FCN while learning from the images at a higher resolution, downsampled by a factor of ds2 =ds1 /2, and optimized using Dice loss L2."
What does it mean by downsampling again by ds2? Shouldn't they upsample prediction by 2 or to original size of high-resolution patch?

@banikr I am not sure of the answers to your follow-up questions, and don't think they would fit in a comment anyway. Could you please ask a new question on the site. You could link this one to help explain what you have already understood.

Check the post: https://ai.stackexchange.com/questions/28067/down-sampling-confusion-in-coarse-to-fine-segmentation @Neil Slater


text:
Is this ExpectiMinimax Tree correctly drawn?
I need help with ExpectiMinimax problem:

Start a game.
The first player flips a coin.
The second player flips a coin.
The first player decides if he wants to flip another coin.
The second player decides if he wants to flip another coin.
Game over.

The winner is the player who has earned more points.
The points are computed as follows:

If the player flips ones and got the head - 1 point.
If the player flips ones and got the tail - 2 points.
If the player flips twice and got 2 heads - 4 points.
If the player flips twice and got 2 tails - 4 points.
If the player flips twice and got one head and one tail - 0 points.

I need to draw the ExpectiMinimax tree associated with this problem, and write the value of each node.
Did I draw the tree properly?

No, your tree is not an accurate representation of the problem.
Hint: Consider the case where both players flip an H on their first coin. Player 1 decides not to flip a second coin. Is the game over?


text:
How to detect multiple playing cards of the same class with a neural network?
I want to train an AI to detect the class (i.e. suit and rank) of playing cards. Playing cards from different decks may use slightly different shapes or colors to represent these attributes, and I want the system to work across many decks. I bought many different decks, scanned and labeled them. Next up would be to create training data with an augmentation library. I found two examples of how other people did that:

Image detection using YOLO algorithm and poker cards
Playing card detection with YOLO

The problem is that I want my AI to be able to detect multiple cards of the same class in one picture. In the examples above, they put a label on the top left corner of a card. This makes sense since it is a very good indicator of what class the card is in. Unfortunately, every card has two of these labels.
I think their solution returns "detected" if any instance of a given label is found. But many cards with the same label could be in the same picture, and I am not sure if I can detect the quantity of the class with this solution. For example, if there are 2 Ace of Clubs in a picture, I would like my system to output "2", rather than "detected".
Do you think it is feasible to mark the whole card, prepare the data accordingly, and train an AI that detects the count as well?

Although it was not crystal clear, we'll assume that by, "Multiple cards of the same class in one picture," is meant that multiple cards of identical suit and rank will be grouped together in the same example image but each card in the image will be selected from a unique deck.
That arrangement would only be fruitful if the objective of training was to classify, flag, or otherwise analyze the same kind of groupings after training. Otherwise, it would likely be most productive and efficient to first devise a way to divide up the images by deck so that the focus of learning is detection of the three dimensions or perhaps just the first two, depending on the intended use of the trained network.

Rank
Suit
Style