text:
Does the term "data augmentation" imply increasing the training dataset?
I have a manuscript that has been reviewed and one of the reviewers commented on my use of the term " data augmentation", saying that it might not be the appropriate term in my case (explained below).
I collected a large dataset of short audio files which are used to train a Convolutional Neural Network. Before being used as model input, each audio file is processed through a pipeline that mixes it with other audio files, changes some of the sound properties (SNR ratio, distorting the audio ...) and finally converts it into a mel-spectrogram. I only modify the existing file and I do not increase my training dataset but I refer to this processing as "data augmentation".
I did not find any definitive definition of what is data augmentation. For instance, Salamon and Bello, 2016 define data augmentation as

the application of one or more deformations to a collection of
annotated training samples which result in new, additional training
data

However, it appears to me that the increase in the training dataset is only a byproduct of the data augmentation.
In any case, I would really appreciate it if you could confirm or not my use of "data augmentation" and I would be grateful if you can provide a reference for this.

augmentation is like not changing it but adding something else.  Not the same as new data, but the original is preserved.  Data transformation might be a viable alternative word.

I'm not familiar with any "authoritative" single definition somewhere, or not sure who used the term first, but I would personally indeed agree with the reviewer you mention. In fact I've made similar comments myself as a reviewer of a paper once (but it was in an entirely different field, nothing related to audio at all, so I probably wasn't the reviewer in question). Personally, when I think of data augmentation, the first example that jumps to mind is adding rotated/reflected versions of images to the training data.
A quick google search suggests that, for example, Shorten and Khoshgoftaar (2019) similarly associate data augmentation with increases in training data in their survey paper on data augmentation for images.
What you describe sounds more like "data preprocessing" rather than data augmentation to me, but I'm not 100% sure exactly what the purpose of your modifications is (which I suppose might change my opinion). Data preprocessing can be almost any procedure that is applied to your original data, like... feature selection, converting images to grayscale if you expect colour to be irrelevant to your task, downscaling images if they're too big and you don't mind decreasing the resolution, etc.

This might indeed be data processing rather than data augmentation. My confusion came from the fact that I am applying all these transformation to add variety to my dataset (as I have 5 days of audio with all files very alike).


text:
Why do Transformers have a sequence limit at inference time?
As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is set, and to allow batching, all sequences smaller are padded.
However, after a Transformer is trained, and we want to run it on a single sequence at inference time, the computational costs are far less than training. Thus, it seems reasonable that I would want to run the transformer on a larger input sequence length during inference time. From a technical perspective, this should be feasible.
I keep reading online that a Transformer cannot be run on a sequence size larger than the one seen during training. Why is this? Is it because the network weights will be unfamiliar with sequences of this length? Or is it more fundamental?

Transformer models have limited sequence length at inference time
because of positional embeddings. But there are workarounds.
Self-attention in transformer does not distinguish the order of keys/values,
it works as if the sequence is a bag of words.
So to expose the sequence order to the model, one typically adds an extra
"positional embedding" vector to each input token embedding.
This extra positional information then allows the model to construct primitives like "attention head that looks at previous token".
There are several variants of how to do this exactly.
One way is to use an extra trainable parameter matrix [L,M]
where L is maximum input length and M is model dimension.
After training a model this way, it simply becomes impossible
to embed tokens with position > L. So we get a hard limit for which there is no workaround.
Another way is to use a non-trainable matrix [L,M]
initialized with a specially crafted set of sinusoids
(original "attention is all you need" paper does this).
A model trained this way would not have a length limit,
because this matrix of sinusoids you can extend to arbitrary size.
Yes another way is to use "relative positional encoding".
With this, for each pair of tokens you take relative position,
which would be in range [-L ... +L], and embed that instead.
Then you inject this vector into attention layer in the right way
(section 3.3 in https://arxiv.org/pdf/1901.02860.pdf).
Now you still kind of have a limit of [-L, L].
But you can always "clip" relative position to this range,
and pretend that all pairs of tokens at distance >= L
have the same relative position ("very far from each other").
And this allows to run inference with longer inputs.

What are the relative positions relative to?!

Thanks for this answer! I had thought positional embeddings were the problem but.I couldn't find anywhere where that was stated explicitly. Thanks for clarifying

One follow up question: even if we are using the sinusoid positional embeddings, would there still be a problem. Granted it is easy to extend the embeddings arbitrarily,  but still since the network has never seen positions beyond the training length limit, wouldnt the network weights be unfamiliar with that part of the function

@HappyFace: "relative position" means distance between query and key in self-attention

To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length.
However, the computational complexity comes from the matrix operations in the attention layer. Those are not trained; There are no trained parameters in the attention mechanism (see figure 2 and equation (1) in Vaswani et al). So, those have to be computed during inference as well.
Another challenge would be the output layer. That layer is a regular feedforward layer and thus has a fixed input size; That is, you cannot add new parameters during inference.
Of course, there is a caveat to this; There are now transformers now that allow recurrence, such as Transformer-XL and Memformer. These do, in a way, allow longer input sequences than "max sequence length".

I think the part about the output layer is incorrect. Usually the output layer is repeated for thesequence length, so that should not be an issue. Also the point about the matrix operations does not answer why transfoermers CANNOT be used for sequences larger than what they were trained on.

@chessprogrammer It's not strictly necessary to repeat it for the sequence length; It helps with training, but the output is in principle a single token (especially if you want to avoid teacher forcing, although that's of course still ongoing research). As Vaswani say in the first paragraph of section 3 in their paper: "Given z, the decoder then generates an output
sequence (y1, ..., ym) of symbols one element at a time."

@chessprogrammer I only mentioned the matrix complexity point because I interpreted your question as being premised on the quadratic complexity being the reason for the limitation.

If you used something like Rotary Positional Embedding -- such as the LLama3 family of models -- then technically speaking, you could do inference with arbitrary long sequence (assuming you have enough computational resources). The transformer itself does not have any inherent limitation.
The real question is more like: will it work well, if no such sequence has been ever seen in training? Will it understand how to interpret a token -2000 from current position, if in training, it has seen at most tokens -1000 from current position?
I tried to overwrite the max_length configuration in llama3-7b, just for fun, and got this warning:
This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (2048). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.
It is common to use shorter sequences on inference, but not longer.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Which of the following probability distribution is generating an iid dataset?
Let $X_1, X_2$ be two discrete random variables. Each random variable takes two values: $1, 2$
The probability distribution $p_1$ over $X_1, X_2$ is given by
$$p_1(X_1=1, X_2 = 1) = \dfrac{1}{4}$$
$$p_1(X_1=1, X_2 = 2) = \dfrac{1}{4}$$
$$p_1(X_1=2, X_2 = 1) = \dfrac{1}{4}$$
$$p_1(X_1=2, X_2 = 2) = \dfrac{1}{4}$$
The probability distribution $p_2$ over $X_1, X_2$ is given by
$$p_2(X_1=1, X_2 = 1) = \dfrac{8}{16}$$
$$p_2(X_1=1, X_2 = 2) = \dfrac{4}{16}$$
$$p_2(X_1=2, X_2 = 1) = \dfrac{3}{16}$$
$$p_2(X_1=2, X_2 = 2) = \dfrac{1}{16}$$
Suppose $D_1, D_2$ are the datasets generated by $p_1, p_2$ respectively.
Then which dataset can I call an iid? I am guessing as $D_1$ since we can prove the random variables are independent and are identically distributed and for $D_2$, iid does not hold.

$\underline{\text{ For }D_1}$
Identical : $p_1(X_1 = x_1) = p_1(X_2 = x_2)= \dfrac{1}{2} \text{ where } x_1 = x_2  \in \{1, 2\}$
Independent: $p_1(X_1 = x_1,X_2 = x_2) = \dfrac{1}{4} = p_1(X_1 = x_1) p_1(X_2 = x_2) \text{ for  } x_1, x_2 \in \{1, 2\}$

We can show that random variables $X_1, X_2$ are not iid if we consider $p_2$.
Is the iid I am discussing is different from making an iid of a dataset as answered here? If not, where am I going wrong?

The probability distributions used to describe i.i.d. vs non-i.i.d. are normally considered per example, whilst here it looks like you want to consider a pd over the whole dataset at once. Whilst that is possible, that is an unusual approach, and I am not quite sure what insight you hope to gain with it. Is that really what you want to ask? The pd examples you used in the linked question were clearly per example

@NeilSlater It is difficult for me to understand  what you mean by per example related probability distributions. I can image only a single pd for a whole dataset. If the random variables obey independence and identical distribution properties as $p_1$ in $D_1$, then I can recognize it as iid else non-iid as $D_2$ because of $p_2$. If possible, please recommend some material so that I can understand in detail what you are saying.

Sorry I don't understand your comment or question. I hope someone can help answer your question.

A sequence of $n$ random variables $z_{1:n} = z_1, z_2, \dots, z_n$ is i.i.d. if

they are identically distributed, i.e. each random variable $z_i$ has the same distribution
the joint distribution of all of them is just the product of the marginal distributions of each r.v.

So, let's imagine a thought experiment in which we throw a coin $n$ times, so you get a sequence of results, but you still don't know what those actual results are (i.e. you don't know whether $z_i$ is heads or tails), because this is just a thought experiment, so we're still talking about random variables and probability distributions and not datasets.
You might think: well, we're sampling $n$ times from the same distribution because we have only 1 coin, so we have only one r.v. In reality, you can think that each throw is associated with a different r.v. $z_i$, but that all these r.v.s, $z_1, z_2, \dots, z_n$, have the same probability distribution, for example, a Bernoulli with the same $p$ (the parameter of the Bernoulli). Now, let's say that $p = 0.5$. This means that, for a single coin toss, there's a 50% chance that the coin will be tails and a 50% chance it lands on heads. This does not have to be the case. In fact, we could also have a weird coin that prefers to land on heads, so let's say that $p = 0.7$, which is the probability it lands on heads. That's fine, and the sequence of random variables $z_{1:n}$ can still be i.i.d. How is this possible?
What's the Bernoulli pmf?
$$f_\text{marginal}(k_i;p)=p^{k_i}(1-p)^{1-k_i},$$
where $k_i\in \{0,1\}$.
So, each $z_i$ has this Bernoulli pmf, with the same $p$. This is the identically distributed part of iid.
For simplicity, let $n = 2$, so $z_{1:n} = z_{1:2} = z_1, z_2$.
So, if $z_1$ and $z_n$ are independent, we have that their joint is just the product of their marginals
\begin{align}
f_\text{joint}(k_1, k_2;p)
&=(p^{k_1}(1-p)^{1-k_1}) (p^{k_2}(1-p)^{1-k_2}) \\
&=p^{k_1} p^{k_2} (1-p)^{1-k_1} (1-p)^{1-k_2} \\
&=p^{k_1 + k_2} (1-p)^{2-k_1 - k_2} \\
\end{align}
where $k_1 \in \{0,1\}$ is the outcome for $z_1$ and $k_2 \in \{0,1\}$ is the outcome for $z_2$.
So, as before, let's say that $p= 0.7$, then that can be written as
$$
f_\text{joint}(k_1, k_2; 0.7) =
(0.7^{k_1}(1-0.7)^{1-k_1}) (0.7^{k_2}(1-0.7)^{1-k_2})
$$
If $k_1=0$ and $k_2 = 0$, we have
\begin{align}
f_\text{joint}(0, 0;0.7)
&= (0.7^0(1-0.7)^1) (0.7^0(1-0.7)^1) \\
&= 0.3 * 0.3 = 0.09
\end{align}
If $k_1=1$ and $k_2 = 1$, we have
\begin{align}
f_\text{joint}(1, 1;0.7)
&= (0.7^1(1-0.7)^0) (0.7^1(1-0.7)^0) \\
&= 0.7 * 0.7 = 0.49
\end{align}
If $k_1= 1$ and $k_2 = 0$, we have
\begin{align}
f_\text{joint}(1, 1;0.7)
&= (0.7^1(1-0.7)^0) (0.7^0(1-0.7)^1) \\
&= 0.7 * 0.3 = 0.21
\end{align}
If $k_1= 0$ and $k_2 = 1$, we have
\begin{align}
f_\text{joint}(1, 1;0.7)
&= (0.7^0(1-0.7)^1) (0.7^1(1-0.7)^0) \\
&= 0.3 * 0.7 = 0.21
\end{align}
So, the probabilities are not uniform, but we still have two independent r.v.s, because we defined their joint as the product of their marginals.
Now, set $p = 0.5$, you will see that we will have $1/4$ for all the combinations of $k_1$ and $k_2$.
Generally, the joint distribution of $n$ independent Bernoulli can be  compactly written as follows
$$
f_\text{joint}(k_1, \dots, k_n; p)
=
p^{\sum k_i}(1-p)^{n- \sum k_i}
$$
Conclusion: you cannot determine whether a sequence of r.v.s is i.i.d. by just looking at the probabilities.

Okay, I think I understood now. If we consider a dataset, then we should not consider features as random variables to call the dataset an iid and we should view instances of the dataset as random variables.

@hanugm You can consider instances of the dataset as random variables, I would say. The dataset would be like the "sequence" here.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is down-sampling the only purpose of using stride?
Stride is used in at least two operations: convolution and pooling. Both operations can be viewed as applying a kernel function on input using a kernel (filter).
Stride determines the amount of "jump" the kernel needs to perform on the input. Obviously, in the extreme case, if the kernel size and stride are one, the input size is the same as the output size. In all other cases, the output size is less than the input size.
So, I am guessing that down-sampling is the only purpose of using stride in any case. Am I true? Else, are there any cases in which stride is used in order to serve another purpose?

The general purpose of stride (along with padding) is to determine the spatial dimensions of the output. So, with appropriate stride (and padding), you can also make the spatial dimensions of the output volume bigger than the ones of the input volume. In fact, transpose convolution, which is used e.g. in the context of convolutional auto-encoders, is based on this idea. Pooling is used for downsampling.


text:
What is the stride information of an image referring here?
In convolutional neural networks, the convolution and pooling operations have a parameter known as stride, which decides the amount of jump the kernel needs to do on the input image. You can get more information regarding stride from follows taken from here

Stride is the number of pixels shifts over the input matrix. When the
stride is 1 then we move the filters to 1 pixel at a time. When the
stride is 2 then we move the filters to 2 pixels at a time and so on.

But, I am not getting what does it mean by stride information of an image at the tensor level. Consider the following paragraph from the chapter named Real-world data representation using tensors from the textbook titled Deep Learning with PyTorch by Eli Stevens et al.

img = torch.from_numpy(img_arr)
out = img.permute(2, 0, 1)

We’ve seen this previously, but note that this operation does not make
a copy of the tensor data. Instead, out uses the same underlying
storage as img and only plays with the size and stride information
at the tensor level. This is convenient because the operation is very
cheap; but just as a heads-up: changing a pixel in img will lead to
a change in out.

It is mentioning about the stride information at the tensor level of an image. Do they mean the strides that are related to the CNN, pooling, etc., or are they referring to any other stride information?

Do they mean the strides that are related to the CNN, pooling, etc., or are they referring to any other stride information?

The stride referred to by the quote "only plays with the size and stride information at the tensor level" is referring to internal storage of tensors. Luckily in most normal conversations about AI logic you do not care about this, it has nothing to do with neural network implementation, but is a lower-level optimisation (compared to some other ways of implementing tensors that would not use an internal stride).
Most general-purpose computer chips do not use multi-part addressing for data, such as (x,y,z), but instead use a single memory location number. To construct a data structure to represent a 2-or-more dimensional tensor means combining more fundamental representations. There are a few different ways to do this. One might be an array of arrays, where the outer array contains pointers to memory addresses of inner arrays. However, a single array with some management numbers attached to it (typically at the beginning, or even as a separate data structure, so you can have multiple tensor views of the same data) has some advantages - some operations on the array are far more efficient. Reshaping and transposing are two operations that can be implemented far more efficiently when the storage is arranged in this way.
So, along with the tensor cell data (the number inside each location), a typical tensor implementation will store information separately about how that data is structured. One useful piece of data is the stride, which is how many memory locations to add for each step in a particular direction in the n-indices view of the data. For example in a 2-indexed tensor with shape (6,8) where the data is stored in rows, then incrementing the column index will take a +1 step in memory location, whilst incrementing the row index will take a +8 step (the number of columns in a row). This is the row stride for that shape.
Technically these two types of stride are conceptually very similar. The difference is the level within the code that they are implemented in. Usually for most work using a tensor library you only care about this "internal" stride happening because it gives you confidence that there is a low cost to a particular tensor operation, such as generating a different shaped view.
Most importantly, unless you are working on your own tensor library, or digging into the internals of one using a low-level language (C), then you do not need to specify the stride values used. Instead you can take the quoted text as a declaration that a particular operation is implemented efficiently and you can use it in your own code with confidence that there is not a high CPU (or GPU) cost to using it.


text:
Multiple labels for the same rectbox?
My goal is to identify the horse in a photo. I'm dealing with about 500 unique horses.
My feeling is that the best way to distinguish one horse from another is by its face. So I trained Yolov5 successfully to find faces at reasonable angles.
I'd like to take this a step further, and teach it to identify which horse's face it sees.
I'm new to this sort of thing (though not programming in general), so the way I assume I should approach this is to add an additional label like face_horsename, with the unique name for the horse (or really, a unique reference to a database of horses).
Is that the right approach? It seems like the Yolo file format doesn't allow for multiple labels for the same box, so my guess is I should just make 2 rectboxes that are identical, but both point to different labels.
Frankly, I'd like to take it even further and label the same thing with the type of "blaze" of the horse's face, and its proper name for the horse's color. So now I'm talking about 4 labels.
Is that the right approach (duplicate boxes with unique labels)?

Duplicate boxes with unique labels make the problem too complex for the model. What I suggest is you use the horse face detection model to get a bounding box of the horse's face, crop the face image and use that image as a training sample for a separate classification model.
I have seen  this method used often in human identification, and dividing the tasks/models seems much more reasonable than trying to solve it in one model.
P.S. Just out of curiosity, you said that

the best way to distinguish one horse from another is by its face

Is this really true? Aren't there better features to use from the body?

Do you have any recommendations for generating the classification model?
As far as whether or not the face is the best way to identify them, I think it's practically the easiest. The legs sometimes are unique, but those aren't always well captured in a photo (especially a portrait).

@GoldenNewby A simple cnn classifier would suffice for a baseline.


text:
Is a recurrent layer same as LSTM or single-layered LSTM?
In MLP, there are neurons that form a layer. Each hidden layer gives a vector of number that is the output of that layer.
In CNN, there are kernels that form a convolutional layer. Each layer gives feature maps that are the output of that layer.
In LSTM, there are cells that form a recurrent layer. Each layer gives a sequence that is the output of that layer.
This is my understanding of the basic terminology regarding MLP, CNN, and LSTM.
But consider the following description regarding the number of layers in LSTM in PyTorch

num_layers – Number of recurrent layers. E.g., setting num_layers=2
would mean stacking two LSTMs together to form a stacked LSTM, with
the second LSTM taking in outputs of the first LSTM and computing the
final results. Default: 1

The description uses the "number of recurrent layers" and the "LSTM" in a similar manner. How I can understand this? Is it costmary to consider a recurrent layer as an LSTM?

Depending on the context, when people use the term LSTM, they either refer to

an LSTM layer,
an LSTM unit (like a recurrent unit in an RNN or neuron in an MLP), or
an LSTM neural network (i.e. an RNN that uses LSTM units or layers).

In TensorFlow, an LSTM is a layer, so you can stack multiple LSTMs to create deeper architectures.
In PyTorch, the class LSTM can create an LSTM layer or multiple LSTM layers stacked together. You also have an LSTMCell, which should be just one LSTM layer.
My answer here should also be useful.


text:
Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?
I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks.
The chain rule of differentiation for a composite function can be expressed roughly as follows:
$$\dfrac{d}{dx} (f_1(f_2(f_3(\cdots f_n(x))))) = \dfrac{df_n}{dx} \dfrac{df_{n-1}}{dy_1} \dfrac{df_{n-2}}{dy_2} \cdots \dfrac{df_1}{dy_{n-1}}$$
We know that multilayer perceptrons are composite functions of layer functions. So, if layers are increasing, then the gradient terms to multiply will increase on the right-hand side.
If all the gradient terms on the right-hand side are between 0 and 1 then the gradient will become less and less if layers keep on increasing the layers. This phenomenon is called the vanishing gradient problem. Similarly, if all the gradient terms on the right-hand side are greater than 1. Then the product will become more and more. This phenomenon is called exploding gradient problem.
Since it is customary to use the same activation function across all the layers in deep neural networks, all the gradients on the right hand behave in a similar manner, i.e. either most of the gradient terms on the right-hand side fall between 0 and 1 or greater than one, which causes either vanishing gradient or exploding gradient problem.
Do my mathematical interpretation of the vanishing and exploding gradient problem true?  Am I missing anything?

It looks correct to me. At least, that's my understanding of vanishing/exploding gradients too.

Your understanding is totally correct. The chain rule is defined as the product of derivatives, and as you well mention, from the mathematical point of view four scenarios can happen (you can visualize them here):

If the terms are in $(-1,1)$, in their limit they tend to 0.
If they are all 1, they stay at 1.
If they are all -1, they alternate between -1 and 1.
If they are greater than 1, they tend to $\infty$. If they are smaller than -1 they tend to $-\infty$.

In practice, however, cases 1. and 4. are the most common, and most of the strategies (e.g. resnet, lstm) are designed to tackle this problem as you probably well know already.


text:
What exactly is data augmentation?
Data augmentation is useful in training. But, I am not sure when can a modification applied to data can be called data augmentation.
Suppose a technique is applied to the instances of a dataset and the size of the dataset does not increase but the instances change slightly, then can I call it a data augmentation technique?
For example, say padding, never increases the size of the dataset but adds some data to possibly every existing training sample. In other cases, such as rotating the images with an angle, then the number of instances increases in the dataset.

Data augmentation typically refers to the creation of new (training) data/instances (e.g. images) by e.g. modifying existing training/instances data in order to avoid over-fitting and improve the performance (e.g. generalization, precision, recall, etc.) of the model.
There are other techniques that attempt to mitigate over-fitting, for example, regularization techniques (like dropout), but data augmentation changes the data that you use rather than how you use it, or how you define and train/test a model.
When you use data augmentation, you're assuming that more information can be extracted from the original dataset. For example, if you augment your training dataset by rotating existing images of objects, then you assume that this can be beneficial for your task (i.e. you expect your model to encounter rotated versions of the same object).
Typical scenarios where you may need data augmentation are when you have small training or imbalanced datasets.
Padding is not a data augmentation technique because you don't do it to avoid overfitting. Padding is a data preprocessing technique because you do it in order for your model to even be able to process your data.
You could read A survey on Image Data Augmentation for Deep Learning (by Shorten & Khoshgoftaar, 2019, Journal of Big Data), which I have only skimmed through, but it looks like a good paper.


text:
How does a bounding box detection network "know" about absolute position?
I've always found bounding box regression a bit weird. There's no positional encoding like in vision transformers, so how does the network "know" the absolute position when producing bounding box coordinates? It gets even weirder when we are dealing with two-stage detectors because, in the second stage bounding box regression, only an ROI is available, not the whole image.

It of course depends on the detection model that is used.
But in your case I think you relate to a Faster-RCNN type architecture for bounding box detection. In this case only the relative position to an anchor is regressed, that is correct. Of course the regression "values" isolated have no information where they are absolute in the image since the same convolution is shared and slided over the whole image/layer. However, after your regression layer is done (i.e. slided the convolution over the whole previous layer) it outputs a new feature map with relative regression values. The regression values itself do not contain the information about the abolute position, but the xy position of the relative feature values inside the output feature map of the regressor combined with the relative values itself gives you the information about the absolute position.
This also makes it "translation invariant", which is an important and wanted feature of Faster-RCNN architectures.

Yeah so we know the anchor positions from the start and we just need the relative scaling of the box. And if I recall correctly, each individual regression head is tied to a specific anchor right?

Yes exactly. The "regression head"-convolution is shared and shifted over the whole input map and at each convolution-window position outputs 4k relative regression values that are written to the corresponding (absolute but unaccurate) position in the output map. Combined with the relative values themself you then get accurate absolute positions.


text:
What is meant by "lateral connection" in the context of neural networks?
A class of CNN is popular due to the implementation of residual connections.
We can use both terms "residual connections" and "skip connections" interchangeably as they refer to the same.

Residual connections are the same thing as 'skip connections'. They
are used to allow gradients to flow through a network directly,
without passing through non-linear activation functions. Non-linear
activation functions, by nature of being non-linear, cause the
gradients to explode or vanish (depending on the weights).

While studying some models that are used in codes of some deep learning projects, I came across the word "lateral connections".
I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.
Am I true? If not, when can I call a connection lateral?

When you say "While studying some models that are used in codes of some deep learning projects", it may be a good idea to provide the source where you read about this term.

Lateral connection in NN simply means that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) then lateral connection implies that unit 1 may be connected to unit 2.
So now the activation of unit 2 is not only dependent on incoming inputs but also on unit 1. The weight assigned to a connection between unit 2 and unit 1 needs to be learned as part of backprop.
Please refer to the below image for more details :

Lateral connections in a feed-forward architecture. Inputs and hidden layer neurons are fully connected as are hidden layer and output layer neurons. Neuron j in the hidden layer also receives the net input of neuron (j - 1) in the hidden layer through a lateral connection

Ref: https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0
Lateral Connection can occur between units in layers and can be applied to feed forward network


text:
Where do the feature extraction and representation learning differ?
Feature selection is a process of selecting a subset of features that contribute the most.
Feature extraction allows getting new features that are not actually present in the given set of features.
Representation learning is the process of learning a new representation that contributes the most.
I can see no difference between feature extraction and representation learning.
Is feature extraction the same as representation learning? If not, where do they differ? Do they differ at the application level only?

Feature extraction (FE) is not the same as representation learning (RL), but they are similar and related.
You describe accurately what feature extraction typically refers to, i.e. the process of extracting (new) features from existing ones or raw data (e.g. images). For example, let's say you have a dataset associated with a car. You have only two features in your dataset: distance and velocity. However, from these two, you can extract a third feature, e.g. the acceleration. So, feature extraction can be performed with a fixed algorithm (e.g. PCA for dimensionality reduction or SIFT) or manually.
Representation learning is the collection of all techniques that extract features automatically from the data (i.e. they learn the features or representations, hence the name representation/feature learning). So, for example, a convolutional neural network trained on ImageNet can (and/or needs to) learn general features in order to solve the corresponding classification task. (Chapter 9 of the book Deep Learning by Goodfellow et al. talks more about this topic.) These features are learned from the data (and that's why CNN's are data-driven), and they can later be exploited for transfer learning (TL), i.e. TL is based on the idea that neural networks learn general representations (which is thus a synonym for features) of data that can be exploited to solve other tasks (sometimes known as downstream tasks, especially in the context of self-supervised learning).
Yoshua Bengio et al. defines representation learning as follows

learning representations of the data that make it easier to extract useful information when building classifiers or other predictors

So, RL is a subset of FE, given that RL also extracts features, but RL emphasizes the extraction of features automatically.


text:
Book(s) for text embedding
Text here refers to either character or word or sentence.
Is there any recent textbook that encompasses from classical methods to the modern techniques for embedding texts?
If a single textbook is unavailable then please recommend a list of books covering the whole spectrum as mentioned above.
Modern textbooks that are similar to Christopher D. Manning, Prabhakar Raghavan and Hinrich Schütze, Introduction to Information Retrieval, Cambridge University Press. 2008 are highly encouraged.
This question asks for textbook/research paper on word embedding only.

I propose you try this. It's about modern Natural Language Processing, Computational Linguistics and Speech Recognition, including Embeddings methods.

Yeah. Chapter 6: Vector Semantics and Embeddings is apt.

Yeah, answer is useful. But I am waiting for a book that explicitly dedicates on that topic.

To find a book that explicitly focuses on embeddings, I think it is something very difficult. My proposal, however, provides a strong theoretical background for this topic.

Having a sound understanding on language processing will help you understand all its concepts. This summarise must reads for NLP.


text:
Which tasks are called as downstream tasks?
The following paragraph is from page no 331 of the textbook Natural Language Processing by Jacob Eisenstein. It mentions about certain type of tasks called as downstream tasks. But, it provide no further examples or details regarding these tasks.

Learning algorithms like perceptron and conditional random fields
often perform better with discrete feature vectors. A simple way to
obtain discrete representations from distributional statistics is by
clustering, so that words in the same cluster have similar
distributional statistics. This can help in downstream tasks, by
sharing features between all words in the same cluster. However, there
is an obvious tradeoff: if the number of clusters is too small, the
words in each cluster will not have much in common; if the number of
clusters is too large, then the learner will not see enough examples
from each cluster to generalize.

Which tasks in artificial intelligence or NLP are called as downstream tasks?

downstream

In the context of self-supervised learning (which is also used in NLP), a downstream task is the task that you actually want to solve. This definition makes sense if you're familiar with transfer learning or self-supervised learning, which are also used for NLP. In particular, in transfer learning, you first pre-train a model with some "general" dataset (e.g. ImageNet), which does not represent the task that you want to solve, but allows the model to learn some "general" features. Then you fine-tune this pre-trained model on the dataset that represents the actual problem that you want to solve. This latter task/problem is what would be called, in the context of self-supervised learning, a downstream task. In this answer, I mention these downstream tasks.
In the same book that you quote, the author also writes (section 14.6.2 Extrinsic evaluations, p. 339 of the book)

Word representations contribute to downstream tasks like sequence labeling and document classification by enabling generalization across words. The use of distributed representations as features is a form of semi-supervised learning, in which performance on a supervised learning problem is augmented by learning distributed representations from unlabeled data (Miller et al., 2004; Koo et al., 2008; Turian et al., 2010). These pre-trained word representations can be used as features in a linear prediction model, or as the input layer in a neural network, such as a Bi-LSTM tagging model (§ 7.6). Word representations can be evaluated by the performance of the downstream systems that consume them: for example, GloVe embeddings are convincingly better than Latent Semantic Analysis as features in the downstream task of named entity recognition (Pennington et al., 2014). Unfortunately, extrinsic and intrinsic evaluations do not always point in the same direction, and the best word representations for one downstream task may perform poorly on another task (Schnabel et al., 2015).
When word representations are updated from labeled data in the downstream task, they are said to be fine-tuned.

So, to me, after having read this section of the book, it seems that the author is using the term "downstream task" as it's used in self-supervised learning. Examples of downstream tasks are thus

sequence labeling
documentation classification
named entity recognition

Tasks like training a model to learn word embeddings are not downstream tasks, because these tasks are not really the ultimate tasks that you want to solve, but they are solved in order to solve other tasks (i.e. the downstream tasks).

Your answer is correct. Features
are transformations on input data that facilitate a downstream algorithm, like a classifier, to produce correct outcomes on new data  It is given here

Are there any upstream tasks as opposed to downstream tasks?


text:
How does the neural network learn when used in the REINFORCE algorithm?
As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good and bad actions?

How does the neural network learn to differentiate between good and bad actions?

Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples where the actions occur in different combinations.
In REINFORCE, when training the neural network, all actions are effectively treated as ground truth "correct", but the gradient is weighted by the return from that time step. The best trajectories therefore get larger gradient steps, and therefore shift the action choices made in those trajectories more towards them than the less good trajectories.
In the case of positive vs negative returns this is very clear (because negative returns will reverse the gradient step away from the action choices that were made). However, even if all returns are negative or all are positive, the REINFORCE algorithm still works - provided there are enough samples of different trajectories with actions taken in different contexts, then there will be a preference for the best action in each state.
REINFORCE with baseline and Advantage-based update multipliers (which are a variation of REINFORCE with baseline) are slightly better than the most basic REINFORCE "race to the top", in that they are more numerically stable and will automatically split into relative positive and negative updates regardless of whether the environment produces positive or negative returns overall.
Ideally you would not run a REINFORCE update step based on a single trajectory at a time, for two reasons:

The trajectory contains correlated data, and neural networks learn badly from it, preferring i.i.d. samples if possible.

A single trajectory does not contain enough information by itself for a learning agent to figure out whether it is good or bad relative to other options.

Neither of these are complete showstoppers, but you should find that many implementations of policy gradient methods collect multiple trajectories before each update step.

I'm having a hard time wrapping my head around this. Each state would have an optimal action(s) compared to other actions. How would REINFORCE learn that by viewing an entire trajectory?
While calculating loss/backpropogation, does the algorithm ever compare the prediction for a state with the reward/value obtained?

The algorthm is gradient ascent on the neural network for the output of actions taken times the return measured when taking that action. Unlike classification which  rates the action against a known correct value, REINFORCE rates each action according to how well the overall trajectory after that point turned out. Clearly this includes impact of other choices. However, on average each state's choices will favour the best action in the long term. The algorithm never compares anything, it simply follows gradients that produce high returns more than it follows gradients that produce low ones.


text:
Why is tanh a "smoothly" differentiable function?
The sigmoid, tanh, and ReLU are popular and useful activation functions in the literature.
The following excerpt taken from p4 of Neural Networks and Neural Language Models says that tanh has a couple of interesting properties.

For example, the tanh function has the nice properties of being
smoothly differentiable and mapping outlier values toward the mean.

A function is said to be differentiable if it is differentiable at every point in the domain of function. The domain of tanh is $\mathbb{R}$ and $ \dfrac{e^x-e^{-x}}{e^x+e^{-x}}$ is differentiable in $\mathbb{R}$.
But what is meant by "smoothly differentiable" in the case of tanh activation function?

That sounds like a pretty subjective description. I would argue sigmoid is also smoothly differentiable because the gradient function is predictable and symmetric. It could maybe be referring to how the tanh derivative exists in one of 3 states: Essentially stationary, linear, or a transition between the two.

In math, as far as I know, "smooth" often refers to the fact that the function can be differentiated multiple times. See this.  To be honest, I never really thought about the properties of the hyperbolic tangent, so I don't know if the author of that excerpt is referring to something else.

A smooth function is usually defined to be a function that is $n$-times continuously differentiable, which means that $f$, $f'$, $\dots$, $f^{(n - 1)}$ are all differentiable and $f^{(n)}$ is continuous. Such functions are also called $C^n$ functions.
It can be a bit of a vague term; some people might even stretch the definition and say any continuous function is smooth (though I'd be a little surprised if I saw that in use, personally). Other people write smooth to mean infinitely differentiable: for example $f(x) = e^x$ can be differentiated as many times as you like.
I guess what the author is trying to point out is that the ReLU rectifier function isn't differentiable. Even if you use the "trick"1 of treating ReLU as differentiable everywhere, you would still get a derivative that is discontinuous:
$$\mathrm{ReLU}'(x) = \begin{cases} 1 & x \ge 0 \\ 0 & \text{otherwise.} \end{cases}$$
So, it's fair to say that ReLU isn't smooth in the same sense of the $\tanh$ function, which has a continuous derivative (and, in fact, you could carry on and consider the higher derivatives).

1 If this doesn't sound familiar, see p. 188 of Deep Learning by Bengio et al. We can get around the fact that ReLU functions aren't differentiable at zero by just pretending it has a well-defined derivative of zero or one. A little dishonest, perhaps, but it works very well.


text:
Train a deep learning model with input as a vector and predicts as a vector?
I am trying to build a Deep Learning model that takes a numeric vector $X$ of dimension $1 \times 50$ and predicts a numeric vector $y$ of dimension $1 \times 50$.
It's a linear regression problem. I am trying to achieve the coefficients/weights that can help me
Code I used:
X = np.array(...) // array of 50 features and 5 sample vectors (shape of X is 5x50)
y = np.array(...) // array of 50 features (shape of y is 5x50)
model = Sequential([Dense(1, input_shape=[5,50])])

optimizer = Adam(0.001)

model.compile(loss='mse', optimizer=optimizer, metrics=['mae', 'mse'])

model.fit(X,y, epochs=250, validation_split=0.25))

So, basically we are achieving X*w ~ y where $w$ is the weights/coefficients that we want to identify using DL.
Programmatically, I tried using the same logic and calculated $w = y . X^{-1}$ for all the vectors. Took average of the coeff and applied on the test data.

What exactly is your doubt? Are you asking whether your procedure is correct or nor?

I am asking for procedure on how to achieve it as I am unable to achieve proper results.

as far as I can see there is a problem in your third line of code. input_shape should be the shape of the input to the network, not the shape of your training data.
If your training data has the shape of 5x50 you would have 5 training samples in your dataset.
For this to work your input has to be 50:
model = Sequential([Dense(1, input_shape=[50])])

Now with model.summary() you can check that the input of your network is (None, 50).
Also note that in a neural network, each neuron has an activation function.
Depending on the activation function $f(z)$ , the output would be $y = f(b+W^TX)$ which is not the same as linear regression.
Hope this helps!


text:
What exactly is an XPU?
I know about CPU, GPU and TPU. But, it is the first time for me to read about XPU from PyTorch documentation about MODULE.
xpu(device=None)

Moves all model parameters and buffers to the XPU.

This also makes associated parameters and buffers different objects. So it should be called before constructing optimizer if the module will live on XPU while being optimized.

Note: This method modifies the module in-place.

Parameters

    device (int, optional) – if specified, all parameters will be copied to that device

Returns

    self

Return type

    Module

CPU stands for Central Processing Unit.
GPU stands for Graphical Processing Unit.
TPU stands for Tensor Processing Unit.
Most of us know that these processing units are highly useful in the research of some computational intensive domains of AI including deep learning. So, I am wondering whether XPU is also useful in AI research since it is used in PyTorch.
From the context, I can say that PU stands for processing unit. But I have no idea of what X is.
What is the full form for XPU? Where can I read about XPU in detail?

This blog post by Intel seems to be explaining it: https://medium.com/intel-tech/the-evolution-of-xpu-and-the-critical-role-of-software-c46970dfcbe9

XPU is a device abstraction for Intel heterogeneous computation architectures, which can be mapped to CPU, GPU, FPGA and other accelerators. The "X" from XPU is just like a variable, like in maths, so you can do X=C and you get CPU accceleration, or X=G and you get GPU acceleration... That's the intuition behind that abstract name.
In order to integrate a new accelerator you need 2 things:

HW support: read about XPU in the official intel XPU webpage.
SW support: see this open Feature Request to include this kind of architecture to the pytorch pool of HW accelerators


text:
What is meant by an axis of a tensor?
Tensor is an ordered collection of elements. The elements are generally real numbers. Tensors are used in deep learning for storing data.
There is a wide usage of the word "axis" related to tensor. Axes are not the same as indices, which are used to access the elements of a tensor. An axis is not the same as an element of a tensor.
What exactly is an axis in a tensor? Is it also a (sub-)tensor obtained from the actual tensor? Or is it any other indexing mechanism? If yes, why it is used?
Suppose $a =[[1, 2, 3, 4],[5, 6, 7, 8],[9, 10, 11, 12],[13, 14, 15, 16]]$ is a tensor. Then what does axis do for $a$?

In machine learning, a tensor is a multi-dimensional array (i.e. a generalization of a matrix to more than 2 dimensions), which has some properties, such as the number of dimensions or the shape, and to which you can apply operations (for example, you can take the mean of all elements across all dimensions). So, a scalar is a 0-d tensor (no dimensions), a vector is a 1d tensor (1 dimension), a matrix a 2d tensor (2 dimensions), a cube a 3d tensor, and so on.
The name TensorFlow really comes from the fact that TensorFlow manipulates tensors all the time. The documentation of TensorFlow describes tensors as follows

Tensors are multi-dimensional arrays with a uniform type (called a dtype)
If you're familiar with NumPy, tensors are (kind of) like np.arrays.

In this context, you can think of the axis as an abstraction for dealing with or manipulating the dimensions (aka shape) of a tensor or to apply certain operations "across dimensions". It's not exactly a synonym for dimension because, in some software libraries (e.g. Keras and NumPy), you can also pass special arguments such as -1 to the axis parameter, which does not correspond to any dimension (there's no $-1$ dimension of a tensor). It's a way to apply certain operations to the tensor only across e.g. the first dimension (axis 0) or the second dimension (axis 2), and so on.
For example, let's say you have a matrix (2d tensor) $A$. You want to compute the average (aka mean) of the rows, i.e. sum all elements of row $i$, then divide by the number of elements (which corresponds to the number of columns in row $i$), and do this for all rows $i=1, \dots, K$. You can do this in NumPy by specifying that you want to apply the mean operation to axis=1. So, after this operation, you will get a 1d tensor (a vector) with $K$ elements (i.e. the numbers of rows of the original matrix): this is what I mean by "across dimensions".
Here's a NumPy example that illustrates the concept.
import numpy as np

A = np.array([[1, 2, 3],
              [1, 1, 1],
              [0, 1, -1]])

# (1 + 2 + 3 + 1 + 1 + 1 + 0 + 1 + (-1)) / 9
m = np.mean(A)
print("Mean of all elements of A =", m)

# [(1 + 2 + 3)/3, (1 + 1 + 1)/3, (0 + 1 + (-1))/3]
m1 = np.mean(A, axis=1)
print("Mean of each row =", m1)

# [(1 + 1 + 0)/3, (2 + 1 + 1)/3, (3 + 1 + (-1))/3]
m2 = np.mean(A, axis=0)
print("Mean of each column =", m2)

Why would you want to compute (in this case) the mean of each row? Because, for example, each row $i$ may correspond to some data associated with a user $i$, so you may want to compute e.g. the average salary for each user for all months (in this case, there are only 3 columns, so 3 months).
Tensors did not originate in machine learning. In fact, in mathematics, tensors are well-known objects, which have some properties, which may be different than the properties associated with the tensors implemented in libraries like TensorFlow, which are really just multi-dimensional arrays with the necessary properties and methods for machine/deep learning. Tensors are also everywhere in quantum computing.
This answer that I wrote a while ago could also be useful.

Imagine the tensor as a some generalized $n$-dimensional hyperrectangle sliced into $n$-dimensional hypercubes. Each element of the tensor is labeled by the position along the given axis, say $(x_1, x_2, \ldots)$.
Axis is not a property of tensor, rather the tensor is embedded in a $n$-dimensional space, where the axes are chosen along the sides of the hyperrectangle corresponding to the tensor.
There are many operations, that can be applied axiswise to tensor. Several examples:

Mean along the axis (choose $0$ without loss of generality). Given the $n$-dimensional tensor $x_{i_1, i_2 \ldots i_n}$ , the result will be $n-1$-dimensional tensor $\frac{1}{N_0} \sum_{i_1} x_{i_1 i_2 \ldots i_n}$, where $N_0$ is the number of elements of the tensor along the $0$-th axis (height).
Standard deviation along the axis. For each index $i_2 \ldots i_n$, calculate $\sqrt{\frac{1}{N_0} \sum_{i_1}  (\bar{x}_{i_2 \ldots i_n} - x_{i_1 i_2 \ldots i_n})^2}$, where $\bar{x}$ is the mean from previous point, and the result will be again $n-1$-dimensional tensor.

For your example $a$ is a $2$-dimensional tensor with $2$ axes. $0$-th axis corresponds to rows, $1$-st axis corresponds to columns.


text:
Is it abuse of notation to use tilde operator in this context?
The following is a way to use tilde (∼) in context of random variables or random vectors.

In statistics, the tilde is frequently used to mean "has the
distribution (of)," for instance, $X∼N(0,1)$ means "the stochastic
(random) variable $X$ has the distribution $N(0,1)$ (the standard
normal distribution). If X and Y are stochastic variables then $X∼Y$
means "$X$ has the same distribution as $Y$.

Consider the following usage of tilde in the paper titled Generative Adversarial Nets
$$x ∼ p_{data}(x)$$
$$z ∼ p_z(z)$$
I am thinking that the following is the standard (and possibly correct) notation
$$x ∼ p_{data}$$
$$z ∼ p_z$$
$p_{data}$ is a probability distribution and $p_{data}(x)$ is not a probability distribution and it is a value in $[0, 1]$. It is same in case of noise probability distribution.
Is it an abuse of notation to use in such a way or is it also a standard and allowed notation to use?

Comments are not for extended discussion; this conversation has been moved to chat.

Since the question is now closed as off-topic by the community, I honestly hope that it will not be re-opened unilaterally by the mod that has an obvious conflict of interest (i.e. has provided an answer below), at least not without a discussion in Meta. As it seems, the community has real concerns about if such a question about general mathematical notation, and tagged as probability and random-variable, has really anything specific to do with AI, just because it happened to be stumbled upon in a ML paper.

Most of the AI textbooks has initial units dedicated to Mathematics useful to AI. So, I think these are on-topic. If some maths question is asked that is not directly or indirectly related to AI, then it can be off-topic.

The notation $p(x)$ is widely used in machine learning (e.g. here) and even statistics (e.g. here). People often use $p(x)$ to refer to a probability distribution (either pmf, pdf, or cdf) rather than just $p$. There is also the notation $p_x$ (or things like $p_{x \mid y}$ for conditional p.d.s), which you will find in some statistics books.
Of course, if you interpret the notation $p(x)$ as the evaluation of e.g. the pdf at $x$, it's a density value, so not a p.d. However, I think, in your case, $p(x)$ is used as $p_x$, i.e. to emphasize that the pd $p$ is associated with the r.v. $x$ or that it's a function of one variable (in this case denoted by $x$ to remind the reader that this pd is associated with the r.v. $x$). That's how I would interpret that notation in this case.
This can be useful if you use the letter $p$ to denote multiple p.d.s associated with different r.v.s (which is often the case); so, by using $p_x$ or $p(z)$, you make it clear which r.v. $p$ is associated with.
I looked at the GAN paper, and sometimes they use $z \sim p_z(z)$ and other times $z \sim p_z$, so there are some inconsistencies in the paper, which is not so uncommon in machine learning papers (sometimes people do that just to avoid verbosity).
One that knows the mathematical definition of an r.v. will also know that an r.v. is a function. So, if $p$ is e.g. a pdf and $x$ in $p(x)$ is an r.v., then $p(x)$ would be a composition of functions. I don't think that people use the notation $p(x)$ for that case, as this can become more complicated: we can easily start to talk about measure theory or pushforward measures (which I only barely heard of). I think that many people in the machine learning community do not have a formal statistics or mathematical background, so they probably just use $p(x)$ because everyone else does.
In this answer that I wrote a while ago, I also say that $p(x)$ could be a shorthand for $p(x = x_i)$, i.e. the probability that $x = x_i$ (some event). This is also common, but that's not how I would interpret $p(x)$ in your case.


text:
Is "kernel" different from "filter" in convolutional neural networks?
Recently I asked a question on how a convolution 2d layer changes an RGB image into a grayscale image. Assume that our task is to convert an RGB image into a grayscale image. I use to believe that filter and kernel are one and the same.
Consider Conv2d in PyTorch.
class torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)

The parameters in_channel, out_channel and kernel_size are key to our discussion here.
I have no doubt about in_channel. It simply says the number of channels in the input image. It is 3 for our task since we have RGB images as input.
The doubt is regarding the parameters out_channel and kernel_size. out_channel refers to the number of channels in the output image. It is 1 for our task since we want grayscale images as output. It is also equal to the number of filters we are needing. So, we just use one filter to convert an RGB image into a grayscale image. kernel_size is the size of the kernel which is showing $3 \times 3$ in our case. Now, my convolution layer is
>>> in_ch = 3
>>> out_ch = 1
>>> m = nn.Conv2d(in_ch, out_ch, 3, 1, 1)
>>> print(m)
Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

Since I have doubt about the conversion of RGB image to grayscale image using a single filter and whose size is showing $3 \times 3$, I checked the shape of weights in the filter and realized that the single filter is a 3-dimensional filter of size $ 3 \times 3 \times 3$
>>> print(m.weight.shape)
torch.Size([1, 3, 3, 3])

Now, the filter size is $3 \times 3 \times 3$ and kernel_size is $3 \times 3$.
So, can I safely conclude that the filter is different from the kernel? Can I conclude that kernel is just a part of filter and filter may comprise several kernels? Or is it true that the usage in PyTorch is a bit misleading since I found that our site is also using the same tag for both filter and kernel?

The term "filter" is (usually) a synonym for "kernel" in the context of convolutional neural networks and image processing.
The reason why the kernel_size is specified as $3 \times 3$ and then you see that the actual size of the kernel (aka filter) is 3d is that the depth of the kernel can be automatically inferred from in_channels, the depth of the input to the convolutional layer: the depth of the kernel is exactly equal to the depth of the input to that 2d convolutional layer. This would not necessarily be the case in the 3d convolutional layer.
As a side note, in other contexts, like support vector machines, the word "kernel" refers to something different, but I will not dwell on this topic here.


text:
Is my understanding on "smooth approximation" correct?
Consider the following details regarding Softplus activation function

$$\text{Softplus}(x) = \dfrac{\log(1+e^{\beta x})}{\beta}$$
SoftPlus is a smooth approximation to the ReLU function and can be
used to constrain the output of a machine to always be positive.

It says that Softplus is a smooth approximation to the ReLU function. Let us consider the analytical form and plot of the RELU function.
$$\text{ReLU}(x)=(x)^+=\max(0,x)$$

The plot of Softplus function is

If we observe both plots, we can see the Softplus is almost similar to ReLU. There is a property for Softplus that ReLU does not have. ReLU is not differentiable at zero and the derivative of ReLU is also not continuous.
If we observe the behavior of Softplus, it is $n-$times continuously differentiable and hence a smooth function.
Since Softplus is both a smooth function and approximates ReLU, it is considered as a smooth approximation of ReLU.
Is my interpretation correct? if no, then what is meant by "smooth approximation" here?

Your interpretation is definitely correct. As you correctly pointed out, the derivative of softplus is continuous and $n$-times differentiable, that makes the function smooth, which is not the case for ReLU.
What is quite interesting here is why softplus can be called an approximation to ReLU.
If we break down the definition of softplus, we note that the parameter $\beta$ controls the smoothness of the function. In fact, we can define a temperature $T = \frac{1}{\beta}$ (much like the softmax function).
Let's denote with $f_T$ a softplus function with temperature $T$. We can easily see that as $T \rightarrow 0$ (or equivalently $\beta \rightarrow \infty$), $f_T(x) \rightarrow \max(0, x)$. In fact, let's compute the limit of the softplus for all $x > 0$:
$\lim_{\beta \rightarrow \infty} \frac{1}{\beta} ln(1 + e^{\beta x})$
Rewrite $ln(1+e^{\beta x})$ as $ln(e^{-\beta x} + 1) +ln(e^{\beta x})$ we get:
$\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(e^{-\beta x} + 1) +ln(e^{\beta x})]
= \\ =
\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[ln(0 + 1) + ln(e^{\beta x})]
= \\ =
\lim_{\beta \rightarrow \infty} \frac{1}{\beta}[0 + ln(e^{\beta x})]
= \\ =
\lim_{\beta \rightarrow \infty} \frac{1}{\beta}\beta x = \\ = x$
Computing the limit for $x \leq 0 $ is straightforward and the result is $0$.
So we see that for high value of $\beta$ (or low value of $T$), the softplus is actually the $\max(0, x)$ function!
Therefore, softplus is a family of functions $f_T$ which approximate ReLU with high precision as $T$ gets to 0 (less smoothness), and low precision with high $T$ (more smoothness).
Among all, we usually use softplus with $beta = 1$ (i.e, $f_1$), and this is why we call it a smooth approximation of the ReLU. Because $f_1(x)$ is smooth ($T=1$) and approximates $f_\infty(x) = \max(0, x) = \text{ReLU}(x)$.
Hope this was useful.


text:
Mapping input vectors of variable length to output vectors of variable lengths with dummy variables
I have a general question about supervised ANNs that map inputs to outputs. It is possible to vary the length of the input and output vectors by inserting some dummy variables that will not be considered in the mapping (or will be mapped to other dummy variables). So basically the mapping should look like this (v: value, d: dummy)
Input vector 1 $[v,v,v,v,v] \rightarrow$ Output vector 1 $[v,v,v,v,v]$
Input vector 2 $[v,v,v,v,v]\rightarrow$ Output vector 2 $[v,v,v,v,v]$
Input vector 3 $[v,v,v,d,d] \rightarrow$ Output vector 3 $[v,v,v,d,d]$
Input vector 4 $[v,v,d,d,d] \rightarrow$  Output vector 4 $[v,v,d,d,d]$
Input vector 5 $[v,d,d,d,d] \rightarrow$  Output vector 5 $[v,d,d,d,d]$
The input and output vectors have a length of 5 with 5 values. However, sometimes only a vector of size e.g. 3 (which is basically a vector of length 5 with 2 dummy variables) should be mapped to an output vector of length 3. So after training the ANN should know that if it for example gets an input vector of length 3 it should produce an output vector of length 3.
Is something like this generally possible with ANNs or other machine learning approaches? If so, what type of ANN or machine learning approach can be used for this? I'll appreciate every comment.
Reminder: Can anybody give me more insights into this?

What is the difference between first two pairs of vectors?

Thanks hanugm for your answer. Basically there is no difference between the first two pair of vectors. I just wanted to show that we have multiple training data with varying length. The first two pair of vectors turn out to have the same size (same number of v) in this example.

Im saying that first I/O vectors and second I/O vectors are same.

Well this is just an example and the v can stand for any number. The first two pair of vectors can but do not have to be identical (most probably they are not). I just have let's say 1000 input and output vectors and an ANN should be trained to map them correctly. The v stand for any number. So most of the input vectors (even if they have the same size) are not identical. But my question is trageting at having input and output vectors with different lengths

Is the output always going to have the same length as the input? In that case you can just simply ignore the part of the output that is past the length of the input.

If the output can have variable length for a set input, then you can take a sequence to sequence approach (used in natural language processing). In that case you predict 1 embedding (word) at a time, and "end" is a possible prediction. So you can get variable length output.

Thanks Taw for your comments. Yes, the output will always have the same length as the input. I am aware that I could just ignore the part that is irrelevant. However, I assume that for the training this will not be beneficial as the ANN will learn how to map the whole sequence to the other. The ANN should just learn how the map the relevant parts of the input and output to each other. Is that somehow possible e.g. by using dummy variables that indicate to the ANN that these parts should not be considered for the mapping?

@Taw: Any comments on my last comment? I'd highly appreciate every further comment from you.

If you only use the relevant part of the output, the irrelevant parts will not contribute to the gradient and so they won't affect training.

Thanks Taw for your answer. How can I only choose the relevant part of the output (and input)? Basically the vectors have all the same length (otherwise I could not specify the ANN model).

@Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you

What is the loss function? You can think about it this way; what gets sent to loss function is the concatenation of the output from NN, and the length of the input. This way the loss is computed correctly. Implementing this is just a coding question

Thanks Taw for your answer and effort. I really appreciate it. Basically I do not understand what you meat by "what gets sent to loss function is the concatenation of the output from NN". The vectors should all have the same length. But for some of them several components should not coundt. How can I "send" this information to the loss function?

@Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you

@Taw: Any comments to my last comment? I'd highyl appreciate every further comment from you

This should be possible but I've never seen it done in practice. Whether or not this will even actually work is unclear to me and will be highly dependent both on your training data and choice of loss. I'd take a step back and look into the literature to see if you can't find a more established approach to your problem, perhaps with RNNs. That being said, I believe the following should do what you're asking.
Consider network $N$ to be a dense neural net with $k$ layers, $N_i$ to be the $i$th layer of $N$, $L$ to be the max length of the input, and $V$ to be the number of terms in the input (the number of $v$s). To accomplish what you want in the above scenario, you can add three additional layers to $N$, $N_{k+1}$ $N_{k+2}$, and $N_{k+3}$:
$N_{k+1}$ is a simple dense layer that has $L$ neurons and takes as input the output of $N_k$. This layer can be skipped if layer $N_k$ already has $L$ neurons.
$N_{k+2}$ takes as input the output $N_{k+1}$ and takes the Hadamard product (elementwise multiplication) of it with a second input, a binary vector of length $L$ with a prefix of $V$ $1$s and suffix of $L-V$ $0$s. For example if $L=5$ and $V=2$, you would supply as the second input the vector $[1, 1, 0, 0, 0]$, which effectively "zeroes out" the third, fourth, and fifth positions.
$N_{k+3}$ is your new output layer, which also has $L$ neurons. $N$ can now be trained and, given the target data is in the proper format to achieve this, it should output results like in your question.

Thanks Oso for your answer. I understand the concept behind it but I would assume that putting the target data in the proper format might be quite challenging. Nontheless I will consider using your approach. Thanks for your help. I upvoted and accepted your answer.


text:
Would AlphaZero work just with a value network?
There is a nice post about the intuition why AlphaZero works.
One of the advantages of using a policy network in the games where a perfect simulator is available (such as chess) is to save computation time by not generating all subsequent moves and then evaluating them using the value network. Instead, we can only focus on the good moves given by the policy network.
However, besides the computation time savings of the policy network, are there any requirements why it needs to be used during training?
What if we would replace the computation of the policy network with this logic: generate all subsequent moves, evaluate them using value network, and create policy from these predictions. Would this still work?
I would appreciate any references where this topic is discussed.

I'm not aware of anyone running a setup of everything that AlphaZero does, minus the Policy Network, and reporting on how well it worked, so I don't think I can provide a definitive 100% certain answer. My intuition says that it would "work" in the sense that it could still produce a very strong agent, but I suspect it could be slower to train and/or not reach as high of a peak.

One important advantage of the Policy Network over the Value Network is specifically during the training phase, and in particular in the beginning of training; the Policy Network receives much more training data. When we run self-play games as in the AlphaZero paper, we get one target for the Policy Network for every distinct state we encounter (so many different targets per state), but only a single target for the Value Network for every full game (just the outcome of the game). There are papers that try to address this in different ways (some extract multiple value targets from the tree, some speed up the generation of self-play games by only running shorter searches in some states and not generating policy targets from the shorter searches, etc.), but ignoring those... in the standard AlphaZero setup, we may expect the Policy Network to train faster / more easily, especially in the beginning. The Value Network can "catch up" afterwards.
A second advantage of the Policy Network over the Value Network is that it may more effectively learn to distinguish good moves from bad moves in highly one-sided game states. Consider a game state $s$ where the player is (almost) sure to win, and can even afford to make a few mistakes. We'll probably get that all the successor states $s'$ get approximately the same value estimates $V(s') \approx 0.999$, with only a tiny amount of variation, and we may be unable to distinguish good moves from bad moves. The value network may no longer have any sort of noticeable preference for good moves over bad moves, until it has made so many blunders/mistakes that it actually becomes necessary to play well again. It would probably still win, but possibly in a less convincing manner (and slower) than it could. It's not even wrong for the value network to learn in this way, since we do want it (ideally) to learn the game-theoretic values, and this would be correct from that point of view. But probably still undesirable. In contrast, the Policy Network is not trained to predict game-theoretic values, or trained just to win. It's actively trained to pick good moves over bad moves. Due to the exploratory behaviour that is inherent in MCTS, MCTS generally ends up preferring more convincing / faster / safer wins, so the Policy Network does actually get an incentive to really play well even in such states where mistakes can be afforded.
A similar story to the above also applies to game states where the player is almost sure to lose; the Value Network gets no incentive to distinguish states from each other and might end up playing poorly in losing states, whereas a Policy Network still gets incentive to play well and "fight back", which may end up allowing it to still reach a win against suboptimal players.

What if we would replace the computation of the policy network with this logic: generate all subsequent moves, evaluate them using value network, and create policy from these predictions.

This particular implementation you suggest would have an additional disadvantage of computational inefficiency: to evaluate the Value Network for all possible successor states, you actually need to 1) generate all those successor states first and 2) run forwards passes of the Neural Network for every successor. Both of those things take time. In contrast, the Policy Network just runs once for the single parent state, and at once computes all the probabilities for all the actions. This is much more efficient.

Related to your suggestion, but quite a bit different because it does not use MCTS (but a different tree search algorithm) as the underlying search algorithm, a rather effective approach that only uses a Value Network (no Policy Network) is described in "Learning to Play Two-Player Perfect-Information Games without Knowledge" by Cohen-Solal, with additional empirical results described in "Minimax Strikes Back".

Thank you for a nice answer! It seems that Minimax Strikes Back is exactly what I was looking for. Just one note to "Policy Network receives much more training data". If I am not mistaken the number of states used for training is the same for policy and value network, is not in? Also the targets for value networks should be equally distributed in the training samples; half of them are -1 and the other half is +1 (if there is no draw). So it is not really clear to me why policy network should learn faster. Can you please elaborate on that or point me to the mentioned literature? Many thanks!

@Druudik Technically yeah, in your experience buffer you get equal amounts of samples for Policy and Value. But the policy targets are different for every state, and specifically "made" for that particular state. The value targets are the same for all states in the full game (ignoring negation for the opposite colours), and really not "made" for that particular state. They're probably fairly accurate for the last few states, but may be really inaccurate for early states. I recommend taking a look at Section 3.1 of https://arxiv.org/abs/1902.10565

I think I am starting to get what you meant by that. Once again many thanks!

Just adding reference to one more article that I've just found which I think explains exactly what you were mentioning https://medium.com/oracledevs/lessons-from-alphazero-part-4-improving-the-training-target-6efba2e71628.

The solution approach for the linked work Mastering the game of Go with deep neural networks and tree search is not only valid for the game Go but can also be used for other games, e.g. chess.
The policy network has the task to learn the rules of the game and is then improved only with the help of the value network by focusing on winning games.
If I understand you correctly, you want to evaluate all the moves using the value network and pick the best one?
For this you need rules - i.e. which moves are allowed to be made at all.
That's what I think you need the policy network for.

Regarding the rules: that is what I've meant by "where perfect simulator is available". Would policy network be requirement also in this case?


text:
Why do we multipy context_size with embedding_dim? (PyTorch)
I've been using Tensorflow and just started learning PyTorch. I was following the tutorial: https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html#sphx-glr-beginner-nlp-word-embeddings-tutorial-py
Where we try to create an n-gram language model. However, there's something I don't understand.
class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)

at self.linear1 = nn.Linear(context_size * embedding_dim, 128) why did we multiply embedding_dim with context_size? Isn't the embedding_dim input size? So why do we multiply it by the context size?

An n-gram language model is a language model trained with n context words. This means you're not feeding the model a single word but n. This is why the dimension of the input layer is "context_size * embedding_dim" or "n * embedding_dims"


text:
How does this TD(0) off-policy value update formula work?
The update formula for the TD(0) off-policy learning algorithm is (taken from these slides by D. Silver for lecture 5 of his course)
$$ \underbrace{V(S_t)}_{\text{New value}} \leftarrow \underbrace{V(S_t)}_{\text{Old value}} + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)} (\underbrace{R_{t+1} + \gamma V(S_{t+1}))}_{\text{TD target}} - \underbrace{V(S_t)}_{\text{Old value}} \right) $$
where $\frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)}$ is the ratio of the likelihoods that policy $\pi$ will take this action at this state divided by the likelihood that behavior policy $\mu$ takes this action at this state.
What I do not understand is:
Assume the behavior policy $\mu$ took an action that is very unlikely to happen under policy $\pi$. I would assume this term goes towards $0$.
$$  \frac{ \pi(A_t|S_t)}{\mu (A_t|S_t)}  = 0 $$
But, if this term goes to $0$, the whole equation would become the following
$$ V(S_t) \leftarrow V(S_t) - \alpha V(S_t) $$
This would mean we decrease the value of this state.
But this doesn't make any sense to me, if the 2 policies are very different, we gain little to no information. Therefore, I would assume the value would be unchanged instead of decreased.
What is my misconception here?

This would mean we decrease the value of this state.

Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy selects an action that is more likely in the target policy than in the behaviour policy. This over-estimate must happen, if your agent experiences some actions with a likelihood ratio close to $0$, it must also experience some actions with a likelihood ratio greater than $1$. This follows because $\mathbb{E}_{\mu}[\frac{\pi(a|s)}{\mu(a|s)}] = 1$*, provided the behaviour policy covers the target policy (behaviour policy has non-zero probability for all actions where the target policy has non-zero probability).
Those actions with a likelihood ratio greater than $1$ are not actually better than before due to being taken off-policy, so their value used in the update will be an over-estimate.
It is only in the limit of large numbers of samples that the adjusted value function will converge on a good estimate of the value function for the target policy.
Basic importance sampling can have problems with increased variance due to this effect. In fact the variance can be shown to be unbound/infinite in some cases. I am not sure if it is the case in TD learning, but definitely the case in Monte Carlo with importance sampling. TD(0) does benefit from reduced variance from bootstrapping, so it probably doesn't have unbound variance. Still, it has higher variance than on-policy TD(0).

*
For any fixed state, $s$,
$$\mathbb{E}_{\mu}[\frac{\pi(a|s)}{\mu(a|s)}] = \sum_a \mu(a|s) \frac{\pi(a|s)}{\mu(a|s)} = \sum_a \pi(a|s) = 1$$

Great explanation, thank you very much.
So it's only when we consider multiple updates it's starting to be a correct update.
If we'd have only 1 or 2 it wouldn't be that good.

@KoKlA Yes, although "just one or two" would also be bad in most on-policy TD(0) as well due to initial bootstrap bias. The difference with importance sampling is that you need even more samples. But the benefit is being able to separate the need for exploring from the need to learn a specific target.

I don't agree on your explanation on the overestimation, because in the tabular case, there is no such thing as overestimation, it's not like Q learning

Say I throw a coin, and play head with P=1, then the value will be $E[Q(x)] = p(H)Q(H) + p(T)Q(T)$, and if I leave a friend that plays heads and tails with P=0.5 I get $ = E[p(x)/b(x)Q(x)] = p(H)Q(H) + p(T)Q(T)$

thus given that he sometimes plays tail, I should not consider any of its rewards coming from them, thus, makes no sense that we should "reduce the overestimation"... instead, in my opinion, the importance sampling ratio should stay in front of the whole update, having $V = V + \pi/b[TD]$

@AlbertoSinigaglia I don't follow your comments, and this answer is from almost 2 years ago, so I am not in the correct space to pick it up and figure out what you are saying. If you are disagreeing with D.Silver's slide, I'm willing to bet you are wrong. If it is just my explanation, perhaps I am wrong, but I cannot figure out what you are trying to say. I suggest ask a new question on the site, and give more details. Don't use "me and my friend" as examples because it is not clear what the behaviour and target policies are - use "behaviour" and "target".

@NeilSlater you are right, maybe a different question and not using friends analogies might be better, sorry for that... let's consider then what Sutton published https://icml.cc/2012/papers/268.pdf page 4 end of first column: (given $u$ being the value function parameters) $u_{t+1} - u_{t} = \alpha\rho \psi (R - V)$, which simplified becomes $u_{t+1} - u_{t} = \alpha \frac{\nabla\pi}{b}(R-V)$... thus the value update is the regression error times the importance sampling ratio, and considering that $E[R] = E[r_t + V(s')]$ we can change it in the parenthesis which brings to my point

@AlbertoSinigaglia Yes, this answer is about correct sampling to meet the expectation, and what would happen in unweighted importance sampling if you neglected to include the zero target updates. This would result in an overestimate of the maginitude of the return - the zero samples are required for the expectation and sampling to match. Please do ask another question on the site if this is not clear

@NeilSlater done https://ai.stackexchange.com/questions/41284/correction-of-value-update-in-off-policy-td0


text:
How to arrange test dataset distribution for an imbalanced classification problem?
I have a dataset that contains 560 datapoints, and I would like to do binary classification on it. 400 datapoints belong to class 1, and 160 points belong to class 2. In the case of an imbalanced dataset like this, how to arrange the test dataset to get valid performance results? Should I keep the same imbalanced data distribution for the test set which is similar to the distribution of the training data, or arrange it in a way that half of the test points belong to the first class and the remaining half belongs to the second class?

We had similar questions in the past. For example, see here.

The test set should represents the "real" data distribution your model will tackle once deployed and used in real applications. So the quick answer is yes, the test data should be imbalanced, which comes also as a sort of forced choice for you considering the super small size of you dataset.
You want to keep in mind though that this makes a bit trickier to interpret the resulting metrics.
For example, if your test set is made of 90 instances labeled A and 10 instances labeled B, a model might predict class A for all instances and still have a final accuracy of 90%, correct, but completely unrepresentative of the real behavior of the model. So make sure to compute metrics like precision and recall for each class.
Extra: considering the size of your dataset, you might consider using k-fold cross validation. Split your dataset into k train-test subsets, train k models and average the resulting metrics. The logic being that with such few instances the dataset is almost guaranteed to contain biases of some sort.

If the real distribution is imbalanced, the test dataset should be imbalanced. However, note that, even if the training dataset is imbalanced (which is the case of the OP), this does not mean that the real distribution is imbalanced. So, if the real distribution isn't imbalanced, then one may want to consider oversampling and undersampling.


text:
Do I need to tune the hyper-parameters or more data if SVR model performs poorly?
I am using non-linear data to SVR and have tried tuning the hyperparameters and still have a poor model performance. Do I need more data or format the data for more suitable results?
I get similar performance for ANNs, decision tree, and random forest (slightly better) and even negative for polynomial regression.
The graphs for test data performance and training data also get a DataConversionWarning
You can find the data I used here
The plots I obtained look like this:
actual vs predicted for test data
actual vs predicted for training data
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR
from sklearn.metrics import r2_score
#
#
dataset = pd.read_csv('Data.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, -1].values
#
#
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)
#
#
regressor = SVR(kernel = 'linear', gamma='auto')
regressor.fit(X_train, y_train.ravel())
y_predict = regressor.predict(X_test)
np.set_printoptions(precision=2)
print(np.concatenate((y_predict.reshape(len(y_predict),1), y_test.reshape(len(y_test),1 )), 1))
#
#
r2_score(y_test, y_predict)
#
#
#model performance
fig, ax = plt.subplots()
ax.scatter(y_test, y_predict)
ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)
ax.set_xlabel('Actual')
ax.set_ylabel('Predicted')
#regression line
y_test, y_predict = y_test.reshape(-1,1), y_predict.reshape(-1,1)
ax.plot(y_test, regressor.fit(y_test, y_predict).predict(y_test))
ax.set_title('Final Prediction-R2: ' + str(r2_score(y_test, y_predict)))
plt.show()
#
#
#training data performance
fig, ax = plt.subplots()
ax.scatter(y_train, y_train_predict)
ax.plot([y.min(), y.max()], [y.min(), y.max()], 'k--', lw=4)
ax.set_xlabel('Actual')
ax.set_ylabel('Predicted')
#regression line
y_train, y_train_predict = y_train.reshape(-1,1), y_train_predict.reshape(-1,1)
ax.plot(y_train, regressor.fit(y_train, y_train_predict).predict(y_train))
ax.set_title('Final Prediction-R2: ' + str(r2_score(y_train, y_train_predict)))
plt.show()

You are using a linear kernel. Why would you expect anything good out of data you know are non-linear? Try different kernels first. Those are what makes support vector machines/regressors non-linear.

I already did try rbf, poly and linear kernels and got the best score of 0.42 with rbf kernel.  regressor_linear = SVR(kernel = 'linear',C=100, gamma='auto') regressor_rbf = SVR(kernel = 'rbf',C=100,gamma=0.1, epsilon=0.1) regressor_poly = SVR(kernel = 'poly', gamma='auto', degree=3,epsilon=0.1, coef0=1) regressor_linear.fit(X_train, y_train.ravel())

Looking at your data, I see that it's only 126 samples long. For datasets of that size, classical statistical methods outperform all ML methods. Look at this paper, where they use the 1045 M3 univariate time series to show that old school statistical methods always outperform machine learning methods for short sequences.
Sure, your data is not univariate, but the argument still stands; Your dataset is nowhere large enough to justify using ML.

That reference paper gave a valid point, I'll work on applying statistical methods and then can compare with the ML models too. Could a larger dataset hopefully give better result with the above code?  Can I use statistical methods/above code to predict multiple outputs? example: dataset = pd.read_csv('Data.csv') X = dataset.iloc[:, 0:3].values y = dataset.iloc[:, 3:9].values If you could do so, please provide me with more references that might help with my work.

@TaqiAhmed Multiple outputs can easily be dealt with by creating multiple models; That's how many SK-learn models actually do it under the hood.
Multiple inputs is much more of a challenge; Look at something like ARIMAX and Vector Autoregressive models.


text:
Best practice for handling letterboxed images for non fully-convolutional deep learning networks?
I'm working on a depth estimation network.  It has two outputs:

A relative depth map
A scalar for scaling the relative depth map into an absolute depth map.  This second output uses dense layers so we cannot use variable-sized input.

We are trying to handle two different dimensions (192x256 and 256x192). The current approach is to letterbox the image, meaning apply black on the image so that it comes out to 256x256. We decided on this approach instead of center-cropping images to 192x192 because we believe we may lose valuable data with cropping.
When using letterboxes, I see two paths:

Ignore the letterbox portions of the image in my loss function.  The loss function will only perform calculations on the original portion of the image.
Set a static value for the letterbox portion and include it as part of the loss.

Is #1 the correct approach?  The network will then be able to predict any depth value for the black letterbox portions without being penalized. I'm concerned with #2 about confusing the network between the letterbox portion and actual dark portions of images.

Padding is indeed the easiest solution. And if no bias is used then masking the extra values during the loss computation is also not necessary, since it's enough to use zero as padding value.
You might be interested though in checking Spatial Pyramid Pooling. This pooling method allows to combine fully convolutional modules and dense layers, i.e, it can be initialized to produce a specific fixed output size while allowing varying input sizes, for both, training and inference.

Thanks for the answer and for the Spatial Pyramid Pooling recommendation!


text:
CNN Architectures for local features vs global context
Kaparthy in his blog post said

[this] hints at the kinds of architectures we’ll eventually explore. As an example - are very local features enough or do we need global context?

I'd like to gain expertise in designing networks for local vs global features. Obviously increasing the receptive field of neurons (though more pooling/striding and bigger kernels) will allow to network to take more global context into account.
Could someone point me in the direction of some readings on the topic?
Are there any specific architectures targeted to local features or to global context?

Well, in regards to properties of CNNs in regards to local versus global features, you should familiarize yourself with the concepts of invariance and equivariance. At some point you should also learn about the Picasso problem which is a consequence of the invariances and equivariances of CNNs + pooling.
That will probably also mean you'll encounter Capsule Nets at some point. While not really CNNs, learning about CapsNets can elucidate the weaknesses of convolutions since CapsNets were made to solve several of those.
There are CNN architectures that, in parallel, use different scales of local features, such as the Inception architecture and ResNext; Both combine local features on different scales, i.e. they use differently sized kernels in parallel to improve classifications.


text:
How to handle an unbalanced dataset when training object detection algorithms?
I am training an object detection model, and I have some very highly unbalanced data annotations. I have almost 11,000 images, all with dimensions of 1024 $\times$ 1024.
Within those images I have the following number of annotations:
*Class 1 - 40,000
*Class 2 - 25,000
*Class 3 - 900
*Class 4 - 500

This goes on for a few more classes.
As this is an object detection algorithm that was annotated with the annotation tool Label-img, there are often multiple annotations on each photo. Do any of you have any recommendations as to how to handle fine-tuning an object-detection algorithm on an unbalanced dataset? Currently, collecting more imagery is not an option. I would augment the images and re-label, but since there are multiple annotations on the images, I would be increasing the number of annotations for the larger classes as well.
Note: I'm using the Tensorflow Object Detection API and have downloaded the models and .config files from the Tensorflow 2 Detection Model Zoo.

One thing to try first is Focal Loss. This particular loss works well for classification or object detection where your dataset is unbalanced and contains many classes. In short, the loss suppresses highly confident predictions and gives the model more room to learn from other less confident classes.
You can read this blog to have more intuition about focal loss.


text:
Do LSTM in tensorflow work sequentially or in parallel
I have a basic understanding how a cell and a layer of an LSTM works. However, I get confused by what "number of units" (as termed in tensorflow) exactly means. A unit is, as far as I understand one "instance" of a LSTM, consisting of $t$ cells, for a sequence of length $t$. When I have more than one unit, do these work in parallel (i.e. 10 units not interacting with each other) or sequentially (ouput of unit 1 is the input of unit 2 and so on).

IIRC a unit is a neuron.

The layer contains multiple parallel LSTM units (hidden states in tensorflow and dimensionality of the output space); they are identical and in parallel learn different things.


text:
Where can I access this research paper on Frechet distance score?
Frechet Inception Distance is a metric that calculates the distance between feature vectors calculated for real and generated images. It is used in evaluations how good the generated images are.
Consider the following citation of the research paper I want to study in detail, which I think is the first paper on Frechet distance

Fréchet, Maurice. "Sur la distance de deux lois de probabilité."
Comptes Rendus Hebdomadaires des Seances de L Academie des Sciences
244.6 (1957): 689-692.

I have no clue on where to access the paper.
In general I get PDFs of almost any research paper due to my institute subscriptions in various publishers. But, I cannot see the pdf or contents of this research paper anywhere.
What can I do for accessing this paper?

The linked paper cites another paper "The Fréchet distance between multivariate normal distributions.", which may be more relevant to you than the paper you're looking for (which is in French). Anyway, even the Wiki article on the Frechet distance mentions the related Frechet Inception Distance here, and there's also a main article dedicated to the FID distance.

The first reference in the Google was link to that paper : https://www2.sonycsl.co.jp/person/nielsen/infogeo/Seminar/Frechet-Fondamental-Distance-Wasserstein.pdf

Please check the title. Are they same?

@hanugm, yes, as far as I understand, maybe the location of title was a bit tricky

And the language is alien to me.

@hanugm - I am not familiar with French, too. However, some claims are clear due to the similarity between European languages. Seemingly, OCR + NLP model can make a reasonable translation :)


text:
Do authors generally use fully connected layer instead of affine transformation?
We generally encounter the following statement several times

The input vector is first fed into a fully connected layer......

Since linear activation functions, such as identity function, can so considered as an activation functions, a fully connected layer can be considered just as an Affine transformation if the fully connected layer uses linear activation function.
So, in theory, a fully connected layer can refer to the following

Just an affine transformation
Affine transformation followed by a nonlinear activation function

Do authors generally choose to use "fully connected layer" for case 2 only or for both cases 1 and 2?

Affine transformations have this property that multiple affine transformations are equivalent to one affine transformation, so you gain nothing by stacking such layers.

@user253751 consider a single one.

According to my experience, it refers to the 2nd case. When you're talking of neural networks, it's implicitly assumed you use a non-linear activation function at the end of each layer.

Yes, typically, a fully connected layer is an affine transformation, which can or not be followed by a non-linear activation function, but, in many (if not most) cases, it's followed by a non-linearity, such as ReLU, sigmoid, or tanh (an exception is when you do regression), which is what makes the neural network be able to approximate non-linear/complicated functions.