text:
Deep Learning Architecture where outputs from two different inputs are used for error calculation
Is there a deep learning architecture where outputs of the same model with two different inputs are used for error calculation (backpropagation)?
Workflow:
Input1 -----> Model ------> Output1
Input2 -----> Model ------> Output2
Loss = criterion(Output1, Output2)
Backpropagation(Loss)

There is at least one model training process that I can think of that works like this in practice. I am not sure what the downvote was for, but perhaps you could explain what your original problem is, and what you want to understand about the model. Otherwise the direct answer to your question is "yes" with no details.

There are known architectures to implement this idea, namely, seamese networks, and also a training strategy, known as contrastive learning, that relies on the idea of comparing the output of neural networks. I will explain both of them briefly.
The idea of seamese networks is exactly what you mentioned. You have a single model, $m$, that receives two inputs, $x_{1}$ and $x_{2}$. Uppon the application of $m$, one has $h_{i} = m(x_{i})$.
Now, seamese networks have been employed in at least two situations that I know of. The first one is known as Contrastive Learning, and seeks to learn a classifier from pairs of examples. A pair $(x_{i}, x_{j})$ is said to be a positive pair if they belong to the same class, otherwise it is called a negative example. The idea is to encode both examples with $m$, then comparing the learned representations through a similarity measure $sim(h_{i}, h_{j})$. As an example of similarity measure, take the inner product,
$$sim(h_{i}, h_{j}) = \sum_{k=1}^{p}h_{ik}h_{jk}$$
Using this, the contrastive loss reads as,
$$\ell(h_{i}, h_{j}) = \dfrac{exp(sim(h_{i}, h_{j}) / \tau)}{\sum_{n=1}^{N}exp(sim(h_{i}, h_{n}) / \tau)}$$
where $\tau > 0$ is known as the temperature, in allusion to Statistical Mechanics. The principle behind this type of learning strategy is the the model $m$ will learn to encoder inputs in such a way that positive pairs (samples from the same class) are mapped close together in the latent space, whereas negative pairs will be far apart. For reference you can consult [He et al., 2020]
The second application that I know of is metric learning. In this case, suppose that you have a metric $d$ that is useful, but very costly to calculate over the input space (e.g. $d(x_{1}, x_{2})$ takes a lot of time to calculate). This is the case, for instance, with the Wasserstein distance. The idea is to precompute a bunch of values $d_{ij} = d(x_{i}, x_{j})$, and to train a seamese network so that $\lVert h_{i} - h_{j} \rVert \approx d_{ij}$, that way we may measure $d_{ij}$ by computing the Euclidean distance over the latent space. This is roughly the idea behind the work of [Courty et al., 2017]
References
[He et al., 2020] He, Kaiming, et al. "Momentum contrast for unsupervised visual representation learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[Courty et al., 2017] Courty, Nicolas, Rémi Flamary, and Mélanie Ducoffe. "Learning wasserstein embeddings." arXiv preprint arXiv:1710.07457 (2017).


text:
Can I help my neural network if I know the sign of the relationships between inputs and outputs
I am attempting to train a neural network where I can say the following:
For most inputs, I know the sign of the relationship between that input and several specific outputs. I.e. whatever set of values the inputs are set to, I can point to individual input and say that so long as the other inputs are unchanged, the output should monotonically increase/decrease as I change this input's value.
This is not to say that the inputs don't interact with each other, they do, but the interaction never changes the sign (i.e. increasing vs decreasing) nature of the monotonic relationships between specific inputs and outputs.
I am wondering how I can use this information to help my network learn fast and/or perform well.

Can you provide some more information about what your network needs to predict?

Also, more information about the problem in general would be helpful. Like how do you know which inputs/outputs have the monotonic relationships?

@respectful: I have 21 inputs and 3 outputs. Out of that I know with confidence the sign of a monotonic relationship between 18 of the inputs and all of the outputs. But due to a confidentiality agreement I can not say exactly what the problem is. I have in the ballpark of 100k input/output pairs for training.

If you are familiar with latex could you formalize the problem statement within the question.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
What is all necessary types of data for a bidirectional RNN to learn embeddings?
Bidirectional RNNs are used for generating the semantic vectors of the text at the sentence level and word level.
In order to train a CNN for the classification tasks, images, and labels/outputs are required in general.
If the data required is dependent on the algorithm used then please consider the algorithm that uses less possible types of data. Suppose if we need to perform a classification task on images: then in general, images with corresponding labels are expected. But, there may be algorithms that also work better with fewer labels available. But, the types of data in both the cases are the same: Images, labels.
I want to know such bare minimum types of data requirements for a bidirectional RNN. Afaik, a text file is enough for it to get semantic vectors since learning in text processing generally happens based on the distributional hypothesis.
Am I correct? If not, then what should be the other necessary data requirements for a bidirectional RNN to generate semantic embeddings?

What do you mean by "distributional hypothesis"?

@nbro It means the words with similar meanings occur in similar contexts.

Ok. I had heard of it before, but I didn't recall it. It may be a good idea to provide some reference that defines this term. Anyway, this question seems to be a bit vague because there are specific techniques to learn embeddings, like word2vec. So, asking what is necessary to learn the embeddings could potentially have many answers, I think. Do you have in mind a specific example (link to code or paper) where embeddings are learned in the context of bidirectional RNNs?

I am asking what is the necessary data/information required by a bidirectional RNN. I am not asking about other techniques for word embeddings.

Suppose images and labels are necessary (bare minimum) for a CNN to perform classification task @nbro

Yes, but to learn the embeddings you need some technique, right? It's not just a model. Of course, the data is needed, like in every machine learning technique, but I don't understand what you're asking here then, if not something like how to process the data with the bidirectional RNN to learn the embeddings. Typically, in ML, you need data + algorithms + models. You have the data, you have the model (bidirectional RNN), then the only thing that is missing are the algorithms from your question.

@nbro So, is it true that the algorithm influences the data requirements in the case of RNN? In CNN,  I can see that this is not the case as the task alone influences data requirement.

@nbro I think the question is clear now. Previously I did not use the word data. So, it might be confusing.

Well, now, I am not sure why you're asking for the "data requirements" or "what is the necessary data?". Are you asking if we need more than one file to learn the embeddings? It's still not clear to me what do you mean by "data requirements" here. Regarding the algorithm affecting the data. Some algorithms may converge faster than others. So, in this sense, they may require more or less data than others.

@nbro I am asking for bare requirements only for the algorithms that require minimum data.

I still don't understand. Minimum requirements in terms of data, but in terms of the size of the data or what? I don't understand.

Consider an analogy with CNN. For the classification task, what data do we require? dataset of images along with labels right? Similarly for the word embedding task using RNN, what data do we need? is the question. Size or any other aspect is immaterial for the question. @nbro

That's why the algorithm is important. Not all CNNs are trained in a supervised way, i.e. with labels. This question needs to be more specific. You cannot just say "what are the data requirements for learning word embeddings" as there are different approaches for word embeddings that probably use different ways to learn from data, some labelled and maybe some unlabelled. Do you see now the problem with your question?

@nbro I got it now. Changed the question accordingly, please check now whether it is clear or not.

Ok, let's put it in that way. What would be an example of an answer you're looking for? Are you looking for an answer like "What you need is just unlabelled text. It doesn't matter the size, but is probably better because of reasong X"? By the way, I wouldn't call "labels" a type of data. Labels are part of the data that you have or not. So, you can have labelled data or unlabelled data.

@nbro The primary intention is to know what is all data needed for classical algorithms for generating word embeddings using bidirectional RNN. Yes, I am waiting for an answer like that. If an answer says the algorithm name along with data expected, then it would be great.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Is the phrase "Feature Pyramid Network" refer to CNN only?
"Feature Pyramid Network" is a network that is used for feature extraction. Since it is pyramid in shape, it might be called so.
Consider the following excerpts from two different sources
#1

A Feature Pyramid Network, or FPN, is a feature extractor that takes a
single-scale image of an arbitrary size as input, and outputs
proportionally sized feature maps at multiple levels, in a fully
convolutional fashion. This process is independent of the backbone
convolutional architectures. It, therefore, acts as a generic solution
for building feature pyramids inside deep convolutional networks to be
used in tasks like object detection.

#2

FPN composes of a bottom-up and a top-down pathway. The bottom-up
pathway is the usual convolutional network for feature extraction. As
we go up, the spatial resolution decreases. With more high-level
structures detected, the semantic value for each layer increases.

Both the sources are giving a strong sense that the phrase "Feature Pyramid Network" must be used for CNN's only as it is used mainly intended for object detection. But the name and purpose suggest to me that any ANN that is pyramid in shape can be attributed as "Feature Pyramid Network"  since any ANN tries to extract features only in the general sense.
Am I true? Can I use the phrase for any arbitrary ANN that is pyramid in shape or is it an exclusive term of CNNs in computer vision only?

No, Feature Pyramid Networks does not only refer to CNNs.
Take the recently trending "Swin Transformer" as an example.

Swin Transformer is a variant of vision transformers, and they do not employ convolutional layers. However, their design allows them to output feature maps in multiple scales (feature pyramids). The authors actually used the feature pyramids for image segmentation tasks using UperNet.
The reason why people use feature pyramids is to use both local and global representations (mostly in images), and CNNs have a structure that naturally outputs such feature pyramids. However, this does not mean that feature pyramids are constrained to only CNNs.

I am not an expert on this topic, but I heard of "Feature Pyramid Networks". Do the authors in the Swin Transformer call their approach "Feature Pyramid Networks". Maybe what you mean is that similar approaches to "Feature Pyramid Networks" have also been developed with other layers? Because the "original" FPNs seem to have been proposed here, so it's possible that some people call FPNs only those that are consistent with that paper, or the term may have started to be used more loosely for similar ideas.

If I remember correctly, related ideas were already developed a long time ago, with SIFT. Again, I am not an expert in FPN, so it's possible I am making here some wrong assumptions.

I agree that FPN is a loosely used term, but the main idea is to aggregate hierarchical feature maps regardless of the backbone. In Swin Transformers, the authors used Swin Transformers as the backbone to extract their hierarchical feature maps.
I found a quote from the paper
"With these hierarchical feature maps,
the Swin Transformer model can conveniently leverage advanced techniques for dense prediction such as feature pyramid networks (FPN) [42] or U-Net [51]."

In that sentence, it seems that they are saying that their approach can use/leverage FPNs, so this suggests that they not the same thing.

hanugm asked whether the phrase "Feature Pyramid Network" must be used only for CNN's or not. My answer is that it is not only applicable for CNNs but any networks (like swin) that outputs hierarchical feature maps.


text:
How can I adapt a trained neural network model to learn from newer data containing additional features?
We shall assume that we have a trained neural network model for some task $A$. The dataset used to train the model contained some $n$ features per sample. Using this dataset, we were able to train a classifier.
After some time, I get some more data for the same task $A$, but with additional features (all relevant). I want to use this new data to improve my existing classifier by utilizing the weights. (whatever it learned from the initial dataset).
This seems very similar to the problem of transfer learning, but my aim is to use an existing model architecture without modifying it too much and make it a better classifier for the same task A without having to do any retraining. (We can assume that data is lost after training, and all we have are the parameters/weights of our model)
After doing some reading, this could be a use case of continual learning but most of the work is for training an existing model for different tasks. I am not able to fully wrap my head around how I could use the same model for the same task, but instead, the structure of the data is changing over time. Can you also refer me to any literature in this space, if any?

When you say that the data is lost, isn't there a way to record it while you're training (e.g. in a database?). That would make the setup simpler as you'd be able to finetune your model on both the old and the new data.

What kind neural network do you want to train? Like CNN, RNN, ResNet, LSTM, GRU, Super Resolution, GPT, GAN and etc. I think suggestion would be widely different depending on neural network structure/model.

To make the dynamic amount of features easier to work with, you can model it as a sequence modeling problem, where for the new task your sequence length increases. (where each "timestep" is a single feature)
Continual learning literature might still be the right place to look, as they try to solve the same problem of catastrophic forgetting that your problem runs into.
Solutions might work are generative replay (catastrophic forgetting literature). Basically a generator that generates samples as if you're sampling from your dataset. Just combine the samples from the generator with the new samples for A and train on that.
So you will have short sequences from the training dataset generated by  the generator and your longer sequences (longer because of the additional features) in your current training set.
Arxiv:1705.08690 (Continual Learning with Deep Generative Replay)
Or meta learning how to incrementally learn. (might be able to make this work for your problem but generative replay is definitively the easiest)
Arxiv:2002.09571 (Learning to Continually Learn)
Arxiv:1909.00025 (Meta-Learning with Warped Gradient Descent)
Where the whole idea generally is to meta learn another network (that controls the main network) or have meta learned "fixed" layers that make sure that when you update the main network with new information the old information isn't lost.
So you can learn incrementally without (or at-least less) catastrophic forgetting.


text:
How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in the episode trajectory. If there is a single agent, then it is straightforward to pass the reward values based on the action performed by the agent in a particular state. But, in the case of multiple players, action by any player $k$ may need the updation of reward for any other player $\ell$. In such cases, I have doubts about passing rewards for player $\ell$.
Suppose, in the state $s_0$, player 1 did an action $a_0$ and causes the reward of $r_1$ to player 1 and -1 to player 2 and leads to state $s_1$. Then while it's turn, player 2 performs some action $a_1$ on state $s_1$ which caused a reward of $r_2$ to it. Then which trajectory should I need to pass among the following?

$s_0a_0r_1s_1a_1r_2 \cdots$
$s_0a_0r_1s_1a_1(r_2-1) \cdots$

I am guessing the second one. But my doubt is that the reward $r_2$ is due to the action of player 2 and $r_2-1$ cannot be viewed as an immediate reward since $-1$ is due to the action of player 1 in the previous time step. How to pass the reward values in this case?

Let me try to understand if I understood correctly what you're trying to do. You're trying to apply REINFORCE to a two-player game. Both players don't take an action at each step, but you make one player take an action at step $t$ and the next player at step $t+1$, and so on. From your attempt to define the reward $r_2 - 1$, it seems that you're also trying to penalise the second player based on the past actions of player 1. But why don't you also penalise player $1$? Are you trying to find a policy only for player 1? It's not clear what kind of policy you're trying to find here...

... because it seems that you're taking trajectories that take into account the moves of both players in order to update the policy, but then you only seem to penalise one player. I am not sure if this approach makes sense. So, maybe you should try to clarify what your policy is supposed to do.

But why don't you also penalise player 1? It is just an example. Player 1 may also get penalized in later steps due to the action of player 2. Are you trying to find a policy only for player 1?. No, I am not trying to find the policy for a single player. I want to update the same NN for all players. @nbro

Important: Is the game a zero-sum game? Chess is, for example, it has simple "I win, you lose" combination. If it is a zero-sum game, then any collection of "points" along the way is secondary. You can treat gaing game assets (such as cash in monopoly) as a reward, but it may make more sense to treat it as state information, so you can use the zero-sum end result to drive the rewards, if all that matters is who wins at the end.

@NeilSlater Do you mean I should allot/fill reward values in the episode trajectory after the completion of the chess game based on the winner/loser?

That's not what I said, I was asking a for clarification about your scenario. However, that may be the best answer if you really do have a zero-sum game, and don't need to do any reward shaping.

Yeah, the game is a zero-sum game. I updated it now. I don't know prior that zero-sum property affects the way of providing rewards in the episode trajectory @NeilSlater. If possible, please provide or tell me how to fill rewards after the completion of the game in the trajectory. Should I feel 1 for all actions by the winner and -1 for all actions by the loser in the trajectory?

@hanugm: zero-sum implies $r_{a,t} = -r_{b.t}$, which most importantly allows you to have a single model for both players (one of your stated goals). Technically you could combine player policy models for non-zero-sum games into a multi-headed network - there are a few different ways to model game playing. I am not sure of the consequences for training a single model using REINFORCE this way.


text:
What does it mean by bottleneck and representational bottleneck in feedforward neural networks?
Consider the following paragraph from section 2: General Design Principles of the research paper titled Rethinking the Inception Architecture for Computer Vision

Avoid representational bottlenecks, especially early in the
network. Feed-forward networks can be represented by an acyclic graph
from the input layer(s) to the classifier or regressor. This defines a
clear direction for the information flow. For any cut separating the
inputs from the outputs, one can access the amount of information
passing though the cut. One should avoid bottlenecks with extreme
compression. In general the representation size should gently decrease
from the inputs to the outputs before reaching the final
representation used for the task at hand. Theoretically, information
content cannot be assessed merely by the dimensionality of the
representation as it discards important factors like correlation
structure; the dimensionality merely provides a rough estimate of
information content.

This paragraph warned us to avoid bottlenecks and also representational bottlenecks. What does it mean by a bottleneck of/in a neural network and representational bottleneck?

I don't feel like providing a formal answer because I am not sure whether these terms "bottlenecks" are formally defined. In fact, we had similar questions in the past,  here or this (actually, I think your question is a duplicate of this last one), but it seems to me that these terms have been used more loosely. Having said that, I think the authors of that quote are using the term "representational bottleneck" as a synonym for just "bottleneck".

Yeah, @nbro. After understanding the terminology regarding this, I also feel that this is a duplicate one. But, while asking the question, I thought that the word bottleneck may be used for different purposes and hence I asked.

what-are-bottlenecks-in-neural-networks?

A bottleneck layer is a layer that contains few nodes compared to the previous layers. It can be used to obtain a representation of the
input with reduced dimensionality.

If you create bottlenecks in the initial layers it can cause loss of information.
Why it is done?
Take the example of an image dataset that contains high-resolution images. High resolution means more pixels means it will need more nodes in the input layer.
Having more nodes will need more computational power to train the network. Hence in such cases, we can use fewer nodes in the next layer. As images are high resolution we might not lose any important information.

You should at least acknowledge the source from which you took that excerpt, which seems to be this answer. So, please, edit your post to provide a link to that answer or the source where you took that quote from. In general, you should not just copy and paste content from places without citing, as plagiarism is at least discouraged and at most not allowed here. Having said that, your answer seems to be partially inconsistent with the quote that the asker provided because it states that the dimensionality is not sufficient for "information content".


text:
is there any proof that metric learning cannot achieve better on image classification task than accepted models (resnet etc)?
Everything is in the title.
Metric learning seems to be closer to our way of thinking than the best performing models (supervised learning CNNs-based models like resnet or efficientnet). I was looking for research papers that would have try a metric learning-based model for classic classification task on the Imagenet dataset benchmark but I could not find it. That is why I am asking the question here.


text:
Parallelize Backpropagation - How to synchronize the weights of each thread?
I implemented a parallel backpropagation algorithm that uses $n$ threads. Now every thread gets $\dfrac{1}{n}$ examples of the training data and updates its instance of the net with it. After every epoch the different threads share their updated weights. For that I simply add the weights of the threads and then divide each weight by the number of threads. The problem now is that the more threads I use the worse the result. For me this means that my way of synchronizing the threads is not as good as it should be.
Is there a better way to do it?

After a whole epoch, with multiple update steps, the neural networks in each thread will have diverged in a way where it may not make sense to take means of the weights. Ideally you should be combining data for each update step. In turn that means you will want to avoid making updates on every example, because the overhead of starting, stopping and combining the threads may lose most of the benefits.
It is common in neural networks to use mini-batches (larger than 1, smaller than the whole dataset), to get more accurate gradients, and for parallelisation. There is often a sweet spot in terms of learning speed (or sample efficiency) with some size of mini-batch. Each mini-batch calculates gradients for all examples, combines them into a mean gradient, then performs a single weight update step.
Use your threads to calculate the gradients for a mini-batch, divided up between the threads, and average the gradients across all threads in order to make a single shared weight update. Using larger mini-batches will make more efficient use of multiple threads, but smaller mini-batches can be beneficial because you get to make more weight updates per epoch.

I am not sure if I really got what you mean: The adding and dividing by n to synchronize the weights is correct but I should use mini-batches (= subset of the training set) instead of the whole set. Does this mean I use more threads or do I just use like 50% of the data in a thread per epoch?
This "sweet spot", how do I know where it is and is there a specific strategy to find it?

@CptK: You will need to experiment with suitable mini-batch sizes, and these will vary depending on the problem. You may also find that the approach that makes most efficient use of threading is not the fastest overall, and there may even be measurable differences in learning metrics that mean the neural network has best results when neither fastest nor most efficient (although IME above a relatively small mini-batch size, the batch-size choice is mainly about speed).


text:
How does back propagation adjust the hidden layers' weights and biases?
I'm new to neural networks and trying to figure out its fundamentals but I cannot fully understand the back propagation algorithm.
In back propagation, I understand we want to go backwards from the last neurons to adjust the weights and biases to that predicted final neurons. For it to calculate error and derivative, It needs to have the last inputs, the predicted output based on the layer's weights and the actual value ( target ).
As In the final neuron layers we have all this information. But how do we calculate the inputs of middle and hidden layers?

Suppose we have the final output ( 0.73 ), we calculate the error and derivatives of W31, W32 and W33; and adjust them to match the final output, Then we shall go one layer back in our network.
Now we need the N11, N12, N13 and N14 values and the target values of N21, N22 and N23 to calculate errors and derivatives, but we don't have them
Should we feed forward the whole network and map all the labels and values of each neuron in memory to be able to access it later? Because it would be very, very memory and resource intensive on large networks.

You can take a look at this site: backpropagation process There is a whole process of backpropagation with examples.

These two videos give a good intuition about how backpropagation works: https://www.youtube.com/watch?v=Ilg3gGewQ5U https://www.youtube.com/watch?v=yXcQ4B-YSjQ The second video also focuses on how it would be implemented in code.


text:
What is the fundamental difference between the synthesis task and sampling task?
Among the list of tasks in machine learning, synthesis and sampling is one of the key task. Consider the following explanation regarding synthesis and sampling task from  Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

In this type of task, the machine learning algorithm is asked to
generate new examples that are similar to those in the training data.
Synthesis and sampling via machine learning can be useful for media
applications when generating large volumes of content by hand would be
expensive, boring, or require too much time. For example, video games
can automatically generate textures for large objects or landscapes,
rather than requiring an artist to manually label each pixel (Luo et
al., 2013). In some cases, we want the sampling or synthesis procedure
to generate a speciﬁc kind of output given the input. For example, in
a speech synthesis task, we provide a written sentence and ask the
program to emit an audio waveform containing a spoken version of that
sentence. This is a kind of structured output task, but with the added
qualiﬁcation that there is no single correct output for each input,
and we explicitly desire a large amount of variation in the output, in
order for the output to seem more natural and realistic.

The explanation does not mention any difference between the two tasks. Both sampling and synthesis, apart from the linguistic differences, I don't know any discriminating criteria, qualities or properties, that separate both tasks in machine learning.
What is the fundamental difference between sampling task and synthesis task in machine learning?

The terminologies can be confusing because of the different ways authors use them. The bottom line is this
The Synthesis task basically refers to creating or synthesizing new data. Creation of data can be purely deterministic, e.g.

.. we provide a written sentence .. to emit an audio waveform containing a spoken version of that sentence

But, a vast majority of the time, such creation has an ..

.. added qualiﬁcation that there is no single correct output for each input, and we explicitly desire a large amount of variation

which is done by a Statistical Sampling procedure. This procedure can be as simple as IID sampling from $\mathcal{N}(\mu, \sigma)$ to more complex MCMC methods.
So basically, "Synthesis" is a broader task that may contain a "Sampling" procedure.


text:
Isssue in understanding the derivation regarding mean squared error
The following derivation is taken from  Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)
I am facing difficulty in understanding the zero derivative related to minimizing gradient of mean squared error $\nabla_w  \text{MSE}_{train} = 0$

$\nabla_w \text{MSE}_{train} = 0$
$\implies \nabla_w(Xw - y)^T(Xw-y) = 0$
$\implies \nabla_w(w^T X^T-
 y^T)(Xw-y) = 0$
$\implies \nabla_w(w^T X^T-
 y^T)(Xw-y) = 0$
$\implies \nabla_w(w^T X^TXw - w^T
 X^Ty
 -y^TXw+y^Ty) = 0$
$\implies \nabla_w(w^T X^TXw - 2 w^T
 X^Ty +y^Ty) = 0$
$\implies 2 X^TXw - 2 X^Ty = 0$
$\implies w = 2 (X^TX)^{-1} X^Ty = 0$

I have had difficulty in understanding the flow of the following two lines

$\implies \nabla_w(w^T X^TXw - w^T
 X^Ty
 -y^TXw+y^Ty) = 0$
$\implies \nabla_w(w^T X^TXw - 2 w^T
 X^Ty +y^Ty) = 0$

The first doubt is about in the first two lines, it is possible only if (which I feel is untrue)
$w^T X^Ty = y^TXw$
Note: $X,y$ here refers to input and outputs of train data


text:
Is there any difference between the phrases "text representation" and "text feature representation"?
Text representation, in simple words, is representing text in sensible numeric form. You can read in detail from the following paragraph

Text representation is one of the fundamental problems in text mining
and Information Retrieval (IR). It aims to numerically represent the
unstructured text documents to make them mathematically computable.
For a given set of text documents $D = \{d_i, i=1, 2,...,n\}$, where
each $d_i$ stands for a document, the problem of text representation
is to represent each $d_i$ of $D$ as a point $s_i$ in a numerical space
$S$, where the distance/similarity between each pair of points in
space $S$ is well defined.

But, I came across the phrase "text feature representation" in research papers. Features, in general, are present in dataset. But, I think, features can be characters or words or documents or complete text (as a single feature?) in the case of the text. I am not sure about what we call features in the text.
So, I am not sure about what is meant by text feature representation. Is it the same as text representation?

I think that literature is simply inconsistent in this regard. But here's a distinction that I think helps to shade a bit of light on this question:
text representation: as you said we have to convert text into numerical variables. This term refers to general strategies to convert text into numbers, like embedding, bag of words, and so on.
text features representation: this is what we feed to a real model. The difference is that it can be a combination of different text representation. For example, I could apply a TF-IDF vectorizer to a corpus and use it to encode each word of a sentence and then use also pre-trained embedding to encode the same sentence/document, concatenating for each word both vectors. Or another commonly used strategy is to use multiple n-grams, which also consist in applying multiple text representations to the same sentence/document.

I feel that what we fed to a real model is the key point...

thanks for the feedback, I edited the answer to stress that


text:
Modelling of output neuron for mixed features?
A dataset in artificial intelligence, in general, consists of some features (say $n$). Assume that $m$ among them are output features. I want to model this function using a neural network. So, input to my neural network is $n-m$ features and output is $m$ features. My question is about the output features.
If an output feature is a continuous random variable, then its corresponding output neuron can be trained to give continuous output. Similarly, if an output feature is a discrete random variable, then its corresponding output neuron can be trained to give discrete output.
But, I never came across the features that are mixed random variables. What is the nature of the output of the neuron that is intended to give the output value for a mixed random variable, which is neither discrete nor continuous in nature?

Can you please provide an example of a feature that is neither continuous nor discrete? Because, right now, I don't understand what you mean by that. Moreover, it may be a good idea to explain why you're interested in this question. Which problem are you having that involves such "neither continuous nor discrete features" (whatever that means)?

@nbro Currently, I can find the usage of mixed random variables in theory, and maybe it is difficult to find them in an application context.


text:
Is knowing underlying probability distribution mandatory for deciding iid property of random variables?
Consider the following information regarding iid random variables

The acronym IID stands for "Independent and Identically Distributed".
A sequence of random variables (or random vectors) is IID if and only
if the following two conditions are satisfied:

the terms of the sequence are mutually independent;

they all have the same probability distribution.

Definition:

Let $\{\mathcal{X}_n\}$ be a sequence of random vectors. Let
$F_{\mathcal{X}_n}{(x_n)}$ be the joint distribution function of a
generic term of the sequence $\{\mathcal{X}_n\}$. We say that
$\{\mathcal{X}_n\}$ is an IID sequence if and only if
$$F_{\mathcal{X}_n}{(x)} = F_{\mathcal{X}_k}{(x)} \forall x, n, k $$
and any subset of terms of the sequence is a set of mutually
independent random vectors.

Thus,

iid is a property for a sequence of random variables.
A joint probability distribution function is necessary to validate whether a sequence of random variables is iid or not.

Thus, the iid property of a sequence of random variables, from 2, is entirely depending on the underlying joint probability distribution function. Am I wrong anywhere?
If I am wrong, is there any other iid property of random variables that do not depend on the underlying probability distribution function?

The point is even you know the distribution, sometimes you can't prove that the sampled data is i.i.d. or not!  (more details in https://stats.stackexchange.com/q/130381/144441). Hence, without knowing the distribution, you have less information, and of course, you can't prove any identically distributed-ness property of the sampled data.
Note that i.i.d. is mostly mentioned as an assumption that is held in the corresponding domain, and you do not need to prove it as a property.

From your conclusion, 1. is correct. But more specifically, it characterizes the nature of an underlying data-generating statistic. A table of results of dice throws is likely iid, but more significantly it is because the dice roll itself is iid.
Not really for 2. since you would be simply calculating for $P(A)P(B) = P(A,B)$ and $P(A) = P(B), \forall A, B$ in the discrete case. Since iid is defined as an iff (if and only if), this characterization is also sufficient.
Note that the iid assumption allows us to characterize the joint distribution in a certain way, which then allows us to compute it. Otherwise, the model might grow to be very complex.

"the dice roll itself is iid" - really? Could you explain what the individual dice roll is independent of and identically distributed to without referencing the other dice rolls in the table?

@NeilSlater strictly speaking it is an assumption on the nature of data generated from dice rolls. we assume the data generated from the dice roll is iid relative to its own statistics. i would believe this is the correct interpretation of the phrase, but it makes more sense.... a dataset cannot be iid relative to itself since the dataset is agnostic to itself; it can only be iid relative to its underlying statistics....

if iid is a claim made on the dataset, then the assertion would need to be reclaimed every time the dataset is appended, even if its from the same source (statistic). on the other hand, if it is a claim made on the statistic, then this is not needed...

a set of coin flips: {H, H, H, H, H, H, H}, by the calculations above, would very much not be identically distributed even though it is a possible outcome of coin flips. strictly speaking then very few datasets would be strongly iid even if sampled from an "iid statistic". it seems to make more sense to talk of iid relative to a data generating statistic than just a dataset.


text:
Can I call any function a signal?
While reading the Notation of the paper titled  Geometric Deep Learning Grids, Groups, Graphs, Geodesics, and Gauges, I came across the following notations.
$$
\Omega = \text{ Domain} \\
 u = \text{Point on domain} \\
x(u) \in \mathcal{X}(\Omega, C) = \text{ Signal on the domain of the form } x : \Omega \rightarrow C
$$
Mathematically, a signal is just a function.
But, every function may not be a signal. There may be some distinction between a mathematical function and a signal.
When can I call any mathematical function a signal? And what is $\mathcal{X}$ in the notation given?

I think $\chi$ is the set of all the signals over the domain.


text:
What is meant by inverting the generator?
Generative Adversarial Networks, in general, consists of two multi layer perceptrons: generator and discriminator. Generator is used for generating samples that are as real as training samples and discriminator tries to discriminate the real and fake samples.
Generator receives noise and generate samples.
Some papers like this use extra networks

To achieve this, one can train a convolutional network to invert
$G$ to regress from samples $\hat{x} \leftarrow G(z, \phi(t))$ back
onto $z$.

And some other papers, I remember saying that we use the same generator architecture to invert the it.
What does it mean by inverting the generator of a GAN? Does inverting means passing samples and getting noise as output by using same or different architectures?

Can you provide a link and/or quote where the term is used? I have studied GANs and not heard of anyone use the phrase "inverting the generator".

Oh, sorry. 4.4. Inverting the generator for style transfer

OK, that does look correct, but I don't understand what the problem is, that short section explains excatly what inverting is for that secnario, including the loss function. Where are you stuck in your understanding?

@NeilSlater I came across multiple ways of doing so. Here, I feel that they are not actually inverting the generator. They are inverting via another network. So, I want to know when can we say that "we are inverting the generator of GAN". Is it by using another network or as I mentioned in the question or both. I mean "is there any standard definition for inverting a generator in GAN?".

In that case you will need multiple links to show the "multiple ways" that you have found, and explain this issue in the question. That is because "inverting the generator" is not a basic concept of GANs - it is not something that can be looked up or referenced. Actually I doubt it is a standard term at all, but if you explain in more detail in the question why you have conflicting views of what the term means, perhaps someone who is more of an expert with GANs can help

If I were to guess, I would say that "inverting the generator" is creating a function that takes an image as input and outputs a vector that would create a similar image if used as input to the generator of the GAN. That is what is happening in the linked paper. Any differences will then be in the approaches used to create that inversion function. It is generally not possible to directly invert a feed-forward network by "running it backwards" - some kind of itertive process or training process will be required, or a different starting architecture.

@NeilSlater Oh! then I will try to edit today only.


text:
Is there any significance for higher order gradients in artificial intelligence?
Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks.

First order gradient: It quantifies the rate of change of a function with respect to its inputs. It is useful in artificial intelligence, especially in gradient-based algorithms, to know about the direction in which the parameters need to be updated.

Second-order gradient: It somehow quantifies the curvature of the function. It is used in artificial intelligence, to know whether the function has convex or concave portions.

In this context, I want to learn whether there is any significance for higher-order gradients in artificial intelligence? Note that higher-order refers to the order $\ge 3$.

Gradient descent presumes a Taylor Series.  They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss.
The learner as a universal function approximator means there can be many configurations that yield minimum loss, and there is usually no global "best".
One of the reasons for multiple traverses through the data using the optimizer is that the local Taylor series estimate is low order (frequently only first order), so the optimum is not apparent after a single pass.  When the weights change, the landscape changes and for a 25-million-parameter network, that change is happening in a 25-million dimensional space.
The higher order terms of the gradient can accelerate the optimization, but they can be prohibitively expensive to compute or estimate.  The Hessian is nice, but has high overhead to compute and store.  Things like conjugate gradient were popular because they are fast and (very) rough estimates of the Hessian.
If you have a cheap way to get decent higher-order derivatives, then there is value in that.

Conjugate gradient does not use an explicit Hessian, it is used for linear systems with symmetric system matrix, it only uses the evaluation procedure for the matrix-vector products, so can be used for large sparse systems where the matrix is never constructed as separate data structure. // Hessian approximations are constructed in quasi-Newton methods like BFGS. Often implemented in a way to avoid the full matrix, only storing the initial diagonal and the data for the rank-1-updates.

@LutzLehmann - good points.  I was meaning approximation in the loose and quick sense, so less bad than not having anything at all, and nothing like to within an error bound.  If you have 25 million weights, then the hessian is a matrix of size (25M)^2.


text:
Will there be any changes in the model's performance due to the usage of very small batch sizes?
I am trying to run a code that has a batch size around 28. I can run the program on my GPU with this batch size.
But, when I modify the code for my requirements and try to run, it is showing an run-time error due to insufficient memory in GPU.
I checked for possible batch-size that I can run and it is just 2-5.
I am not sure whether there is any issue if I run with such small batch sizes? I mean, will there be any performance issues keeping aside the time it takes?

there shouldn't be any significant differences. the expected value is the same....

Batch size affects how many training updates (steps) will happen during each epoch.
When the batch size is small, this means that the model sees fewer data in each weights update. Thus, your question really depends on the data you have, along with the corresponding task (classification / RL etc.)
If your data is highly imbalanced, then I would not suggest a small batch size, since the probability of seeing a positive instance would be far smaller (assuming you take uniform batches).
For an RL task, imagine using a replay buffer of past experiences and your agent had very few good action selections during the only exploration process. Then a small batch size would make the agent training very difficult, since most of the time samples with not good action selections would be seen. As a result, the agent may drift from good policies.
For a classification task, what I always do, is to make stratified batches. That is each batch has the same label percent as the whole dataset. And most of the time it works for better than uniform batches even for smaller batch sizes. For RL, I would recommend higher batch sizes or similar clever ways of sampling.

Task is generative models. I am running GANs. But, according to your answer, can I assume that very small batch sizes leads to poor results?

I would prefer bigger sizes. But I think your problem is very data-depending.

If you think I have helped you with my answer, please check it as correct.

Sure and thanks. I generally upvote and accept nowadays if I understand completely since I may be wrong.


text:
How does noise samples from uniform distribution contribute to the diversity of generator output?
In a Generative Adversarial Network (GAN), there are two multi-layer perceptrons. One is the generator network and another is a discriminator network.
The input for the generator network is a noise vector $z$. The input for a discriminator network is either a generated sample $G(z)$ i.e., the output of a generator network or a training sample $x$ for a training dataset.
My doubt is regarding the input of the generator. The noise vector is generally sampled from the standard normal distribution.
$$z \sim \mathcal{N(0, 1)}$$
Although I am not sure, I think : since the values in the normal distribution vary, the output of the generator can vary accordingly.
But some of the research papers say that the noise vector can also be sampled from a uniform distribution i.e., $z \sim \mathcal{U(a, b)}$ for $a<b$.
$$ U(x) = \begin{cases}
      \dfrac{1}{b-a} & x\in [a, b] \\
      0 & x\not\in [a, b] \\
   \end{cases}
$$
It is clear that uniform distribution does not vary like normal distribution and takes only two possible values, hence all samples have equal probability in the given range. Then how can it contribute to the diversity of the output of the generator network?

Samples from the uniform distribution do not take only two possible values. And why do you think  "diversity" requires a non-uniform distribution?

"The probability of rolling a 1 is 1/6. The probability of rolling a 2 is 1/6, and so on. So does that mean the only number I can get from a dice is 1/6?"

hence all samples have equal probability in the given range but I am confused what is the need of such constant distribution @user253751

What is the need of a normal distribution @hanugm?

@user253751 because it provides samples with varying probability values.

What is the need for samples with varying probability values? It is easy to construct varying probability values from fixed probability values using a nonlinear transformation.

@user253751 Then according to you, any random probability distribution will work. Am I right?

Probably.......

As pointed out in the comments, a random variable $z$ sampled from a uniform distribution $\mathcal{U}(a,b)$, doesn't take only two values but any value between $a$ and $b$ with equal probability $\frac{1}{b-a}$.
The distribution you use for $z$ in a GAN doesn't have a theoretical justification. The fact that it's an $\mathcal{N}(0,1)$ is never used in the proof of the convergence of the generator to the data distribution in the original GAN paper (see Section 4.1).
However, it should be easy and fast to sample from given that you must take samples at each training step. Why certain authors choose one distribution over the other is usually down to empirical trial and error. GANs are notoriously hard to train and the distribution of $z$ is one of the many hyperparameters of the model.
Other generative models though have theoretical justification for the choice of the distribution of the random variable given to the decoder. In VAEs, it is a Gaussian $\mathcal{N}(0,1)$ because the loss function contains a KL term $\text{KL}(p_\theta(z|x)||p(z))$ which has a nice closed form when $p_\theta(z|x)$ and $p(z)$ are Gaussian distributions (see Section 3 in the original VAE paper).

The main idea of the latent vector $z$ is that it represents some high-level features of images, and the interpretation could be that latent features are normally distributed.
However, since we are trying to approximate (map) a complex multimodal real distribution using a simple one, the feature space $Z$ becomes entangled. So, in practice, we just feed random noise into the generator to add stochasticity. Otherwise, the network would produce the same output each time.
As already discussed in the comments, it doesn't really matter which distribution you use. The only advantage I see of using a normal distribution is that we can use the truncation trick to get better (but less varied) results.


text:
Why does critical points and stationary points are used interchangeably?
Consider the following paragraph from Numerical Computation of the deep learning book.

When $f'(x) = 0$, the derivative provides no information about which
direction to move. Points where $f'(x)$ = 0 are known as critical
points, or stationary points. A local minimum is a point where
$f(x)$ is lower than at all neighboring points, so it is no longer
possible to decrease $f(x)$ by making inﬁnitesimal steps. A local
maximum is a point where $f(x)$ is higher than at all neighboring
points so it is not possible to increase $f(x)$ by making inﬁnitesimal
steps. Some critical points are neither maxima nor minima. These are
known as saddle points.

In short, points where $f'(x) =0 $ are called critical points, or stationary points.
But, according to mathematical terminology, the definitions are as follows:
#1: Critical point

A function $y=f(x)$ has critical points at all points $x_0$ where
$f'(x_0)=0$ or $f(x)$ is not differentiable.

#2: Stationary point

A point $x_0$ at which the derivative of a function $f(x)$ vanishes,
$f'(x_0)=0$. A stationary point may be a minimum, maximum, or
inflection point.

It can be noticed that the definitions that are given in the deep learning book do match exactly with stationary points since the only premise is $f'(x)=0$. The definition for critical point is not apt since a critical point can also be a point where $f'(x)$ is nonexistent.
Is there any reason for using the terms critical points and stationary points interchangeably? Is there no need to address the points where $f'(x)$ does not exist?

A critical point of a function $f$ can be

a stationary point (i.e. $f'(x) = 0$), or
a point where the derivative is undefined (for example, in the case of the absolute value function $f(x)$, $x=0$ is a critical point, as $f$ is not differentiable at $x=0$).

So, all stationary points are critical points.
These notes provide more examples of how to find critical points of a function, so they could be useful.

Is your writing completed? Please address, if possible, the Is there no need to address the points where f′(x) does not exist? part, which is causing issue.

@hanugm Are you asking if we can ignore the critical points that are not stationary points? If yes, in which context?

yea true, else they may not interchangeably use them.

@hanugm The point is: you can always refer to a stationary point as a "critical point" , but you cannot always refer to a critical point as a "stationary point". Some cases, where the derivative is undefined, may require a little bit of care, so you don't have to ignore all critical points. I've just read the answer by Taw, and it seems to give an example of where you need to pay attention to functions that are not differentiable at some points (e.g. ReLU).

From reading the text, it's clear that the authors are using critical point to mean the same thing as stationary point, so they are not using the proper mathematical definition.
More generally, automatic differentiation will return a gradient at nondifferentiable points. Ask tensorflow or pytorch to take the gradient of $\text{ReLU}(x)\big|_{x=0}$. They return 0, even though it should technically return NaN, since it doesn't exist. Theoretically, the question of how to deal with nonsmooth functions is well understood - see subdifferential for example. In the relu example it actually returns an element of the subdifferential but there are pathological examples where this is not true.


text:
Why data required for hyperparameter tuning is considered as an additional data?
Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following paragraph from Chapter 5: Machine Learning Basics from the book titled Deep Learning (by Aaron Courville et al.)

Most machine learning algorithms have settings called hyperparameters,
which must be determined outside the learning algorithm itself; we
discuss how to set these using additional data.

My doubt is about the usage of the word 'additional' in the paragraph. Afaik, a small part of dataset under consideration is used to validate and hence in determining the hyperparameters, called as validation data. It is also a part of the dataset as training and testing data. You can check the section 5.3 for more details.
If yes, what is the need for the usage of the word 'additional'? Is it true that data for setting hyperparameters is taken outside of the underlying dataset?

I think it just tries to emphasize that you need three, non-intersecting, chunks of datasets: training, validation, and test. So, you need some data in addition to the training data to tune hyperparameters. You can simply create the train/test/validation splits by sampling without replacement from an initial dataset. You don't need anything additional than this initial dataset.

The quote is from the introductory chapter, the word "additional" is used once in the OP's linked chapter, as is the word "dataset". Validation or dev  are not mentioned at all. I think it is less emphasis, than a simple introduction without using all the jargon.

Surely you understand the concept of overfitting: if you are optimizing over hyperparameters under the same underlying data, then technically you are optimizing a new dataset $(\theta, D)$, where $\theta$ are hyperparameters, and $D$ is the original dataset.
This is not good, because in theory, $\theta$ should be independent of the underlying specific $D$ in question, so anything learned is likely to be overfitted to the specific $D$. We mitigate this by requiring additional data for training hyperparameters, such that anything learned about $\theta$  would be about the underlying data-generating statistic and not just any specfic $D$.


text:
What are the Calculus books recommended for beginner to advanced researchers in artificial intelligence?
Calculus is a branch of mathematics that primarily deals with the rate of change of outputs of a function w.r.t the inputs.
It contains several concepts including limits, first-order derivatives, higher-order derivatives, chain rule, derivatives of special and standard functions, definite integrals, indefinite integrals, derivative tests, gradients, higher-order gradients, and so on...
Calculus has been heavily used in optimization and maybe in several other aspects of artificial intelligence.
What are the Calculus textbook(s) recommended that cover all the concepts required for a researcher in artificial intelligence?

I don't have any books in mind (hence the comment rather than the answer), but from my experience most of ML has not been too bad with the calculus, it's mostly just multi-variable calculus (which is really just summing the gradients). If you cover that and basic calculus you'll be good for most mainstream algorithms

Answer: Calculus James Stewart is the best for a beginner.

I started to learn Calculus studying engineering with James Stewart Calculus ( maybe the best for beginners and is really didactic ), Problems in Mathematical Analysis Demidovich ( best for me because simplicity, fast, but few multivariable focus and difficult for learn ), Nikolai Piskunov - Differential and Integral Calculus (again difficult to learn but teachers used for prepare his test), Calculus with Analytic Geometry Swokosky, Louis Leithold Calculus and Purcell Calculus. this books are the popular base books for an engineering degree in mostly all universities.
However the best way for approach Calculus to Artificial Intelligence is focus in the chapter that are directly related to IA and we have:

Multivariable Calculus (also could help for understand faster linear algebra, eigen values&vectors, $ R ^n $ spaces,etc )
Directional Derivatives ( For Gradient Descent )
Infinite Sequences and Series
Partial Derivates ( you need know one variable derivatives for go to Partial Derivatives )
Vector Calculus
Jacobian
Of course all that need a deep understanding of integrals and derivatives no forget.
Calculus.
none of this can be learned without knowing algebra, matrices, geometry, trigonometry and logic math ( Elementary Subjects )

I can tell you my experience learning Calculus James Stewart 7th and build a summary about the topics.

Multivariable Calculus 6 Chapters about that.
Directional Derivatives Section 14.6
Infinite Sequences and Series Chapter 11
Vector Calculus Chapter 16
Jacobian Transformation ( 15.10 Change of Variables in Multiple Integrals)
Of course all that need a deep understanding of integrals and derivatives no forget. 6 Chapters about that.

As a reference this is the index:

I am looking forward for more books about advanced calculus especially with focus in multidimensional calculus or applied math for artificial intelligence, I have found more books about statistics approach than calculus/math approach.

http://index-of.co.uk/Mathematics/Calculus%20-%20J.%20Stewart.pdf


text:
How to understand slope of a (non-convex) function at a point in domain?
Consider the following paragraph from Numerical Computation of deep learning book that says derivative as a slope of the function curve at a point

Suppose we have a function $y= f(x)$, where both $x$ and $y$ are real
numbers. The derivative of this function is denoted as $f'(x)$ or as
$\dfrac{dy}{dx}$ . The derivative $f'(x)$ gives the slope of $f(x)$
at the point $x$. In other words, it speciﬁes how to scale a small
change in the input to obtain the corresponding change in the output:
$f(x+ \epsilon) \approx f(x)+\epsilon f'(x)$.

Slope of a function $f(x)$ at a point $a$ is generally defined as the $\tan$ of the angle made by the tangent line at the point $a$ on the curve of the function $f(x)$ to the positive x-axis in anti-clock wise direction. That is, if $\theta$ is the angle made by the tangent of the curve $f(x)$ at a point $(a, f(a))$ to the positive x-axis in anti-clock wise direction. Then the slope of $f(x)$ at point $a$ is $\tan \theta$.
In theory, tangent line should touch the curve of $f(x)$ at a single point only. Most of the textbooks draws nice convex curves and then show slope as $\tan \theta$. But, i think it is not possible for many functions to draw a tangent line at a point that touches the curve at that single point only. Else it may be a tangent line or some other traversal.
How to understand slope as $\tan \theta$ in such cases? Where am I going wrong?

Slope is not defined like this. You are confusing slope with angle. The slope definition will be more natural, as seen below.
Intuitively, the slope of a curve at a point is directly how steeply it changes as a function of change of horizontal distance: a flat road doesn't change in height as you drive on it, but driving on a ramp constantly elevates you as you drive up it. Typically this is understood as a ratio, so a 45 degree angle has a slope of 1, and a slope of 30 degrees has a slope of $\sqrt3$. This will be formalized in trigonometry, but if I recall correctly, the definition can go the other direction as well (that the ratio approach defines the angle).
We use ratios measurements of slopes instead of degree measurements of angles because calculating the derivative using the limit definition gives the slope as a ratio directly. This graphic shows the idea rather nicely, but this article provides a clearer exposition.
There's also the case that the slope as a ratio makes more sense since the backpropagation technique is effectively adding/subtracting $ε$ to some $x$ given known $\Delta f(x)$, the error signal, relying on the equation you posted above: $f(+)≈()+′()$.
We can rewrite the equation above to obtain $() - f(+) ≈ - ′() $. Since $\Delta f(x)$ and ′() are known, we can also compute $$, but note that only the righthand expression of $$ is computed since $() - f(+)$ is obtained as $Δ()$. We can resolve this with the update rule $x \rightarrow x - \epsilon$

Do you mean $\dfrac{dy}{dx} = \tan \theta$ may not be valid in some cases?

i think you are confusing the word "tangent" for the trig expression $\tan$. these are different.

$\theta$ inside $\tan \theta$ is the angle made by which line? It is tangent line right?

relative to what other angle? you'll eventually have to use the limit approach anyways i believe...

sorry i meant to say: the line that defines the tangent is characterized by the limit approach; without it, you will find it hard to "draw a line" that only "touches the curve" at a single point


text:
What all does the gradient tells us other than the direction to move parameters?
Gradients are used in optimization algorithms.
I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the opposite direction of the gradient to get optimal values.
Thus the gradient provides direction to update parameters.
Is it the only information provided by the gradient? Or does it provide any other information that helps in the training process?

gradient magnitudes also tell you how much you need to update the weights, in addition to the direction (i.e., increment or decrement).

@SpiderRico If it is contributing in terms of magnitude, I feel the learning rate is decreasing the magnitude... So, I'm feeling that the magnitude of the gradient is just nominal.. Is it true?

Yeah you can adjust for that with learning rate. However, note that magnitude of a gradient is not the same for every dimension. So, some dimensions would still be changed more than the others.

@SpiderRico Feel free to provide a formal answer if you think you have enough information to at least partially answer the question.

Momentum was big.  It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum.  It operates against sequential measurements of the error.  This means that several estimates of the gradient give better local picture of the loss-surface.
The error has a magnitude, so the gradient has both direction and magnitude.  It tells us the direction to go, but also how far to go.  The space is complex, so many momentum methods smooth the magnitude and direction by combining thousands of gradient estimates.
Dataset distillation is interesting because it can require a 10x larger network to learn a task, then distill that learning into a 1x network to do the task.  This is a universal-to-specific transformation, using a global approximator to find the local landscape that works is very different than building an approximator for use only within that landscape.  The converged space of both networks is the same, but one is contrived in vastly fewer parameters.  The gradient in the around the optimum, the perturbed gradient given the training data, tells how to perform the simplification.
Saliency maps use back-propagation (single-pass gradient on fully trained networks) to infer interior structure and operations of complex neural networks.


text:
How many directions of gradients exist for a function in higher dimensional space?
Gradients are used in optimization algorithms. Based on the values of gradients, we generally update the weights of a neural network.
It is known that gradients have a direction and the direction opposite to the gradient should be considered for weight updation. In any function of two dimensions: one input, and one output, there are only two possible directions for any gradient: left or right.
Is the number of gradient directions infinite in higher dimensions ($\ge 3$)? Or does the number of possible directions are $2n$ where $n$ is the number of input variables?

Let's look at the definition of gradient:

In vector calculus, the gradient of a scalar-valued differentiable function $f$ of several variables is the vector field (or vector-valued function) $\nabla f$ whose value at a point $p$ is the vector $r^{[a]}$ whose components are the partial derivatives of $f$ at $p .^{[1][2][3][4][5][6][7][8][9]}$ That is, for $f: \mathbb{R}^{n} \rightarrow \mathbb{R}$, its gradient $\nabla f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{n}$ is defined at the point $p=\left(x_{1}, \ldots, x_{n}\right)$ in $n$-dimensional space as the vector: $^{[b]}$
$$
\nabla f(p)=\left[\begin{array}{c}
\frac{\partial f}{\partial x_{1}}(p) \\
\vdots \\
\frac{\partial f}{\partial x_{n}}(p)
\end{array}\right]
$$

First of all, the gradient is not a single value or a vector, it's an operator that given a function returns another function (note that in the definition $$\nabla f$$ map from $\mathbb{R}^n$ to $\mathbb{R}^n$ again), which can be used to compute a vector for each point of a field. So, it doesn't really make sense to talk of a gradient direction per se, since the direction actually belongs to the single vectors associated with each point of the field. How many directions do these vectors have? Well, it depends on the field. A plane has 2 dimensions, hence 2 directions in which you can move, in the same way, $\mathbb{R}^n$ will have $n$ dimensions, hence $n$ directions in which you can move.
Note also that in gradient descent the gradient is computed with respect to the cost (loss) function:
$$W_{t+1} = w_t - \alpha(\partial C / \partial  w)$$
Mathematically this means that:

we have a set of weights $w$
we use these weights to produce an output given a specific input
we then compute an error (or cost, or loss) between the input and output, i.e. we compute at which point of the field of the cost function we end up due to our current weights
we then compute the gradient of the cost function, i.e. we look at all points of the field of the cost function to understand which vector has the biggest magnitude (we care most about the magnitude rather than the direction) and finally
we update the weight in such a way that the next value produced by the weights when computing the cost function will be in the same direction as the previously found vector.

When people say "gradient", as in the case of gradient descent, people usually refer to the gradient vector and not the gradient operator. It's important to distinguish the two, as you're doing, but it's also important to say that people usually refer to the gradient vector. You also say "A plane has 2 dimensions, hence 2 directions in which you can move". I would say that it has 2 degrees of freedom for the direction, but the directions of these vectors will not just be 1 of 2 possibilities, in case that's what you mean. In other words, in a 2d space, you have infinitely many directions.


text:
Confusion about conversion of RGB image to grayscale image using a convolutional layer with 2-dimensional filters
Let us imagine $x$ as a tensor containing 1000 RGB images, each of size $64 \times 32$.
>>> x = torch.randn(1000, 3, 64, 32)
>>> print(x.shape)
torch.Size([1000, 3, 64, 32])

I am using a 2d convolutional layer that converts RGB images to single channel (say grayscale) images
>>> in_ch = 3
>>> out_ch = 1
>>> m = nn.Conv2d(in_ch, out_ch, 3, 1, 1)
>>> print(m)
Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

I passed the tensor $x$ in the convolutional layer and obtained another tensor of 1000 grayscale images, each of size $64 \times 32$.
>>> output = m(x)
>>> print(output.shape)
torch.Size([1000, 1, 64, 32])

Now, I can say that my convolutional layer converted an RGB image into a grayscale image using 2d kernel.
How it is doing?
RGB image has 3 planes each of size $64 \times 32$. If a kernel of 2 dimensions is used, then we will get 3 planes in output, corresponding to R, G, and B. How is it possible to convert an image with 3 channels into an image with one channel using 2d kernel?
I can visualize easily if I use a 3d kernel since the kernel considers three channels simultaneously and produces a single feature map for an RGB image.

Useful : https://stackoverflow.com/questions/56675943/meaning-of-parameters-in-torch-nn-conv2d

Yes, the kernel is 3D in this case - or 4D as in 3x3x3x1. In the general case you can have multiple output channels, making it 3x3x3x8 for example. The number of channels isn't a convolution dimension because the filter does not "slide"/"translate"/"move" over this dimension. It's still a 2D convolution, and then the channel part of this operation is thought of separately from the convolution part. The 4D kernel is a bunch of 2D kernels. If you have 1 input channel and 1 output channel then it's just one 2D kernel. Or you can think of it as a bunch of 3D kernels if you like. Or a single 4D kernel. These are just arrays of numbers... you can slice an array up however you like, if you can find a way to think about it.

Note the groups parameter of Conv2d, which affects how the channels are convolved. The default is 1, which means:

At groups=1, all inputs are convolved to all outputs.

If you set it to 3 (and 3 output channels) then the Conv2d layer would maintain the channel separation.

Okay! seems that kernel size is misleading. Actually the kernel size is not $3 \times 3$. It is $3 \times 3 \times 3$.

(+1) for Yes, it is really a 3D convolution (in this case), because I need to understand remaining to appreciate the answer further.

Yes, it is really a 3D kernel (in this case),?

@hanugm The convolution kernel is actually 3x3x3. There could also be multiple output channels so you can say it's 3x3x3x1. It's still a 2D convolution because there are two dimensions where the kernel translates/slides/moves.

Ha. So, if possible try to change 3D to 2D in your answer. It may be misleading.

@hanugm I edited the answer


text:
What does it mean by strong or sufficient gradient for training in this context?
It has been mentioned in the research paper titled Generative Adversarial Nets that generator need to maximize the function $\log D(G(z))$ instead of minimizing  $\log(1 −D(G(z)))$ since the former provides sufficient gradient than latter.

$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(z))]$$
In practice, the above equation may not provide sufficient gradient
for $G$ to learn well. Early in learning, when $G$ is poor, $D$ can
reject samples with high confidence because they are clearly different
from the training data. In this case, $\log(1 −D(G(z)))$ saturates.
Rather than training $G$ to minimize $\log(1 −D(G(z)))$ we can train G
to maximize $\log D(G(z))$. This objective function results in the same
fixed point of the dynamics of $G$ and $D$ but provides much stronger
gradients early in learning.

A gradient is a vector containing the partial derivatives of outputs w.r.t inputs. At a particular point, the gradient is a vector of real numbers. These gradients are useful in the training phase by providing direction-related information and the magnitude of step in the opposite direction. This is my understanding regarding gradients.
What is meant by sufficient or strong gradient? Is it the norm of the gradient or some other measure on the gradient vector?
If possible, please show an example of strong and weak gradients with numbers so that I can quickly understand.

Technically, the term "sufficient graident" isn't quite well-defined. But usually what it means is the magnitude of the gradients are so small that it is unable to drive the optimization process.

@AyanDas Does it mean they update weights very slowly or do not converge?

The terms "insufficient gradient" or "not strong enough gradient" usually means that the magnitude of the gradient vector is too small or nearly zero that they can't drive the optimization properly.
Not having sufficient gradient is similar to having a very low learning rate - they are not only slow (in terms of convergence) but also drifts the optimization in a poor direction and get stuck in local minima.


text:
What is an "additional channel dimension" contain in batch normalization?
Consider the following explanations regarding batch normalization layers in PyTorch
#1: one dimensional batch normalization

class torch.nn.BatchNorm1d(.........)
Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D
inputs with optional additional channel dimension)............

#2: Second dimensional batch normalization

class torch.nn.BatchNorm2d(..........)
Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension)

#3: Third dimensional batch normalization

class torch.nn.BatchNorm3d(..............)
Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension)

All these say that there can be an extra element to each vector undergoing batch normalization and is related to channel.
Is the channel referred here is same as the channels of the image? If yes, then what does it contain? Does it contain the number of channels in that particular layer?
Else, what does this additional channel contain?.


text:
Example for batch normalization in an aritifical neural network
Suppose the following is the neural network I want to train and assume that there is a batch normalization layer for each layer of the neural network
.
My focus is on the activity of batch normalization layer of the hidden layer. Assume that mini-batch size is $3$. The following are the first four outputs of the hidden layer without using batch normalization layer.
$$\left(\begin{array}{c} 4 \\ 5 \\ 2 \\ 1 \\ \end{array}\right), \left(\begin{array}{c} 3 \\ 4 \\ 6 \\ 0 \\ \end{array}\right), \left(\begin{array}{c} 1 \\ 4 \\ 7 \\ 9 \\ \end{array}\right), \left(\begin{array}{c} 3 \\ 5 \\ 7 \\ 9 \\ \end{array}\right)$$
So the first mini-batch is$\left(\begin{array}{c} 4 \\ 5 \\ 2 \\ 1 \\ \end{array}\right), \left(\begin{array}{c} 3 \\ 4 \\ 6 \\ 0 \\ \end{array}\right), \left(\begin{array}{c} 1 \\ 4 \\ 7 \\ 9 \\ \end{array}\right)$ and has the following statistics
mean = $\mathbb{E}[X] = \left(\begin{array}{c} \dfrac{8}{3} \\ \dfrac{13}{3} \\ 5 \\ \dfrac{10}{3} \\ \end{array}\right), Var(X) = \left(\begin{array}{c} \dfrac{14}{9} \\ \dfrac{2}{9} \\ \dfrac{14}{3} \\ \dfrac{146}{9} \\ \end{array}\right)$
So, I am thinking that the batch normalization can be applied during from fourth iteration since mini-batch of outputs of hidden layer are available and so fourth vector  $\left(\begin{array}{c} 3 \\ 5 \\ 7 \\ 9 \\ \end{array}\right)$ will be mapped to $\left(\begin{array}{c} 0.26 \\ 1.41 \\ 0.92 \\ 1.4 \\ \end{array}\right)$
Is it true that if we apply batch normalization the outputs of neural network are
$$\left(\begin{array}{c} 4 \\ 5 \\ 2 \\ 1 \\ \end{array}\right), \left(\begin{array}{c} 3 \\ 4 \\ 6 \\ 0 \\ \end{array}\right), \left(\begin{array}{c} 1 \\ 4 \\ 7 \\ 9 \\ \end{array}\right),\left(\begin{array}{c} 0.26 \\ 1.41 \\ 0.92 \\ 1.4 \\ \end{array}\right)$$
I am thinking that I may be totally wrong because the batch normalization layer is not active till the completion of first mini-batch and also using the stats of first mini batch on the outputs of next mini-batch. If wrong, I want to know what is wrong or a simple numerical example by at-least running the batch normalization layer of neural network for mini-batch + 1 number of iterations.


text:
Is there any difference between an objective function and a value function?
I found the usage of both objective function and value function in the same context.
Context #1: In the paper titled Generative Adversarial Nets by Ian J. Goodfellow et al.

We simultaneously train G to minimize $\log(1 −D(G(z)))$. In other
words, $D$ and $G$ play the following two-player minimax game with
value function $V (G,D)$:
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$$

Context #2: In the paper titled Conditional Generative Adversarial Nets by Mehdi Mirza et al.

The objective function of a two-player minimax game would be as
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x|y)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z|y)))]$$

In fact, the second paper also iterated context #1 i.e., used the term "value function" at another place.
We can observe that objective function is a function which we want to optimize

The objective function is the most general term that can be used to
refer to a cost (or loss) function, to a utility function, or to a
fitness function, so, depending on the problem, you either want to
minimize or maximize the objective function. The term objective is a
synonym for goal.

Since the generator or discriminator has to perform optimization, it is agreeable to use the term objective function in this context.
But what is the definition for the value function and how is it different from the objective function in this context?

Reading closely the "criteria" you quote, I don't think there is any contradiction; in both cases, $V (G,D)$ seems to be the value function, while the objective function is $$\min_G \max_DV(D, G) = \mathbb{E}{x ∼ P{data}}[\log D(x)] +
\mathbb{E}{z ∼ p_z}[log (1 - D(G(z)))]$$ (or $$\min_G \max_DV(D, G) = \mathbb{E}{x ∼ P_{data}}[\log D(x|y)] +
\mathbb{E}_{z ∼ p_z}[log (1 - D(G(z|y)))]$$)

@desertnaut So, there is no difference between the both and they are same!

I didn't say that, neither I implied it; I said that seemingly the objective function is a functional that includes the value function. V(D, G) (value) is not the same with min max V(G, D) (objective).

The value function may be used in the GAN paper because GANs are inspired by game theory, where terms like utility, utility function and value function (just like in reinforcement learning) are used (the first two for sure, but I am not sure about the usage of the term value function in game theory, as I am far from an expert in game theory).  If you want to know more about the usage of the term value function, this Wikipedia article could be useful (or maybe make things more confusing).
Having said that, it seems to me that the usage of the term "objective function" in the conditional GAN paper is a bit sloppy. They probably meant the optimization problem.
However, it's also true that the notation used by the original authors of the GAN can also be confusing. They wrote
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))] \label{1}\tag{1}$$
Here, $V(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$, so they could have written \ref{1} as follows
$$\min_G \max_D V(D, G) = \min_G \max_D\mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))] \label{2}\tag{2}$$
or just
$$\min_G \max_D V(D, G) \label{3}\tag{3}$$
Then clarify that $V(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x)] +
 \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]$.
This is clarified in this paper (equations 2.1 and 2.2., page 5).
So, in the GAN, we're optimizing $V$, so $V$ is the objective function; thus, in this case, the term "value function" is a synonym for "objective function". In this case, the optimization problem is a $\color{blue}{\textrm{min}}$$\color{red}{\textrm{max}}$ game, i.e. we $\color{red}{\textrm{maximize}}$ and $\color{blue}{\textrm{minimize}}$ at the same time two terms of the objective function, i.e. $\color{red}{\mathbb{E}_{x ∼ P_{data}}[\log D(x)]}$ and $\color{blue}{\mathbb{E}_{z ∼ p_z}[log (1 - D(G(z)))]}$ (this is explained in the GAN paper!). In practice, they optimize two slightly different objectives, but they are equivalent. See algorithm 1 in the GAN paper.
So, as I said in my other answer, the objective function is the function that you want to optimize (i.e. minimize or maximize), so it's usually a synonym for loss/cost/error function (in case you want to minimize it) and can be a synonym for value function (in case you want to maximize it, for example, in reinforcement learning), as it seems to be the case in the GAN (although, in the GAN, you maximize and minimize the value function).

The Wikipedia article you link to states that: "The value function of an optimization problem gives the value attained by the objective function at a solution, while only depending on the parameters of the problem"; so, it would seem that the value function and the objective function are indeed different concepts.

@desertnaut I would agree with that, for instance the value function would not usually include regularisation terms, whilst the objective function can do


text:
Do researchers generally treat tensors just as mathematical objects with certain shape?
Most of the practical research in AI that includes neural networks deals with higher dimensional tensors. It is easy to imagine tensors up to three dimensions.
When I ask the question How do researchers imagine vector space? on Mathematics Stack exchange, you can read the responses

Response #1:
I personally view vector spaces as just another kind of algebraic
object that we sometimes do analysis with, along the lines of
groups, rings, and fields.
Response #2
In research mathematics, linear algebra is used mostly as a
fundamental tool, often in settings where there is no geometric
visualization available. In those settings, it is used in the same
way that basic algebra is, to do straightforward calculations.
Response #3:
Thinking of vectors as tuples or arrows or points and arrows... is
rather limiting. I generally do not bother imagining anything visual
or specific about them beyond what is required by the definition...
they are objects that I can add to one another and that I can
"stretch" and "reverse" by multiplying by a scalar from the scalar
field.

In concise, mathematicians generally treat vectors as objects in vector space rather than popular academic/beginner imaginations such as points or arrows in space.
A similar question on our site also recommends not to imagine higher dimensions and to treat dimensions as degrees of freedom.
I know only two kinds of treatments regarding tensors:

Imagining at most up to three-dimensional tensors spatially.

Treating tensors as objects having shape attribute which looks like $n_1 \times n_2 \times n_3 \times \cdots n_d$

Most of the time I prefer the first approach. But I am feeling difficulty with the first approach when I try to understand codes (programs) that use higher dimensional tensors. I am not habituated with the second approach although I think it is capable enough to understand all the required tasks on tensors.
I want to know:

How do researchers generally treat tensors?
If it is the second approach I mentioned: Is it possible to understand all the high dimensional tensor-related tasks?

I am curious why, while you posted How do researchers imagine vector space? to Math SE (where it arguably belongs indeed), you decided to post a similar question about a different mathematical object (tensor) here instead of there as well.

@desertnaut It is because I found the usage of tensors in AI at some places are difficult for me to understand. Suppose Unsqueeze operation in PyTorch. These type of operations on tensors may be new for mathematicians. So, I choose to ask in our site.

I agree. Understanding and manipulating tensors is a crucial skill, specially for working with Neural Networks.

I would say they are treated as multidimensional arrays of numbers. They are not visualized in their actual dimension. Sometimes small ones will be visualized when someone is trying to explain a concept that requires it.
You may have, for example, a variable uint8 training_batch[100][200][400][3];. This is a batch of 100 RGB images with 200x400 pixels in each image. A pixel is an array of [3] numbers; an image is an array of [200][400] pixels; a batch is an array of [100] images. There's no more structure than that. You don't have to try to imagine a 4D array of numbers. (In this particular case you could easily imagine an array of images though)
What is useful to imagine is what each dimension means. The first dimension is the image within the batch. The 2nd and 3rd dimensions are the pixel position in the image. The 4th dimension is the R/G/B channel.
If I reduce a tensor along a dimension, I wouldn't think of it as flattening, but rather as using up a dimension. If I want to compute the average colour of each image, I reduce the 2nd and 3rd dimensions and get another tensor of shape [100][3]. Now there's no width or height dimension anymore, just image and channel.
If you reshape the vector to [100][240000] so you can compute a matrix multiplication for a dense layer, now the 1st dimension is still the batch number, and the 2nd dimension is essentially meaningless but you have 240000 arbitrarily-indexed numbers per image. You could also reshape it to [100][80000][3] and have 80000 arbitrarily-indexed pixels, but still, be able to use the channel number.
Disclaimer: I'm not actually a researcher.


text:
Does generator in conditonal GAN obey probability laws?
In probability, we have two types of probability functions: unconditional probability $p(x)$ and conditional probability $p(x | y)$. Both are fundamentally different and the latter can be obtained by the following equation
$$p(x|y) = \dfrac{p(x, y)}{p(y)} \text{          provided   } p(y) \ne 0$$
I never heard of formal definition for conditioning except for conditional probability function.
But in case of neural networks, I came across the notion of conditioning.
$$\min_G \max_DV(D, G) = \mathbb{E}_{x ∼ P_{data}}[\log D(x|y)] +  \mathbb{E}_{z ∼ p_z}[log (1 - D(G(z|y)))]$$
Since neural network $D$ is intended to implement a probability function, we can at-least think about conditioning on an input. But the neural network $G$ is not intended to implement probability function. $G$ is intended to provide datasamples by learning an underlying probability distribution whose output is not in the range $[0, 1]$.
Does $G$ obey the laws of probability? If yes, how, since its output is not restricted to $[0, 1]$?  If no, then why the authors use the notation of conditional probability for $G$ also?

Why would the output have to be in the range [0,1]? Who said G's output is a probability?

G is more like a function for sampling from a probability distribution

@user253751 Confusion is due to the notation of conditional probability $G(x|y)$.

@user253751 I didn't see any function till now of the form f(x|y) except conditional probability distribution function p(x|y).

I don't see where it's implied that G is a probability distribution. G is a function, whose output conditioned on one variable has a probability distribution, but it isn't one.
z is random noise which is G's source of randomness. y is something that isn't random. We call G over and over with the same y and different random z's and look at the distribution of the output.
For example, here are some outputs of one possible G(z,y), for different z, when y=3. z is not displayed, only G(z,y). This G outputs one digit.
2262662626626262626262626626262262622226626622626662626666262626262626262
These digits have a probability distribution. About half of them are 2 and about half of them are 6. P(G(z)=2|y=3) = 0.479 and P(G(z)=6|y=3) = 0.521 (approximately). Even though neither 2 nor 6 is a valid probability.

So, the notation has to be $G(z,y)$, Then what might be the reason for using $G(z|y)$?

@hanugm Notation does not matter. Notation does not have to be a certain way. It seems that both z and y are inputs to G. They probably write G(z|y) because it reminds you of conditional probability.


text:
Are there any other metrics available for calculating the distance between two probability distributions other than those mentioned?
The divergence between two probability distributions is used in calculating the difference between the true distribution and generated distribution. These divergence metrics are used in loss functions.
Some divergence metrics that are generally used in literature are:

Kullback-Leibler Divergence
Jensen–Shannon divergence
f-divergence
Wasserstein distance

Some other divergence measures include:

Squared Hellinger distance
Jeffreys divergence
Chernoff's $\alpha-$divergence
Exponential divergence
Kagan's divergence
$(\alpha, \beta)-$product divergence
Bregman divergence

I think some naive divergence measures include

Least-squares divergence
Absolute deviation

Along with these, are there any other divergence measures available to compute the distance between the true probability distribution and estimated probability distribution in artificial intelligence?

You're already listing many measures. Probably all the major ones used in the literature that I know too. Anyway, keep in mind that our site is not appropriate for asking questions that require very long answers that require a lot of effort and time to write. So, it may be a good idea to reformulate your question that you provide more details about why you're interested in more measures and maybe ask for a reference that covers them (rather than asking for us to enumerate them), if you're interested in an extensive list. In that case, you may want to use the tag [tag:reference-request].

Having said that, in this answer, I provide a link to a paper that talks about the topic, but I don't remember which measures are listed there (maybe you already list all the ones mentioned there too).

@nbro Thanks... I rephrased, I think now even any single metric is eligible as an answer.


text:
What is the significance behind having small kernel sizes over having one large kernel size that covers the entire input in a CNN?
I have hardly ever seen anyone cover the entire input image with a filter of the same dimensions. I was wondering why that is the case, and if the performance in say, an image detection application would decrease if someone used kernel size = the size of the input image itself?

What is the output in this case? It is a single real number?

@hanugm Sure, let's assume that it has a two cases - one with a single real-valued output and the other a probability output vector of size 'n'


text:
Which product operation should be used in affine transformation?
Affine transformation as I am aware can be expressed as either dot product followed by addition or a matrix multiplication followed by addition
$$a.x+b$$
$$a^{T}x + b$$
where the first one is based on dot product and the second one is based on matrix multiplication. it should be noticed that $a, x$ are column vectors here and $b$ is a real number.
In case if a matrix $a$ is compatible (say order $m \times n$) and $x, b$ are column vectors of order $m \times 1$. Then affine transformation is (generally) a matrix multiplication followed by a vector addition.
$$a^{T}x + b$$
I think it is correct.
I have doubt about the product operation between $a$ and $b$ if $a, b$ and $x$ are tensors of higher dimensions. In the case of tensors, should we need to perform Hadamard product or normal matrix multiplication? Or are they both equivalent like in the case of affine transformation on column vectors?
I got this doubt because I encountered an affine transformation that neither uses dot product nor matrix multiplication but uses Hadamard product.

Background: Recently I came across an affine transformation that applies Hadamard product on reshaped weight $a$ and input $x$ and then adds reshaped bias $b$ to it
Initially, the dimensions are as follows
a -> [r, c]
x -> [r, c, d1, d2]
b -> [r, c]

Later they reshaped weight and bias to the shape of the input
a -> [r, c, d1, d2]
x -> [r, c, d1, d2]
b -> [r, c, d1, d2]

and finally, they are performing a * x + b
here $*$ is Hadamard product, an element-wise multiplication operation and (i think) is entirely different from normal matrix multiplication.
Is there any clue on what they did? Is it possible to view Hadamard's product as normal matrix multiplication?

please do comment in-case of any clarifications needed.


text:
Is stability an attribute of model or training algorithm used or combination of both?
From this answer, stability is attributed to a learning algorithm

A stable learning algorithm is one for which the prediction does not
change much when the training data is modified slightly.

At some other places, I read the phrase "stability of neural network model". I am not sure whether the stability of a learning algorithm and the stability of a model are the same or not. If same then

A stable model is one for which the prediction does not
change much when the training data is modified slightly.

Is it true? If not, is there anything called stability of a model and is different from the stability of a learning algorithm?
Suppose I am training a neural network model with a gradient descent algorithm. For which one, do I need to attribute stability or instability? Is it to the neural network model or to the gradient descent training algorithm? Or it should be attributed to the combination of both?

is one for which the prediction does not change much when the training data is modified slightly. Is example back to do it again

@Rachrachben Do you mean it applies to both model and training algorithm?

Can you provide a source for the second part(stable model) as well?  The answer you refer to cites a Wikipedia article which gives a list of stable learning algorithms. Neither gradient descent nor neural networks are on the list.


text:
Can we change bias and control the output of neural network?
I have read the use of Targeted Adversarial Attacks for making the model perform better. But can we change the bias of the neural networks and control the outcome of the network rather than changing the input. if yes, can you share some resources or research papers on targeted bias in neural networks?

No, the bias is part of the network, not one of the inputs, so an adversary has no ability to manipulate it.
Unless the adversary can change the network, in which case they don't need to use any tricks, because they can just change it to one that outputs what they want!