text:
Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?
Assuming we use an MSE cost function of the form
$$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 =  E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
The Stochastic Gradient Descent is used to approximate the true update algorithm, which looks like this
$$\theta_{t+1} = \theta_{t} - \frac{\eta_t}{2}\nabla_{\theta}(E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
to this
$$\theta_{t+1} = \theta_{t} - \frac{\eta_t}{2}\nabla_{\theta}(U_t-\hat{V}(S_t,\theta_t))^2$$
where, for simplicity, $U_t$ represents an unbiased estimate of the true value function $V_{\pi}(s_t)$. This expression is the source of many learning algorithms used in reinforcement learning.
One of the conditions for SGD requires that samples used for updating the parameters must be I.I.D according to the distribution $\mu(s)$. However, in both on-policy and off-policy learning methods, updates at each time-step are based on trajectories generated. Since, along a trajectory, the state $s_{t+1}$ depends on the state at ${s_t}$, this means that the sample used to update $\theta_t$ and $\theta_{t+1}$ are not independent. Many, if not all, sample-based learning algorithms used in RL rely on using SGD, such as the Gradient Monte Carlo Algorithm but I've not really seen anywhere that mentions these algorithms have the "issue" that I mention so I feel like I'm missing something. More formally,

My Question: Does the fact that parameter updates are not I.I.D mean we can't really use stochastic gradient descent AT ALL in learning algorithms, and, if so, why then do these algorithms "work"?

As far as I know, this question applies equally to all forms of parameterised function approximation that are used with learning algorithms (tabular functions*, linear functions and non-linear functions). But, if anyone knows a special reason as to whether these cases should be treated separately could they make clear why
*I understand that when learning algorithms with tabular functions, there exists theory beyond SGD that ensures convergence, however, I'm not entirely sure what this theory is and whether if this makes them exempt, so if anyone knows whether or not it does make them exempt could they also make this clear!

Edit:
It has been highlighted in the comments that replay buffers have been used to resolve the issue of correlated sampling in cases such as DQN and variants of it. This implies correlated sampling is an issue in these cases. Aside from this, I've not heard of replay buffers being used elsewhere (correct me if I'm wrong), so why are replay buffers needed with this off-policy NN approach but not in other learning algorithms given that they all suffer from the issue of correlated sampling.

off-policy methods uses samples from trajectories but they are typically sampled uniformly at random from a replay buffer, so at any given time the samples you're using to perform the update are i.i.d. (there would be a non-zero probability that you could draw two samples that 'consecutively' followed each other in a trajectory but this probability will be extremely small for a large replay buffer). Also, tabular methods are not parameterised function approximations. There are no parameters involved and we are not approximating the value functions.

Addressing your second point, can't tabular methods be considered as a parameterised function approximation where we have a parameter corresponding to each state/state-action pair? furthermore, even when using tabular methods, we start of with an initial estimate of value functions and update them using value-iteration or a similar algorithm, this estimate/approximation only converges to the true value function as the number of updates tend to infinity.

hmm, yes, I suppose they can, but usually in RL if you refer to function approximation you are talking about using a parameterised function to represent the look-up table. Yes, okay, you are technically always 'approximating' the value functions, unless as you say you can allow the algorithms to run for an infinite amount of time. The latter part of your question asks whether tabular methods are 'exempt' from your main question about SGD which is yes they absolutely are because tabular methods don't use SGD.

As for your first point, replay buffers do help resolve the issue of iid sampling but i've only ever seen it in discussed in special cases such DQNs (off-policy non-linear function approximators) whereas the issue I raise is, i believe, more general than this.
Replay buffers could probably be used to resolve this issue in general but I guess I am wondering if it is even an issue because it doesn't really seem to be mentioned anywhere

replay buffers solve the problem in general, as long as they can be used, and they can be used in theory for any off-policy learning algorithm. They are used in many algorithms such as DQN as its variants, DDPG, Soft Actor Critic. It is hard to say whether it is a problem in general as there is much more research involving off-policy algorithms due to their superior sample efficiency. I have certainly trained an algorithm using on-policy algorithms (e.g. actor-critic) and not encountered any issues but this was on very simple environments.

Also the general form of the TD/MC update algorithms stem from using SGD. The general form can then be reduced to special cases depending on the type of function approximator used (such as the tabular approximator)

they don't use SGD explicitly, there is no gradient being taken anywhere in the updates. Either way, the answer to your question regarding tabular methods is that there is no assumption that the updates must require i.i.d samples. All of the 'main' methods that you will have seen for the tabular case (MC/TD learning/Q-learning) all are proven to converge.

SGD theory is used, the "stepsize" is the learning rate and the different between the target and our estimate is the gradient. Here's a link to another SE question where this is briefly discussed
https://ai.stackexchange.com/questions/18187/is-there-a-simple-proof-of-the-convergence-of-td0

that link does not mention SGD, they talk about stochastic approximation theory which is not the same thing (thought they may be related). You are certainly drawing analogs to SGD, which you are correct because they have similar update rules. But our estimate is not a gradient of any function.

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many Tabular methods are proven to converge, for instance the paper by Chris Watkins titled "Q-Learning" introduces and proves that Q-learning converges. Also you include tabular methods as being parameterised function approximators. This is not true. Tabular methods maintain an estimate of the value function in a look-up table for each state-action pair and there is no function approximator being used.
Now for non-tabular methods, i.e. Deep Reinforcement Learning. Here we are using SGD only to optimise the parameters of the networks (assuming of course that we are using NN's as our function approximators, but this is a fair assumption if you read the literature). This is why off-policy methods are typically preferred because they can use a replay buffer which allows the use of data from any past trajectory. When using a replay buffer we sample random past experiences which de-correlates the data and allows the i.i.d. assumption to hold when using SGD.
If we were to use an on-policy algorithm, in theory SGD may not converge to any local optima because we are violating the i.i.d. assumption. However, all this means is that we are not guaranteed to converge. For instance, I have run an experiment using REINFORCE (an on-policy learning algorithm) using NN's as function approximators and they were able to obtain an optimal policy. However, this was for a very simple environment using modestly sized networks, so this is likely why they were able to be trained using non-i.i.d. data. A more in depth question/answer as to why NN's require i.i.d. data can be found here.
To address your edit, Replay Buffers are not required but if we can use them, then why would we not? It helps to maintain the i.i.d. assumption and so it helps us obtain a local optima. If we did not use them then there would not be the guarantee that it would converge. As an aside, Replay Buffers are typically used because they make the algorithm have a greater sample efficiency - this means we can obtain an optimal policy using much less data.
If you are wondering why don't on-policy methods use a replay buffer, the answer is because they are on-policy. The actions used in updates must be taken according to our current policy that we are learning the value functions for (or are optimising the policy of in Policy Gradient Methods). This is not the case in off-policy algorithms - e.g. in Q-learning we are learning the value functions of the greedy policy but we follow a different policy that allows for exploration.

Thanks for the thoughtful answer, I didn't know replay buffers couldn't be used on-policy that was interesting to find out. I had a look at the proof by Watkins, even there the robinson monroe conditions need to be satisfied for the proof, bounded learning rates, which is a requirement for stochastic approximation https://en.wikipedia.org/wiki/Stochastic_approximation, so i'm not convinced on this point still i'm afraid. Although I'm happy to concede that correlated sampling is not a concern in this case as a result of Chris Watkins proof.

i thought I may have missed something but I guess there are no assurances of convergence in non-tabular on-policy methods which is surprising and even in off-policy methods replay buffers only mitigate the issue.
I guess in practice convergence is never really achieved anyway, and sometimes not even preferred, as long as these methods can reduce the error to some degree they are appropriate

@questions You may be interested in this question.

what exactly are you not convinced by? if you could be clear with your question maybe I can shed some more light on it, but as for the actual question you asked in your main question, I have answered this so there’s nothing left to be ‘convinced’ about.

Your provide an anecdotal example of an on-policy algorithm that worked for you in one case, your reason speculates why it may or may not have worked for you in this context. I'm looking for a rigorous answer that can address the following points 1) If SGD conditions are violated should convergence ever occur? 2) if convergence can occur why? does it converge to the right place? under what cases? Although its useful to know about practical implementations and features such as replay buffers that can help, I'm just focused purely on these algos from a theoretical standpoint

Also, non-tabular methods doesn't mean Deep RL as i mention in the original question there are other function approximators, not just DNNs. In fact there is a lot more theory that exists for linear function approximators since they have convex cost functions and the fact they can be easily expressed, but again correlated sampling should mean we violate the conditions of the very algorithms we use to optimise them

I explained this in the answer. IID data is required to guarantee convergence to a local optima. Just because it isn’t satisfied it doesn’t mean it wont happen.
I know DRL are not the only non-function approximators, but they will never (except in extremely rare cases) be used in practice whilst DRL offers such great performance. There a proofs out there (if I remember correctly) that show that using linear function approximators to approximate a value function will still converge for certain algorithms.

I don’t think you will find an answer as to why these methods using eg NNs as function approximators still converge in practice (most of the time). It is a huge gap in the theory of RL and an open research question.

"IID data is required to guarantee convergence to a local optima" That is just paraphrasing what I said in the question. "Just because it isn’t satisfied it doesn’t mean it wont happen" - When you use any tool in any context, if the use case conditions of the tool are not  satisfied why should you expect the tool to work at all let alone "most of the time".

Let us continue this discussion in chat.

As David Ireland has mentioned in his answer, despite correlated sampling, learning algorithms can still converge to the optimal policy/value function.
The reason learning algorithms still may produce the correct results despite the use of correlated sampling is to do with another property, that does not rely on i.i.d sampling, that is, the average regret of SGD tends to zero at the limit. A couple of sources I found that highlight this are

Page 26 of the slides available from the computer science department at the university of British Columbia Machine Learning and Data mining

The introduction of a paper discussing non-iid stochastic optimisation - A. Agarwal and John C. Duchi

Minimising average regret implies that the SGD will, at the limit, produce the best set of parameters that perform well on the encountered samples however this is does not indicate whether the best set of parameters actually performs well.
Extra
This is in fact a general issue that extends beyond the reinforcement learning algorithms. Programs that implement SGD don't typically sample IID! as mentioned in an MIT lecture on stochastic gradient descent at around 40 minutes the lecturer discusses it's more efficient to sample without replacement from the training set making it non IID for which there is not much theory about.

caveat
I'm not 100% sure what the implication of minimising regret means but given regret is of the form
$$regret(\theta_1,\ldots,\theta_t) = \sum_{t=0}^T\left(L(v_\pi(s_t),\hat{v}(s_t,\theta_i) -L(v_\pi(s_t),\hat{v}(s_t,\theta_*)\right)$$
Then I understand the model that minimises regret at time T as the one that has the minimum loss for all training samples encountered up to time T . But what I'm not sure about is how this relates to minimising the cost function, it probably requires another question!

The fact that your answer seems quite different from the other answer shows that your question was originally unclear. Please, next time, before asking a question make sure to clarify what you're really looking for. I would also suggest that you make the title of your question more consistent with your actual question and the question in the body, so that people do not intepret your question in multiple ways, depending on what they think your doubt is.

I think this is more an issue with the clarity of my answer, what I'm trying to explain here is why reinforcement learning algorithms may work despite samples not being I.I.D although I haven't explicitly said this which I should probably do.
My answer is more of a supplement to the other answer that  indicates reinforcement learning algorithms can work despite non-iid sampling and suggests ways to circumvent the issue of non-iid sampling. I'll edit my answer to highlight this


text:
Is the GAN architecture better suited for medical image denoising than the CNN?
I'm considering using GANs for medical image denoising, based on previous literature, like this and this. My input to the GAN would be a high-noise image and my ideal output would be a low-noise, high-quality image.
Is the GAN architecture better suited for applications where the inputs are just random noise? Is the discriminator necessary in this case or is it better to just use a Deep CNN/Autoencoder? How do I justify using a GAN for my application?

Could you please share your "previous literature", so that we can get an idea of what you are trying to do? I guess something like a CycleGAN might be useful for you, where you can translate one "style" of image into another. (see here for example) Maybe in your case the styles could be "low noise images" and "high-noise images". How big is your dataset? You would need a large amount of images with and without noise, but they don't necessarily need to be paired.

@Mafu I was looking at this and this as previous literature. I have a dataset with around 10000 paired patches (128x128) and I could acquire more if required. Thank you for your comment and please do let me know if I can provide any other information, I am currently reading more about CycleGANs.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
If one of the inputs to a neural network (that represents a policy) is noisy and degrades the performance, would this architecture solve the issue?
I'm using genetic algorithms to train deep reinforcement learning (DRL) agents, similarly to what was done in this paper. DRL policies are therefore represented by deep neural networks, which map states into deterministic actions. My state space consists of three state variables $v_1, v_2$ and $v_3$. Variable $v_1$ is extremely noisy and seems to be degrading the performance (i.e. the return or discounted cumulative reward) of my RL agent but, for certain reasons, I have to include it. The return is precisely my fitness function.
Currently, my DNN looks like this:

There is only 1 output since the action space is 1-dimensional (one degree of freedom, which is used to control a system).
The DNN tends to overfit more quickly when $v_1$ is present. I'm considering creating a custom NN that looks like this:

By doing this I would reduce the complexity of the influence of the variable $v_1$ on the output, since the number of layers between $v_1$ and the output node would be reduced.
I have reasons to believe that the optimal control depends linearly or (something close to linearity) on $v_1$.
Does this make any practical sense and are there are reasons why one should avoid doing this?

Yes, you could have a subset of the neural network that uses a linear activation function that gets as input $v_1$, but I don't know if this will really be beneficial. Let us know when you try that.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why does the relativistic discriminator increase the probability that generated data are real and decrease the probability that real data are real?
I was reading the ESRGAN whitepaper, where I came across this line:

Relativistic discriminator [2] is developed not only to increase the probability that generated data are real but also to simultaneously decrease the probability that real data are real.

Can somebody explain this?


text:
What is the purpose of "alignment" in the self-attention mechanism of transformers?
I've been reading about transformers & have been having some difficulty understanding the concept of alignment.
Based on this article

Alignment means matching segments of original text with their corresponding segments of the translation.

Does this mean that, with transformers, we're adding the fully translated sentences as inputs too? What's the purpose of alignment? How exactly do these models figure out how to match the different segments together? I'm pretty sure there's some underlying assumption/knowledge that I'm not fully getting -- but I'm not entirely sure what.

It's basically: how do the output neurons that are supposed to output "économique" know to connect to the input neurons that see the word "economic"? We could just connect every output position to every input position but then it would have a small maximum size limit and wouldn't be translationally invariant because the weights would record the exact positions (output word 5 -> input word 2).

Alignment:
We all know that a good translation cannot be done just by splitting words, converting them, and concatenating them back. Otherwise, a dictionary would be just enough. One translation problem is on the alignment of the words.
For example:
Uma maçã grande e vermelha
(1)  (2)  (3)  (4)  (5)
 |      \ /   ______/
 |       X   /
 |      / \ /
 |     /   X
 |    /   / \
(1) (3) (5)  (2)
 A  big red apple

RNN
This article starts by showing how a RNN translator works and their underlying difficulties. And alignment is a huge pain for RNN because either you'd need another method* to solve it, so RNN could focus on smaller tasks each time.
*And this method usually requires a labeled dataset (like the example above), which is quite tedious to create.
What if, instead of hacking an external element to guess the alignment, we could just send the whole text and train a single neural network to both:

Somehow solve the alignment problem.
Use that to predict the next word.

Wouldn't that be awesome? Introducing:
Transformers
They have a built-in self-attention component that scores all previous words according to their relevance for the next translated word!
TL;DR:
Transformers will automatically solve alignment while translating.


text:
How to treat (label and process) edge case inputs in machine learning?
In every computer vision project, I struggle with labeling guidelines for border cases. Benchmark datasets don't have this problem, because they are 'cleaned', but in real life unsure cases often constitute the majority of data.
Is 15% of a cat's tail a cat? Is a very blurred image of a cat still a cat? Are 4 legs of a horse, but the rest of its body of the frame still a horse?
Would it be easier or harder to learn a regression problem instead of classification? I.e by taking 5 subclasses of class confidence (0.2,0.4,0.6,0.8,1.) and using them as soft targets?
Or is it better to just drop every unsure case from training or/and testing set?
I experimented a lot with different options, but weren't able to get any definitive conclusion. This problem is so common that I wonder if it has already been solved for good by someone?

Unfortunately, the answer here is that "it depends". People have taken different approaches to this problem and I'll describe a few here. None of which however is the "right" answer.
Labeling
When generating benchmark datasets, we actually do have this problem. To be honest, most of the time the labeling is done to the best ability of the human. Sometimes ambiguous or difficult cases are separated out and cleaned, but, usually, labelers are given a set of concrete guidelines to say whether or not something is a cat. In the case where a human is unsure, usually, that data is thrown out or moved over to the "difficult" pile. Unfortunately, this difficult pile isn't publicly available for the most part in many public datasets. However, if you look at most public datasets, even with significant cleaning these cases exist.
Bayesian Deep Learning
One common theme that I've seen is that people add proper probabilistic uncertainties into their models. This is different than the output of a softmax at the end of an object detection network. The output of a softmax is just some number $\in [0, 1]$ that represents the regressed output of a classification within the model. For example, in SSD the softmax is just the classification of a specific anchor box. There is no real "certainty" information associated with it. In most standalone models (and without some pretty hard assumptions) it doesn't have any rigorous probabilistic meaning.
So how would we add a probability? How can we say that "There is a 80% chance that this image is a cat?. The paper, "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? does a pretty good job at looking at this problem.
Basically what they do is build Bayesian models to break have their models explicitly output two types of uncertainties.

Aleatoric uncertainty
captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model – uncertainty which can be explained
away given enough data.

You can go through the paper to get a better understanding of what's going on, but, basically, they fit models to regress both the uncertainty associated with the model, and the uncertainty associated with the data.
Zero-Shot Learning
What about the case where you have a very well defined object, but you've never seen it before? Let's say you've seen a bunch of horses but you've never seen a zebra. As a human, you would look at it for a while and basically just assume that it's a horse with black and white stripes. There is an entire field of machine learning dedicated to this topic. I'm not an expert in it personally, but there are plenty of resources online if you're interested.
A Practical Note
In industry, we usually try to scope the problem as best as we can such that we don't have to deal with this as much. Now, there are times when this is clearly inevitable. What I've seen if an object isn't clearly detectable, then the algorithm might fallback to just saying that there's "something" there. Consider the case for self-driving cars. It's good to detect if there are pedestrians in the road, but if you don't know if something is a pedestrian it's still useful to know that there's something in the road. For this, you can fallback to unsupervised methods to help distinguish objects. From a labeling perspective, you could imagine an ontology of objects for this purpose. At the root node of this ontology would be just, "something in the road" and branches off to "car", "pedestrian" or, "bike" for example. If a labeler is not sure if something is a pedestrian, but it's def something in the road that shouldn't be hit then it would be labeled as "something in the road". Again though, this is highly dependent upon the application.

Just one correction. You say that the softmax probability represents "specifies how certain the model is that the cat is a cat", but that's not really correct, as you even say that these deterministic models do not really model "uncertainty". See this post https://ai.stackexchange.com/q/24872/2444, for example. Regarding your practical note, this seems to be similar to having another class in your training dataset which is "unclear object". See also this: https://ai.stackexchange.com/q/4889/2444

For the softmax maybe I wasn't using the correct terminology. The softmax output depends on whatever you are regressing. In the case of SSD for example, the softmax represents the classification regression target on the class of the anchor box.

I'll update the post to reflect this. Thank you!

Hm, I'm not sure "that represents the output of a regression target within the model." is correct, but maybe I'm misinterpreting this part. The softmax is  used in the context of classification, so why are you saying "regression target"?

Yes, it is classification but you are still regressing some value between zero and one. In industry I've heard the generated labels that are used for SSD and the RCNN families as "regression targets". Which I guess could be confusing.

Ideally, softmax is required to represent certainty of the model but, in reality, this is not the case. Since, the model learns to minimize error during training, it tries to assign maximum probability for its predictions. This situation causes very "confidential" model which assigns over 90% most of the time. https://proceedings.neurips.cc/paper/2020/file/aeb7b30ef1d024a76f21a1d40e30c302-Paper.pdf


text:
Can cryptocurrency charts be estimated using neural networks?
If I were to make a neural network that predicts the value of e.g. Bitcoin tomorrow based on the chart of the last month, would that work? Of course, 100% accuracy cannot be reached, but a success rate over 50% on determining if I should buy or sell Bitcoin could be very profitable. Have there been any attempts to create such neural networks so far?

Autoregression problem assumes that there is some relationship between the present and the past  values. But in fact, there is too much randomness and unpredictability. I cryptocurrency cannot be predicted with a reasonable accuracy like the stocks market in principle

The simple answer is : no. Especially crypto that is a vary manipulated market.

There are some academic works on that, but basically what you're asking for is a time-series estimation.
Here's a paper that tries to estimate Bitcoin price, for example: PAKDD: Forecasting Bitcoin Price with Graph Chainlets.

To make this answer more useful, it may be a good idea to at least briefly describe their approach in the paper.

It's a very complex task. The price of bitcoin could be affected by NEWs(political desicions, network upgrades, other cryptocurrencies), whales buying and selling crypto, network hashrate.
You can try to search for predictions after lets say after 5 minutes, based on the data inside the orderbook on Exchanges. The market is very volatile, you can make profits of shorter period of time.
https://github.com/Sapphirine/BITCOIN-PRICE-PREDICTION-USING-SENTIMENT-ANALYSIS
https://github.com/miroblog/limit_orderbook_prediction


text:
Visualizing the Loss Landscape of Neural Nets: Meaning of the word 'filter'?
I found myself scratching my head when I read the following phrase in the paper Visualizing the Loss Landscape of Neural Nets:

To remove this scaling effect, we plot loss functions using filter-wise normalized directions. To obtain such directions for a network with parameters $\theta$, we begin by producing a random Gaussian direction vector $d$ with dimensions compatible with $\theta$. Then, we normalize each filter in $d$ to have the same norm of the corresponding filter in $\theta$. In other words, we make the replacement $d_{i,j} \leftarrow d_{i,j} \| d_{i,j}\|  \| \theta_{i,j}\| $

I'm completely unclear what the authors are referring to when they refer to the filters of the vector $d$ in weight space. As far as I can tell, the vector $d$ is a standard vector in weight space ($W$) with a number of components equal to the number of changeable weights in the network. In my opinion, it could be said that each layer in the network can be visualized as a vector in weight space ($\theta_{i}$) with:
$$\theta = \sum_{i}\theta_{i}$$
and then maybe these vectors $\theta_{i}$ are called filters? But how this would have anything to do with the random vector $d$, generated in this space, remains a complete mystery to me.


text:
Is there a convention on the order of multiplication of the weights with the inputs in neural nets?
Is there a convention on how the input data and the weights are multiplied? The input data can be anything, including the result from the previous layers.
There are two options:
Option 1:
$$\begin{bmatrix}i_1 & i_2\end{bmatrix} \times \begin{bmatrix} w_1 & w_2 & w_3\\w_4 & w_5 & w_6\end{bmatrix} = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$
Option 2:
$$\begin{bmatrix} w_1 & w_4\\ w_2 & w_5\\ w_3 & w_6\end{bmatrix} \times \begin{bmatrix}i_1 \\ i_2\end{bmatrix}  = \begin{bmatrix}i_1*w_1 + i_2*w_4 & i_1*w_2+i_2*w_5 &i_1*w_3+i_2*w_6\end{bmatrix}$$

The conventions I have seen tend to post-multiply rather than pre-multiply, although there are examples in the literature which adopt the opposite convention.
Some examples include:

In Deep Learning: An Introduction for Applied Mathematicians, a layer with input $x \in \mathbb R^n$ and output $f(x) \in \mathbb R^m$ is computed by
$$ f(x) = \sigma(Wx + b)$$
for a matrix $W \in \mathbb R^{m \times n}$ of weights, a vector $b \in \mathbb R^m$ of biases and an activation function $\sigma \colon \mathbb R^m \to \mathbb R^m$.

In Deep Learning by Goodfellow, Bengio and Courville, they compute a layer in chapter 6 (see p. 171) using

$$ f(x) = \max \{ 0, W^T x + c \},$$
where the $\max$ function is applied componentwise and gives the ReLU activation.

In the paper Federated Learning with Matched Averaging, the authors describe a fully-connected layer using pre-multiplication,

$$f(x) = \sigma(xW),$$
where they omit the bias to simplify notation and as before apply $\sigma$ entrywise. In this case of course we need to view $x$ as a row vector and will obtain a row vector as the result.
Ultimately, as long as you are clear it doesn't matter a great deal: in the end, all that changes is the shape and values of the matrix.


text:
Does Monte Carlo Tree Search not work on games without the same initial state?
I'm curious how you would apply Monte Carlo Tree Search to a game that has a random initial state. You generate a tree where the root node is the initial state, then you expand if the options from that state are not explored yet.
I'm also wondering how this works in 2 player games. After your opponent moves, does each state in the tree have a key to look up in a dictionary? Otherwise, the algorithm won't know what to do when there's a jump in a state between choosing your action on your turn and when your opponent moves, unless you also store your opponent's move in the tree.

If the initial state is not always the same, but if your agent is allowed to observe what the initial state is before it has to start running the search algorithm, there's basically no problem; it has all the information it needs when it starts running the tree search. This is how we typically use MCTS (or any other tree searches): we first observe what the current state looks like, and then start running the tree search for this state.
If for whatever reason you already have to start running your tree search without being allowed to observe which initial state has been randomly drawn, you can easily just pretend that you actually do have a "dummy", deterministic initial state which you can observe right before the real initial state, and pretend that the random sampling of an initial state is actually a random event / random transition resulting from a "dummy action". Then you can handle this scenario in exactly the same way that you would any normal game that has non-deterministic transitions (such as games that involve dice): if you have access to explicit knowledge of which stochastic events are possible, and what their probabilities are, you can encode them as chance nodes. If you do not have such explicit knowledge available, you can use an "open-loop" MCTS. See also: https://ai.stackexchange.com/a/13919/1641

Presence of opponents / other players in the game is a different issue. MCTS and other game tree search algorithms were designed specifically for this, they have no problems handling that. In a tree search, you do not just have nodes for your own agent and the states in which it is allowed to act; you also have nodes for any opponents and any states in which they are allowed to act, and the tree search enables you to reason about what actions your opponents are likely to take.

Does that mean you keep in memory multiple trees with the root being different starting states? Also how does MCTS handle an opponent, because it plays in simulation there's going to be a jump in state not caused by the agent. Does this mean every state is stored in a dictionary so the agent can locate that subtree? What does MCTS do if it reaches a state not in it's tree?

@user8714896 No, a single search tree is sufficient. If you're dealing with the first case from my answer (where you can observe the state you're actually starting in), any part of the search space that starts with a different initial state from the one you're really in is completely irrelevant, you don't even generate any search tree for any of those other cases. If you're dealing with the second case from my answer, you have a single tree, but multiple different cases below the root in the tree (i.e., different subtrees)

States in which the opponent picks the action have nodes in the search tree in exactly the same way that states in which your agent is the mover do. There's nothing special about them. Your agent can reason about what your opponent would like to do in those nodes, and simulate actions accordingly. When you've moved on to your next turn, you can either just start off with a brand new search tree, or reuse relevant parts of the previous search tree (by following the moves that you made as well as those that your opponent made, and discarding the other subtrees below the old root)


text:
Why are the weights of the previous layers updated only considering the old values of the weights of the later layer, not the updated values?
Why are the weights of a neural net updated only considering the old values of the later layer, not the already updated values?
I use this example to explain my problem. When applying the backpropagation chain rule, the weights  of the previous layer ($w_1, w_2, w_3, w_4$) are updated making use of the chain rule:
$$\frac{\partial E_{total}}{\partial w_1} = \frac{\partial E_{total}}{\partial out_{h1}} * \frac{\partial out_{h1}}{\partial net_{h1}}*\frac{\partial net_{h1}}{\partial w_1}$$
He then says:
$$\frac{\partial net_{o1}}{\partial out_{h1}}=w_5$$
Although he has already calculated the updated value for $w_5$, he uses the old value of $w_5$ to update $w_1$? Because the updated value of $w_1$ will have an impact on the outcome together with the updated value of $w_5$?

Your idea is basically to use some intuitive kind of temporal difference, ie, use the difference between latest obtained value (by whatever method) and current value to update the attended independent variable's value, which is not used in backprop but reinforcement learning.

The basic idea of gradient descent is:

Calculate the gradient of some score with respect to parameters that you can control

Take a step in the direction of that gradient that improves the score (subract a multiple of gradient - for gradient descent - if you want to minimise some cost function)

The backpropagation using chain rule in neural network layers is part of the first part, calculating the gradient.
If you interleaved these steps during a single calculation, by taking an update step before propagating the gradient over all layers, you would not propagate the true gradient, but some interim value. There is no guarantee this value would reflect the true gradient of the affected layer when compared to training data and loss functions.
There are some manipulations of the gradient that are acceptable and improve convergence. For instance momentum, and various forms of weighting based on previous gradient calculations. However, I am not aware of any successful attempt to perform updates with partially-calculated gradient then attempt to continue the interrupted gradient calculations over the mixed updated and not-yet-updated parameters.
It is also possible to calculate gradients and then only update some of the possible parameters. In some cases this is useful, for instance it is a good choice when performing transfer learning, by only updating a few layers close to the output. This constrains the network to keep a lot of its trained parameters as-is, which reduces the chances of over-fitting to the smaller dataset that the network is learning from when transferring.

This is an excellent question, and the literature is generally silent on whether it's an improvement. I'm personally rather curious about this result!
There are some practical reasons it's not generally done:
Framework structure
Most ML frameworks separate the calculation of the gradient from the update, because users often want to mess with the gradient before updating the weights (e.g. Adam, momentum, etc). The way that most frameworks are implemented makes it difficult to do the optimizer calculation inline with the back-propagation.
Batch sharding
Most large networks have the batch calculation sharded over multiple chips. This means that when actually doing the gradient update, the partial sums need to be sent off-chip to be summed and added to the weights.
This normally isn't too bad because the transmission of the partials sums for one weight run in the parallel with the calculation of the gradients for the next weight.
But if we update the weights before back-propagating the gradient, then we need to wait for all the partial sums to be received and totaled, then the resulting update being transmitted back to all the chips before the next step in back-propagation can occur. This puts the network latency on the critical path, which makes everything very slow... So it's almost certain that it's faster to do the dumb thing with more steps, than the smarter thing with fewer steps. (The improvement in the gradient would have to save a huge fraction of steps to be worth it).


text:
Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?
If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a probability distribution over the 4 actions, due to the softmax activation function.
Is the policy (based on the neural network) a stochastic policy, even if the action space is discrete?

Yes, for aforementioned case the policy is stochastic. You can select the action for example based on $\epsilon$-greedy policy. which is also a stochastic policy.

You said, "Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete?" Why can't policy become stochastic in discrete action space? I can't understand your perspective.

I just want to give more details about the problem when i mentioned the discrete action space. Sorry, it's not important.

In summary my question is: A policy Is deterministic because it return an action or return a probability"; or because the policy is applied in a stochastic enviroment or deterministic enviroment?

Hi @Xtailker. The policy being deterministic means that the agent would always pick the same action in the same state. The policy being stochastic means that the action that is picked determined by the probabilities. So, if after computing softmax for the state, you use a random number to determine which action to use, then your policy is stochastic. If, for example, you decided to always use Argmax for deciding your action - then your policy would be deterministic. However, if you add probability epsilon to pick a random action instead of argmax, your policy would again become stochastic.

Policy is how the agent decides what to do. Stochastic policy means that there's some randomness involved in the decision making

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete?

Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in the range $[0,1]$ and the sum across all allowed actions is $1$. The two concepts of determinism and discrete actions are entirely separate.
The optimal policy in many situations can be deterministic. If there is only one deterministic optimal policy, your learned policy should be also close to deterministic if the learning process has been successful. That is, the probabilities of optimal actions should all be close to $1$, all the rest close to $0$.
If there is more than one possible optimal policy, your learning agent may have learned a stochastic "mix" of them where in some states it is equally good to take more than one action and the probabilities may be split between those good actions. This will still be optimal and not a problem. If that is the case you should expect to see many action choices close to $0$ and in each state a select few (maybe one) that sum to close to $1$ between them.
In the case of discrete actions, you can derive a deterministic policy from your neural network function by taking the argmax of the action probabilities. This is worth trying. It will round away bad actions that are close to 0 probability due to approximation in the neural network.
In practice sometimes a little randomness in a policy works better for real-world problems with imprecise measurements or other unknowns. It may even be necessary for adversarial environments or where there is key information missing. The only way to find out though is to try both stochastic and deterministic interpretations of the neural network output for your policy.

Thank's for your answer. But, is deterministic or stochastic because "the policy return an action or return a probability" o because the policy is applied in a enviroment stochastic or deterministic?

@Xtalker Sorry I do not understand your comment at all. Which components are you asking whether they are deterministic or stochastic? And in what cases?

The policy is the probability distribution over the actions. If for any action, $a$, $P(a) = 1$, then this is deterministic policy otherwise the probability distribution over action is stochastic policy. Note that, don't confuse stochastic environment with stochastic policy. They are not the same thing. You can have a stochastic policy in a deterministic environment.

Thinking the policy was trained in supervised mode, in a deterministic problem (such as a rubix cube). The action space is discrete and finite, and all actions has deterministics results. The output of the neural network will be a softmax function with a probability distribution about the confidence of the next action. However, the enviroment it is deterministic

@Xtalker In that case the policy produced natually by the method you have used is stochastic (although it could approximate a deterministic one). You may find that changing the softmax to an argmax will give you a deterministic policy suitable for the problem, as described in the answer.


text:
Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm
This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question.
In the A3C algorithm from the original paper:

the gradient with respect to log policy involves the term
$$\log \pi(a_i|s_i;\theta')$$
where $s_i$ is the state of the environment at time step $i$, and $a_i$ is the action produced by the policy. If I understand correctly, the output of the policy is a softmax function, so that if there are $n$ different actions, then we get the $n$-dimensional vector output
$$\pi(s_i;\theta')=\left(\frac{e^{o_1(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},\frac{e^{o_2(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},...,\frac{e^{o_n(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right),$$
where the $o_j(s)$ are softmax layer activations obtained from forward propagation of state $s_i$ through the neural network.
Do I understand correctly that in the A3C algorithm above the term $\log \pi(a_i|s_i;\theta')$ refers to
$$\log \pi(a_i|s_i;\theta') = \log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)$$
with index $j$ referring to the position of the largest element in vector $\pi(s_i;\theta')$ above? Or maybe all action options should be contributing according to their probabilistic weights, like so
$$\log \pi(a_i|s_i;\theta') = \sum_{j=1}^n\log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)~~~?$$
Or perhaps neither of these expressions is correct? In that case, what is the correct explicit formula for the expression $\log \pi(a_i|s_i;\theta')$ in terms of softmax layer activations $o_j$?


text:
Why does sigmoid saturation prevent signal flow through the neuron?
As per these slides on page 35:

Sigmoids saturate and kill gradients.
when the neuron's activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero.
the gradient and almost no signal will flow through the neuron to
its weights and recursively to its data.

So, if the gradient is close to zero, then the error correction would be very minimal. But why would that cause that no signal flow through the neuron?
$$w(n+1) = w(n) - \text{gradient}$$
That would only cause the weights not to change.

I think they mean no signal will flow through the backpropagation since one node close to zero leads to turn off all the gradient of the previous nodes.


text:
Is there any research on the detection of the user's emotion and stress based on the mouse movement and keyboard?
I have to create a model that can detect the user's emotion and stress level based on their mouse movement and keyboard typing activity. I didn't found any research work based on this. Is there any research on this?

A quick Google Scholar search gives me plenty of literature on this topic. Maybe your question can be more specific so we know what you are looking for?

@Mafu Feel free to provide an answer below with links to some reliable and potentially useful research papers (maybe briefly describe their approach, although this was not required in the question). Maybe the user was not aware of Google Scholar. If you think this question is unclear or needs more details, you should flag it as such. See also https://ai.stackexchange.com/help/privileges/vote-down.


text:
Using LSTM model to train spatial inputs
I have an $x$-$y$ plane, inside that plane I have 9 paths $(p_1, p_2, \dots, p_3)$. Each path is classified into one of the three classes $(c_1, c_2, c_3)$. Each path has 100 coordinates points i.e $((x_1, y_1),(x_2, y_2), \dots, (x_n, y_n))$. Totally I have 1800 input coordinate points. Now I am interested in training the LSTM model in such a way that if I feed some test path $p_{10}$, the model should be supposed to predict which class it belongs to. This is my problem definition. Regarding this, I have some questions

First of all, is it necessary to use LSTM models to obtain a solution?
Are there any other simple models to attain a solution to this problem?
I did some literature surveys for this kind of problem using LSTM, they are having time has one of the parameters along with $(x_1, y_1, t_1)$.

The paper I have read is "A Single-Shot Approach Using an LSTM for Moving Object Path Prediction".
I am a beginner to sequence model neural networks. A link or examples to similar works is very much beneficial.

Hello. Welcome to AI SE. Ideally, each post contains 1 main question. What your main question in this case (so that we know what you're looking for)? By the way, the 3rd point is not a question, but a statement.

I am looking for answers/ suggestions for my first two questions.


text:
Why are most commonly used activation functions continuous?
I have come to notice that the most commonly used activation functions are continuous. Is there any specific reason behind this? Results such as this paper have worked on training networks with discontinuous activations yet this does not seem to have taken off. Does anybody have insight into why this happens, or better yet an article talking about this?

Here and here are two related questions.

I had seen both before posting but none of them discusses why they are not wildly used or why most activations are continuous.

Hi @BIM, check this out, has some interesting thoughts. It's about step-function though.


text:
Gradual decrease in performance of a DDPG agent
I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some hyperparameter configurations this is followed by a rebound and again a slump. Here's what a typical reward plot looks like:
Since I'm new to reinforcement learning (this is my first shot at it), I don't know if this a common phenomenon. I know of catastrophic forgetting, but I believe that's not the case here, since this is more akin to a "languishing dementia". As far as I understand, "catastrophic forgetting" is an abrupt event, which contrasts with a gradual change I've been seeing in my attempts.
Is this some kind of general phenomenon with coverage in the existing literature or is this rather a quirk of my specific setup (algorithm + hyperparameters) for which the solution would be "change the setup"?
For reference, the implementation I'm using: https://github.com/hirekk/pytorch-rl

This could well be catastrophic forgetting. A few weeks ago, I came across a paper (I don't remember the name now, but if I find it I will post this comment as a answer below) that actually states that similar patterns of the performance can be due to catastrophic forgetting within the same environment, which is actually a multi-task environment.

@nbro Thank you very much, I wasn't aware that catastrophic forgetting can manifest that way. If this paper you mention managed to find its way back into your hands, I'd appreciate greatly you pointing me to it.


text:
What can be an example for the prior knowledge used in Deep Learning systems?
It is known that machine learning algorithms expect feature engineering as an initial step. Now, consider the following paragraph, taken from 1.1 The deep learning revolution of the textbook named Deep learning with PyTorch by Eli Stevens, Luca Antiga, Thomas Viehmann, regarding the role of feature engineering in deep learning

Deep learning, on the other hand, deals with finding such
representations automatically, from raw data, in order to successfully
perform a task. In the ones versus zeros example, filters would be
refined during training by iteratively looking at pairs of examples
and target labels. This is not to say that feature engineering has
no place with deep learning; we often need to inject some form of
prior knowledge in a learning system. However, the ability of a neural
network to ingest data and extract useful representations on the basis
of examples is what makes deep learning so powerful. The focus of
deep learning practitioners is not so much on handcrafting those
representations, but on operating on a mathematical entity so that
it discovers representations from the training data autonomously.
Often, these automatically created features are better than those that
are handcrafted! As with many disruptive technologies, this fact has
led to a change in perspective.

The paragraph clearly saying that we need to inject some form of prior knowledge into the learning system. What can be a concrete example for such prior knowledge we are used in deep learning systems?

Here is a related (if not duplicate) question.

I would distinguish at least 2 cases when it comes to a generic expression like prior knowledge:

generic extra information provide to a model, really close if not the same as feature engineering.
literal prior probability distributions used to initialize or guide a model during training.

For the first case there's plenty of examples that we can provide. The most intuitive maybe is use of masks in computer vision. Let's say we want to clean an image (i.e. haze removal) as a pre processing step for a self driving car system. Then we could train a model and feed to it not only the image captured by the camera, but also a depth mask estimated using another model. In this case the other model works as a prior distribution, since the model is not learning it, it just leverage that extra information that comes with the image.
For the second case, there are specific class of models that learn and sometimes require prior distributions as an input, the most known to me are Bayesian Neural Networks.

Why bothering providing a prior for these models? Well, there are at least 2 reasons: sometimes we have information about a system we're trying to describe, so we can make the training more efficient, for example we're might trying to fit a coin toss model, but we know the coin is not fair and that the resulting probabilities returned by the model should not be 50/50. The second reason is that sometimes we also want to train models that do not return only raw probabilities, but also estimate the uncertainty level of those probabilities. To do that, the model learns a posterior probability over the data, and it does that by updating an initial prior. Note that the prior for this class of models can also be randomly initialize.

This answer provides good information, but it mixes concepts, so it may be confusing. For example, you say that the second way to interpret "prior knowledge" is "literal prior probability distributions used to initialize or guide a model during training", but then you give the example of "Bayesian neural networks". It's true that BNNs are based on having distributions over the weights, but in your second point you mention probability distributions to initialize or guide a model during training, which seemed to suggest that you were referring to ways to initialise the weights.

Note that I am not saying that BNNs are not a good example where we can introduce prior knowledge. In fact, in BNNs, we can really specify priors for the weights (as priors in the Bayes' rule, which are used to introduce "prior knowledge"), but it just seems that you were referring to the initialization of the weights and other techniques to guide the training of non-Bayesian neural networks. Another thing you may want to point out is the relationship between "prior knowledge" and "inductive bias" (and/or model architecture).


text:
When can we call a feature "hierarchical"?
Features in machine learning are the attributes of the elements of a data set. They are considered as random variables in probability.
Consider the following excerpt from 1.1: The deep learning revolution of the textbook named Deep learning with PyTorch by Eli Stevens, Luca Antiga, Thomas Viehmann,

On the right, with deep learning, the raw data is fed to an algorithm
that extracts hierarchical features automatically, guided by the
optimization of its own performance on the task; the results will be
as good as the ability of the practitioner to drive the algorithm
toward its goal.

When can we call a feature hierarchical? Does it refer to a random variable that is a (function on) derived from some other random variables?

It may be a good idea to quote/cite something that uses the term/expression "hierarchical feature". I suppose that you're talking about the hierarchical nature of neural networks, which are interpreted as (hierarchical) feature extractors. However, I am not sure if "hierarchical feature" is a common term.

Yeah, @nbro. It is true. So, we can call a feature hierarchial only if it is obtained from hidden or output layers of a neural network?

I'm not sure if this term is well-defined. It's common to say that neural networks learn "hierarchical representations of the data" (because they are essentially hierarchies of layers), and representations can be viewed as "features" of the data, but I've never seen a mathematical definition of "hierarchical feature". Intuitively, hierarchical features are associated with the hierarchy that the neural network represents, but I don't know the exact answer to your question because, as I said, I am not aware of a mathematical definition of "hierarchical feature".

I guess they are just referring to the hierarchical nature of the neural network, which "stores" the "representation" of the data. For example, here, after a quick search, they use the term "Hierarchical Feature Extraction". This makes more sense to me, because they are just saying that the "feature extraction" is hierarchical, i.e. we extract the features as a hierarchy, they are not saying that features are hierarchical.

But, saying that features are hierarchical would also be ok, if you're just referring to the neural network as the representation/features of the data. Note: this is just my interpretation of that term. So, take these words with a grain of salt.

Ha, @nbro. It seems that they used the word hierarchical in a loose sense.

You can find a brief explanation of hierarchical feature selection in the following from "An Empirical Evaluation of Hierarchical
Feature Selection Methods for Classification in
Bioinformatics Datasets with Gene Ontology-based
Features" paper:

Hierarchical feature selection is a new research area in machine
learning/data mining, which consists of performing feature selection by exploiting dependency relationships among hierarchically structured features.

Therefore, hierarchical features correspond to the dependency structure between features.


text:
What kind of algorithm or approach can I use to find a specific type of object in an image?
What kind of algorithm or approach can I use to find a specific type of object in an image?
In particular, I am interested in finding an object like a windmill in an image taken, for example, from Google Maps. The image could be something like this

If you already have images, what is the average object dimension in pixel?

The computer vision problem that you are describing is object detection, i.e. the problem of finding the location of specific objects in an image and label them correctly with their names.
There are many resources on the web (or in books) that describe this problem more in detail and examples (which also include code) to get you started with it (e.g. this one).
In any case, to solve this problem, you will (probably) need a labelled dataset $D$. So, if you don't have it, you will need to (manually) collect many images, similar to the ones you describe, and find the objects of interest in them (i.e. their locations, which may be specified as a bounding box), and assign the corresponding name to each of them. At the end of this process, you should have a dataset of the form $D = \{(x_1, y_1), \dots, (x_1, y_1) \}$, where $N$ should be as big as possible, $x_i$ is an image, and $y_i$ are the labels (also known as targets), which, in this case, should be both the location and name of all objects of interest in $x_i$. The specific format of the labels depends also on the specific model that you will train to solve this problem. You can find different models online for this task, such as YOLO.

If your object is relatively small in the entire image, Yolo version 2 could be one of choices. Here is one discussion in Reddit.
If you want to see other methods, please check out list of codes from the Airbus Ship Detection Challenge.
This list is from Nikolay Sergievskiy LinkedIn profile. It looks like he was one of winners of the DIU xView Detection Challenge in 2018.


text:
Did the unsolved XOR problem in "Perceptrons: An Introduction to Computational Geometry" 1969 book really cause the winter of the AI in 1974?
Winter of AI definition:

periods of reduced funding and interest in artificial intelligence research, due to unmet expectations after a period of hype. There have been at least two major AI winters in 1974-1980 and 1987-1993.

I read this question: Did Minsky and Papert know that multi-layer perceptrons could solve XOR?

Minsky and Papert show that a perceptron can't solve the XOR problem. This contributed to the first AI winter, resulting in funding cuts for neural networks. However, now we know that a multilayer perceptron can solve the XOR problem easily.

Does someone have evidence to support that the "unsolved XOR problem" in Minsky and Papert's book caused the first winter of the AI? Or was it a succession of unsolved problems for neural networks and loss of interest that caused the cuts on funding?

Notice that the excerpt you quote uses the term "contributed", not "caused".


text:
What is the effect of gradient clipping by norm on the performance of a model?
It is recommended to apply gradient clipping by normalization in case of exploding gradients. The following quote is taken from here answer

One way to assure it is exploding gradients is if the loss is unstable
and not improving, or if loss shows NaN value during training.
Apart from the usual gradient clipping and weights regularization that
are recommended...

But I want to know the effect of gradient clipping by normalization in the performance of the model in normal or general cases.
Suppose I have a model and I run up to 800 epochs without gradient clipping because of the reason that there are no exploding gradients. If I run the same model with gradient clipping by norm, even if it is not necessary, then does the performance of the model decline?

Why gradient clipping accelerates training: A theoretical justification for adaptivity They use gradient clipping not just for RNNs, but also image recognition.

I think it's a good idea to tag your posts with more general tags, so that the context is immediately clear. For instance, in this case, gradient clipping is technique that is used for training neural networks with gradient descent, so, as I did, you could have added the tags that you see now. Of course, this is not a requirement and it's just my personal suggestion, but the tags can also clarify the context (as I said): if used only one tag and that related topic could occur in 2 different general contexts, that could make the question unclear.


text:
When would it make sense to perform a gradient descent step for each term of a loss function with multiple terms?
I am training a neural network using a mini-batch gradient descent algorithm.
Now, consider the following loss function, which is composed of 2 terms.
$$L = L_{\text{MSE}} + L_{\text{regularization}} \label{1}\tag{1}$$
As far as I understand, usually, we update the weights of a neural network only once per mini-batch, even if the loss function is composed of 2 or more terms, like in equation \ref{1}. So, in this approach, you calculate the 2 terms, add them, and then update weights once based on the sum.
My question is: rather than summing the 2 terms of the loss function $L$ in equation \ref{1} and computing a single gradient for $L$, couldn't we separately compute the gradient both for $L_{\text{MSE}}$ $L_{\text{regularization}}$, then update the weights of the neural network twice? So, in this case, we would update the weights twice for each mini-batch. When would this make sense? Of course, my question also applies to the case where $L$ is composed of more than 2 terms.

Technically, nothing prevents you from doing so. When you have mulitple losses, you may call .backward() at each term separately.
However, I wonder, whether it makes sense to optimize each individual path as a separate objective, since if we have multiple of them - we would like to solve several tasks simultaneously.
Probably, it could be beneficial as some kind of regularization. One makes steps away from the gradient, but overall in the direction, which makes the model less prone to overfitting. But the choice of the batch size, learning rate - seem more straightforward way to achieve this. There is also a ~2x times additional computational overhead since we backpropagate twice.
In some sense, It is done in the training of GAN's. Instead of backpropagating for discriminator and generator simultaneously - one calculates the loss separately for each model and updates weights.

I am not sure if the process defined in the question is meaningful at all. If you mean to simply add the contribution of each $L$ without running the algorithm for the mini-batch, it makes no difference at all if you make one update or more; as the loss functions' contribution are simply added in the update. If on the other hand you mean to run the algorithm on the mini-batch once for each $L$ before calculating its contribution including the constraints, it would most probably end up with a step in the wrong direction. Constraints (at least $L_1$ and $L_2$) do not depend on the input or the output of the model, they do not compare the prediction with the target value; they are functions of the weights only. There is no reason to expect trying to minimize this function by itself will improve the model; it will actually harm the progress by taking a (small or large) step in the wrong direction.

So, $k$ updates for each mini-batch is totally wrong?

@hanugm It is k updates for each batch, where batch means the entire dataset. One update for each mini-batch.

Oh. In my question, I used the word "batch" instead of "mini-batch".

Sorry to inform you that we made an edit. Please read it again if possible,


text:
Is there any reason behind bias towards max pooling over avg pooling?
Consider the following excerpt taken from the chapter named Using convolutions to generalize from the textbook titled Deep Learning with PyTorch by Eli Stevens et al.

Downsampling could in principle occur in different ways. Scaling an
image by half is the equivalent of taking four neighboring pixels as
input and producing one pixel as output. How we compute the value of
the output based on the values of the input is up to us. We could

Average the four pixels. This average pooling was a common approach early on but has fallen out of favor somewhat.
Take the maximum of the four pixels. This approach, called max pooling, is currently the most commonly used approach, but it has
a downside of discarding the other three-quarters of the data.
Perform a strided convolution, where only every $N$-th pixel is calculated. A $3 \times 4$ convolution with stride 2 still incorporates input
from all pixels from the previous layer. The literature shows promise
for this approach, but it has not yet supplanted max pooling.

The paragraph is mentioning that the research community is biased towards max-pooling than average pooling. Is there any rational basis for such bias?

I've found out rather a good explanation on Quora.

Max pooling extracts the most salient features - edges, cusps, whatever.
Average pooling operates smoothly - collects features, that are relevant to any part of the image.
Max pooling throws some information, It can be thought as some sense of "forgetting", whereas Average pooling depends on the whole input, despite the output representation is compressed.
There are cases, where each of these operations may not be good for feature extracting (from here):

In th first case Max Pooling will produce simply a white background, and in the second after Average pooling one will get a pale grey strip (although I think, it should be lighter, than depicted).
Probably It would make sense, to mix these two approaches, and pool half of the filters with Max Pooling, and another half with Average Pooling, although, I am not aware of the use of this approach in the literature.
The most flexible and expressive approach is the stridden convolution and the Average pooling, for example, is a particular case of it, but it introduces a certain (although not big) additional cost for storage of new parameters.


text:
Are there standardized forms of the Turing Test?
Most computer science instructors will tell you that the Turing Test is more a theoretical or conceptual thought experiment than an actual exam that someone (or something!) can formally sit and receive a score on. A thread here on AI Stack Exchange confirms this.
Considering all of this, have there been any significant attempts to create a standardized form of a Turing Test that could be rolled out widely and used to evaluate various AI constructs? Obviously, none of these standardized testing systems could be considered The Only True Turing Test (TM), but perhaps they could have their place in research as a way to benchmark or categorize various algorithms or evaluate the work of students.
For example, I'm imagining hearing a graduate student muttering the following:

My AI construct passes the Johnson-Smith Turing Test 1992 and the Hernandez-Dorfer 2017, but it's still failing the Takahashi-2003 Advanced Elite. What am I doing wrong? Maybe if I tweak this routine here. [click]. Darn, still fails.

Either a fully automated test (e.g. just login and click to sit the exam) or a standardized system involving trained human judges (e.g. a la 21st century medical board exams) who apply standardized written rubrics would be acceptable as long as the criteria for passing are standardized rather than left to the judgment of untrained personnel or random passersby.


text:
What sort of out-of-the-box technology could be used to create work similar to artist Refik Anadol?
Refik Anadol has machines view actual pictures and then has the machine create its own images. This video shows some of the stuff he does.
What kind of out-of-the-box tools (e.g. a Python package) or algorithms produce similar things to what he does?
I am hoping to play around with it and see what can happen.  Not sure what to even Google for or search for.

I have provided an answer covering the work with some ideas, but I suspect that what you are really interested in is the GANs. Hence I provided a link to a library for those and not as much detail on the other parts.

@confused, try Clip Interrogator: https://huggingface.co/spaces/pharma/CLIP-Interrogator converts his work to a description, which can then be drawn in that style: https://youtu.be/rQq5i0lTMNE

For a Google term you could use "computational creativity". It covers a wide range of ideas, and the artist here is not using one single tool or approach.
There are clearly a range of different techniques that went into the artist's installations that were shown on the video. I have an idea about a few of them:

Some are from basic animation and story-telling, and not "generated" art - at one stage the installation lists all the files that were used for instance. This is a way to communicate something about how the piece was produced. It would have been constructed into a video sequence by the artist, probably without any AI, although it is always possible to write a script that produces "decisions" from some kind of arbitrary mapping, and an artist may decide to cede control in that way as part of a piece.

There is some kind of particle system or physics-based simulation behind the flowing coloured clouds of cubes (note the physics does not have to be real physics, it often is not close, but inspired by some physical process such as flowing liquids, or maybe biological such as swarming insects). There a few different frameworks to generate these - Blender could do it for example, plus many games engines are capable of such renders. It is also common to have these types of systems hooked up to a data source that provides input of perturbations - movement, forces, shapes - that drive the dynamics, and are somehow meaninful in the context of the piece. It is not clear whether the artist has done that in this case, but it is also common to hook these systems up to data inputs in real time, such as location of gallery visitors, weather data from around the world etc. Choices of colour gradients, size and shape of elements etc will be curated by the artist to capture a certain mood.

Given the rest of this installation, the flowing cubes animation could well have something to do with the photos and how they were processed. However, it is not at all clear from viewing it whether this is the case.

The mutating photos look a lot like the output of GANs (generative adversarial networks) or VAEs (variational autoencoders). The liquid-like flow between images is caused by picking a trajectory through a latent space that these image generators learn from observing many images. A popular Python implementation of StyleGAN 2 would definitley be capable of producing output of the quality seen in the video.

Potentially the overall mix of the artwork, transitions and layouts could be a curated video sequence by the artist, or it could be dynamically mixed by a function reacting to a live data feed. In this case, the mixing seems more on the human-curated side - the installation could be e.g. a 15 minute video loop showcasing a fixed rendering of interesting parts of the spaces selected by the artist.

There are probably other things. For instance, the artist has gone to some trouble to curate their data, also using machine learning techniques when they discuss filtering out images of people. I think the effort that has gone into the installation, and the desire of the artist to showcase that is why some of the installation includes storytelling about how it was made.

Thanks!  Yeah I was mostly interested in how to get a computer to recognize images and I guess learn from it so it can create it's own.  The after effects I can probably figure out - plus that will depend on style.  But this is very useful starting point on things to look at.

@confused If you think this answer is useful, you should upvote it.


text:
How should I define the reward function for a stock trading-like game?
Problem setting
Consider a game like trading a stock

At each step, the agent can buy / sell a stock.
Trade is a pair of buy and sell actions.
We can set the reward for the sell action as the profit made in this trade. Although this might be misleading, since whether a trade is good or not depends on both buying and selling timing.
We don't know the reward for buy, whether a buy is good or not depends on when you sell.

Questions

How should the reward scheme be for a game like this? i.e., whether one action is good or not depends on other actions taken before it?
Can we set a scheme like this: set the reward to be zero at each step, no matter it is a buy or a sell. Only give the reward at the end of the game, say after 1000 steps, and the reward, in this case, is the total money made?

how should the reward scheme be for a game like this? i.e., whether one action is good or not depends on other actions taken before it?

The reward scheme should always be a "natural" representation of the goals of the agent. If the goal of your agent is to make a profit, then the reward should be the amount of profit*.
It is important to RL, and becomes critical for rewards that may be delayed, that the state representation captures all the information that impacts those rewards. That means representing the direct effects of actions, so that they can be tracked. In the case of a trading bot, it will be important at a minimum that the state includes the current portfolio. If the agent can access data about the amount and value of stock it currently holds, it has a chance to predict likely profits later.

can we set a scheme like this: set the reward to be zero at each step, no matter it is a buy or a sell. Only give the reward at the end of the game, say after 1000 steps, and the reward, in this case, is the total money made?

You can, but that looks like it is making the problem harder than it needs to be.
I would naturally set the reward to be the profit on each transaction (including negative reward for buying stock). The agent should be able to use that to figure out long term rewards, and will learn faster if it is given this more direct feedback. RL algorithms are well-suited to figuring out the need to invest at an earlier time in order to make profit later. If there is also a time horizon, that should be factored into the state - the agent should be made aware that the game will end in so many time steps otherwise it may end up holding stock instead of selling it in time to meet the final evaluation.
One important caveat is that real trading problems are hard, because the real environment is highly variable and competitive. The competition includes intelligent people with advanced qualifications in statistics. The chances of making a trading bot that is successful in a real trading environment are low.

* Sometimes this "natural" rerpresentation is too sparse, and you may want to look into reward shaping. That is adding interim rewards or spreading out an existing reward, to assist the learning agent. For a stock trading game for instance, it may help to allocate some reward for gaining ownership of a stock, perhaps the nominal resale value minus the fees for selling it.
However, where it is possible to stick with the simplest direct interpretation of the agent's goals, then you should do that. Reward shaping comes with the potential risk that the changes you have made will impact what it means for the agent to perform optimally.


text:
When can I call an entity a hyperparameter?
As per my knowledge, any entity that is learnable by a training algorithm can be called a parameter. Weights of a neural network are called parameters because of this reason only.
But I have doubts about the qualification of hyperparameter.
Hyperparameter according to my knowledge is an entity that needs to be learned outside of the training algorithm. But, a lot of entities can come into the picture if I want to follow this definition.
For example, the selection of the type of neural network, number of layers, number of neurons in each layer, presence of batch normalization layer, type of activation function, number of parameters, type of parameters (integer, float, etc.), number of epochs, batch size, type of optimizer, learning rate, etc., and I can able to list a lot of entities like this.
Is it okay to call anything that needs to be learned outside the training algorithm a hyperparameter?

In older machine learning literature the given definition of hyperparameters was explicitly the same used in Bayesian statistics, i.e.

a hyperparameter is a parameter of a prior distribution

For example, in Christopher M. Bishop's "Pattern Recognition and Machine Learning" (Springer, 2006), hyperparameters are introduced in the following paragraph (page 30)

Now let us take a step towards a more Bayesian approach and introduce a prior distribution over the polynomial coefficients $\mathbf{w}$. For simplicity, let us consider a Gaussian distribution of the form
$$
p(\mathbf{w} \mid \alpha)=\mathcal{N}\left(\mathbf{w} \mid \mathbf{0}, \alpha^{-1} \mathbf{I}\right)=\left(\frac{\alpha}{2 \pi}\right)^{(M+1) / 2} \exp \left\{-\frac{\alpha}{2} \mathbf{w}^{\mathrm{T}} \mathbf{w}\right\}
$$
where $\alpha$ is the precision of the distribution, and $M+1$ is the total number of elements in the vector $\mathbf{w}$ for an $M^{\text {th }}$ order polynomial. Variables such as $\alpha$, which control the distribution of model parameters, are called hyperparameters.

In modern machine learning literature though, the definitions became more operational. For example, in Ian Goodfellow, Yoshua Bengio, Aaron Courville - Deep Learning (2016), we can read

Most machine learning algorithms have hyperparameters, settings that we can use to control the algorithm's behavior. The values of hyperparameters are not adapted by the learning algorithm itself (though we can design a nested learning procedure in which one learning algorithm learns the best hyperparameters for another learning algorithm).

So, there is room for interpretation, even though I personally find more technical and precise the old reference to Bayesian statistics. It is clear from that definition that every variable not belonging to the parameters used in the prediction phase but only during training are indeed hyperparameters. Moreover, it is clear that the choice of hyperparameters affects the distribution of learned parameters once the model training reaches convergence.
To elaborate a bit more on the modern definitions, what I don't like about the example taken from Deep Learning is the lack of further explanation about the meaning of "model behavior". Does it refer to weight updating during training? Final metrics score? Both and more? In other words, what are hyperparameters supposed to affect? Of course, these loose definitions do not stop machine learning practitioners from using hyperparameters in the right place, but no surprise about doubts emerging like this question.

I would also like to note that the 2 definitions that you quote are not necessarily inconsistent with each other.

Is it okay to call anything that needs to be learned outside the training algorithm a hyperparameter?

I think so, yes.
Personally, I would reserve the term to discuss values that I could choose freely in any given experiment, but that were not learned directly from the training data by the model class I was using. Or perhaps with a tighter restriction for practical purposes, things I would be willing to investigate and change in order to achieve the goal of getting good metrics from the learning agent - it doesn't matter from a practical perspective what I label a value for something that could be changed, if I am not considering changing it.
Choices that I am not interested in making for a given experiment might be hyperparameters to someone else who is performing a different kind of search.
Discrete, top-level choices such as whether to use linear regression versus XGBoost versus a deep neural network might technically be considered hyperparameters, and a sophisticated search algorithm may even automatically test across them. However, I rarely see this kind of choice discussed using the term hyperparameter.


text:
Why does TD (0) converge to the MLE solution of the Markov model?
Why does TD (0) converge to the MLE solution of the Markov model?
Let's take the Example 6.4 in Sutton and Barto's book as an example.

Example 6.4: You are the Predictor Place yourself now in the role of the predictor of returns for an unknown Markov reward process. Suppose you observe the following eight episodes:
$A,0,B,0; B,1;B,1 ;B,1 ;B,1 ;B,1 ;B,1 ;B,0$
...
But what is the optimal value for the estimate $V(A)$ given this data? Here there are two reasonable answers. One is to observe that $100 \%$ of the times the process was in state $A$ it traversed immediately to $B$ (with a reward of $0$); and because we have already decided that $B$ has value $\frac{3}{4}$, therefore $A$ must have value $\frac{3}{4}$ as well. One way of viewing this answer is that it is based on first modeling the Markov process, in this case as shown to the right, and then computing the correct estimates given the model, which indeed in this case gives $V(A)=\frac{3}{4}$. This is also the answer that batch $\mathrm{TD}(0)$ gives.

Given the TD(0) update rule $V(S) \leftarrow V(S)+\alpha\left\lceil R+\gamma V\left(S^{\prime}\right)-V(S)\right]$, how can we deduce that it will get the MLE solution and thus $V(A) =\frac{3}{4}$?

As a general comment, convergence proofs in RL don't really exist for any nontrivial settings. Q learning, discrete spaces and I have read something involving linear approximation but overall they are extremely difficult.

Your question is very similar to this one, but you also provide a specific example, so I will not close this one as a duplicate of that one.


text:
What is the place of ontologies in artificial intelligence?
Very much a general question here, from a somewhat uneducated perspective.
I'm currently part way through an MSc in AI, and at the minute I am taking a module on Knowledge Engineering and Computational Creativity. The professor taking the class obviously does research in this area and is saying that ontologies are becoming very important in the world of AI, or it may be more accurate to say he is suggesting they are becoming more important.
I intend to look into the work he does, and ask him a few questions, but, generally, I was wondering where this type of research sits in the world of AI. Is it something being worked on a lot? Is it becoming bigger?
I am interested because I do find the topic interesting, and I will have a research project coming up soon, and while I do want to work on an interesting topic, I also want to work on a relevant topic, so any information would be great.

Hello. Could you please focus on 1 specific topic (either ontologies, semantic networks, or logic, but not all 3 at the same time) at a time? I am asking you this so that people can focus on one topic at a time. Maybe someone only knows the answer to the question "Where are ontologies being used nowadays?" (which seems to be one of your questions, if I interpreted your post correctly; if not, let me know) and knows nothing about semantic networks. You can ask the related questions in a separate post and I would recommend that you do that.

@nbro - Hello! Yes of course, not a problem. I will edit my post to focus on a single topic. And yes, you have interpreted my question correctly. Thanks for the feedback too.

I found a definition: "In AI, an ontology is a specification of the meanings of the symbols in an information system." To me this relates to semantics and surely some area computational linguistics.  It's in this view it's a technical subject, related to input and output, but it's also a deeply philosophical subject, and has been heavily treated in the speculative literature.

An example of the philosophical dimension is the "Grecian Room" where two subjects who can't see each other converse in Ancient Greek. Neither subject can know if the other actually understands the language, or is just following a set of instructions to simulate knowledge. From a technical perspective, directed graphs are and n-dimensional network topologies are examples. (The Symbol grounding problem is an example of a technical challenge that is thought to have philosophical implications.)  So I'd say ontology is pretty central to AI in both domains.

Your question is somehow unclear/ambiguous. What exactly are you asking? Are you asking 1. "Where are ontologies being used in the context of AI?" or are you asking 2. "Is research on ontologies being done in AI, and for what purpose?". I guess these questions are related, but "place of ontologies in AI" to me seems ambiguous and your questions in the body of the post suggest that you're only interested in whether people in AI are doing research on ontologies. So, please, edit your post to clarify your question.

The answer below seems to answer the question "What is an ontology, and where it's used"? But doesn't directly address or emphasize where ontologies are used in AI.

Computational creativity is not something I know anything about. However, I work in knowledge engineering. This falls into the areas of knowledge representation and reasoning known as semantic web technologies in general. More recently Google has popularized the name "knowledge graphs" and now many people tend to talk of knowledge graphs rather than the semantic web, even though, strictly speaking, knowledge graphs are a small subset of semantic web technologies.
This is a massive field of research, which is widely used in bioinformatics, healthcare, property management, and many other fields to enable semantic search. I myself work in bioinformatics where we use 265 ontologies spanning close to 7 million concepts that are used to enable semantic searches across around 300 petabytes of data. In fact, it is knowledge graphs that are at least in part used to enable Google searches and information provided Google info boxes. Hence, many people are already using the results of knowledge engineering without knowing it.
So what is an ontology? An ontology defines concepts and relations between concepts. This has been done in computer science for ages, so what makes ontologies and the semantic web so special?

Each concept and relation is assigned globally unique identifiers, e.g. URI, IRI, PURL
Ontologies are defined in a machine-readable syntax, e.g. XML, JSON-LD
The semantic web defines a generic data model able to describe arbitrary data called RDF triples. In particular, this helps people to define their data before having to define a schema (as is the case with relational databases).
An RDF triple <s, p, o> expresses that subject (s) and object (o) is related via predicate (p).
SPARQL is a query language for querying RDF triples
Ontologies are equipped with formal semantics based on mathematical logic, which enables artificial intelligence reasoning procedures to infer implicit knowledge from explicit knowledge. E.g. RDFS and OWL.
JSON-LD, RDF, SPARQL, RDFS and OWL are W3C standards.

Here is a book that can give you a reasonably gentle introduction to aspects of the field and how it is used in practice: The knowledge graph cookbook.

Hi! That is a fantastic answer and does help a lot, and gives me other things to look into. I don't have the reputation to upvote your answer, but when I do, I will. Thanks for the reply and the information!

Hello. Thanks for contributing to our site! Your answer seems good. You may be interested also in this question.


text:
In this example of fuzzy c-means, what is the difference between "sigma" and "center" for the clusters?
In this example, what exactly do "Cluster" and "Sigma" mean? (They chose random coordinates for the three centroids of the groups)

Centers: Cluster centers, returned as a JxN array, where J is the number of clusters and N is the number of data dimensions.
Sigma: Range of influence of cluster centers for each data dimension, returned as an N-element row vector. All cluster centers have the same set of sigma values.

Please, elaborate on the difference.

In the example you link to, the sigma parameter has got nothing to do with the clustering; it is only used to generate sample data for illustration. It defines the spread of cluster elements around the (pre-defined) centroid of each cluster.
This is done for demonstration only: you generate clusters which you know exist, and then check that the cluster algorithm can detect those clusters correctly. In normal centroid-based clustering the centroid does not have a specified range -- each data point it simply assigned to its nearest centroid, however far away it it.


text:
How to choose the new layer and objective function for transfer learning on a neural network?
I have a base model $M$ trained on a data say type 1 for task $T$. Now, I want to update $M$ by applying transfer learning for it to work on data type 2 for the same task $T$. I am very new to AI/ML field. One common way I found for applying transfer learning is to add a new layer at the end or replace the last layer of the base model with a new layer, and then retrain the model on new data (type 2 here). Depending upon the size of type 2, we may decide whether we retrain the whole model or only the new layer.
However, my question is that how do we decide following:

What should be the new layer(s)?
Should the objective function while retraining be the same as the one used for the base model, or it can be different? If different, then any insights on how to figure out a new objective function?

P.S. Data of type 1 and type 2 are of the same category (like both are logs or both are images), however are significantly different.

What do you mean by "same task $T$"? Do you mean one of the two general tasks of classifying or regressing? If you change the dataset, then you may also be changing the task, that's why I am asking this question.

By same task I mean that they are doing the exact same thing. However the data is changing considerably in my case. Hence, I am trying to fine tune the model for the new data (very small size available) using the transfer learning techniques. Hope that answers yours question @nbro

Can you be more concrete? For example, the task is classifying dogs vs cats, but the images are different (e.g. they have higher resolution in the second dataset?)

Lets say the task is about compressing some data using autoencoders. The AE is trained on one type of log data now a variation of data has come up and need to update the AE accordingly. @nbro


text:
Is there any point in adding the position embedding to the class token in Transformers?
The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches.
Is there any point in doing so?
The purpose of introduction positional embeddings to the Transformer is clear - since in the original formulation Transformer is equivariant to permutations of tokens, and the original task doesn't respect this symmetry one needs to break it in some way to translational symmetry only - and this goal is achieved via the positional embedding (learned or fixed).
However, the class token is somehow distinguished from the other tokens in the image, and there is no notion for him to be located in the [16:32, 48:64] slice of the image.
Or this choice is simply a matter of convenience? And additional parameter, indeed, has a negligible cost, and there is no benefit as well as harm of the addition of positional embedding to the [CLS] or any special token?


text:
What is the best way to train a text-based regressor model?
I want to build a deep learning model that can predict a continuous value (LogP in this case) given text inputs (SMILES notations in this case), the dataset is as illustrated below.

SMILES notations
LogP

C1CCCC(C)(C)1
1.98

...
...

I have never tackled text data, I mostly worked with numbers-based datasets (or images).
My questions are:

What is the best model for this case? I believe RNN based architectures, such as LSTM and GRU, are the most suitable.
What about recent architectures such as Transformers?
How can/should I convert (or embed, or encode) the text inputs (SMILES) to feed them to my model?


text:
How to detect the sine wave signal with different frequency using neural networks?
I'm wondering if there is a way to use a neural network that can detect the noisy sine wave, where the frequency is not constant. In other words, I'm not looking for a solution that would detect the signal of a particular frequency (say 50Hz), but a solution that can detect any sine wave signal in say range (100-1000Hz).

Probably. Just train it on a bunch of different noisy sine waves and signals that aren't.


text:
How to train neural networks with multiprocessing?
I am trying to figure out how multiprocessing works in neural networks.
In the example I've seen, the database is split into $x$ parts (depending on how many workers you have) and each worker is responsible to train the network using a different part of the database.
I am confused regarding the optimization part:
Let's say worker 1 finished first to calculate the gradient, now it will update the network accordingly.
Then worker 2 finished the calculation and it will also attempt to update the weights. However, the gradient it calculated was for the network before it was updated by the first worker. Now, the second worker will attempt to update the network with a bad gradient.
Did I miss something?

depending on your ML framework (e.g, TF, pytorch), beside the docs they have pretty nice explanation on several of their ways of distributed training. ANW, check out the Hogwild! training https://www.youtube.com/watch?v=l5JqUvTdZts

multiprocessing or gpus allows you to process several batches at the same time, but indeed gradients and optimizer can be updated only after all relevant batches have been processed. So usually one process or one gpu is reserved for this specific task. Pytorch for example collect all gradients always in cuda0 (the first, or the only, gpu available).

If you think about it this is isn't too different from how non-multiprocessing training works - we calculate the gradient for a whole lot of data items, without updating the network, then we update the network with all the gradients at once!

@Sanyou This is an important question, although it's quite general, given that there may be different ways to do this. If you can provide a formal answer, I would suggest that you do so, because it may be useful to other people in the future too.

@EdoardoGuerriero I wanted to write the comment above by also mentioning you, but I couldn't do that. It seems that you have some useful information that at least partially answers the question. Maybe you can elaborate on what you wrote above to also write a formal answer below.