text:
The SOTA of derivative-free optimization
As titled, I want to ask what is the SOTA of derivative-free algorithm.
I am not familiar with this thing at all, the only derivative-free optimization algorithm I am familiar with is GA, and others like Bayesian Optimization, I just know the name, but I don't even know what they do.
The model I want to optimize is some bi-lstm models with about 3 layers, which has the input shape of (timesteps~=1024, features=512), and the output shape of (timesteps, 256).
What I also know is that the model is a little big, but the loss calculation is not differentiable, and RL algorithms are not suitable for me, so it lead me down to this.

For some implementations you can look at nevergrad from facebook.

The problem is not the input size but the model size.
Indeed, derivative-free/zero-order optimization methods usually tend to estimate a descent direction that correlates with some notion of local gradient (that might happen not to exist because the loss is not differentiable)
Now, you can consider zero-order optimization as a "practical" finite difference method:
$$
f'(x) = lim_{\,h\,\rightarrow 0} \frac{f(x+h) - f(x)}{h}
$$
Thus, what they aim to do is to find a multidimensional direction $h$ that is a descent direction
At this point, you can see that in order for such direction to have a span of the whole parameter space, you need $N$ function evaluation (in the general case), thus will be very computationally expensive to optimize the problem if your bi-lstm is very big in model size
However, nothing stops you from taking a direction that might not span the whole space
Clarified that the problem is the model size, you can consider any evolutionary strategy algorithm for such problem, something like Cross Entropy Methods (CE) or CMA-ES should work fine for your problem
You can find improved versions in the citations of the paper of such methods, though they might lack an implementation in the programming language you are using

Thanks, I will investigate more. Can I understand the formula you mentioned as follows: create an estimator of loss using maybe a neural network, and during the update, we both calculate the distance of the estimator and the real loss, and then update the model using estimator loss?

no, that was just an intro to CE and CMA-ES, which is "perturb your weights x by random quantity/direction h N times, and move toward the ones that worked the best"


text:
Direct formula for calculating the optimum matrix which minimizes the perceptron error
Suppose we have a perceptron without bias and $f(x) = x$ as activation function and matrices $X,Y,W$ that input training data are columns of matrix $X$, $Y$ is targets matrix (columns are ordered with attention to the related inputs) and $W$ is the weights matrix in perceptron. Also $X_i$ denotes the $i$th column in matrix $X$ and so on for other matrices. We know we want to minimize this:
$$
E(W) = \sum\limits_{i} ||WX_i - Y_i||^{2}
$$
In a class I saw someone said we can find the optimum $W$ by by calculating  $\frac{d}{dW}E(W)$. Then with attention to the
\begin{align*}
||V||^{2} = V^{T}V \ \ \ \ \ \ \ \ \ \ \ \ \ \ & (1)
\end{align*}
, when $V$ is a columnar vector, he continued:
\begin{align*}
\frac{d}{dW}E(W) = 0 & \Rightarrow (WX - Y)X^{T} = 0 \\
& \Rightarrow W = YX^{T}(X^{2})^{-1}
\end{align*}
He claimed that above formula gets the optimum $W$. But I do not know is that true and how does he calculate $\frac{d}{dW}E(W)$? If it is not true, is there any correct direct formula for achieving $W$? Also can you provide references for more studying?

The idea is correct, the last formula is wrong. In general $X$ will not be square, usually one has much more data than parameters. The data points will also be in general position, so that $X$ has maximal rank and $XX^T$ is invertible. The equation then transforms to
$$
WXX^T=YX^T\implies W=YX^T(XX^T)^{-1}.
$$
There is no way to get an $X^2$ into the formula.
With a SVD $X=U\Sigma V^T$ where $U$ is square, $\Sigma$ invertible and $V$ isometric one gets $W=YV\Sigma^{-1}U^T$, which is connected with the idea of a pseudo-inverse.

Thank you very much. But also can you propose a good reference about these operation to me? I want to study more in this topic.

What you are doing here is just linear regression. This should be in most linear algebra and statistics texts. See Moore-Penrose pseudo-inverse.

As you understand, $E$ is the definition of loss function.  This function defines square of the difference between weights applied to $X_i$, namely output of the perception, and $Y_i$ the desired target value (as a ground truth). So, for finding the optimum $W$ we should find a one that make the derivative of the loss function zero, based on the given $X$s and $Y$s.
As, the error function $E$ is a convex function (a square function which is always positive), our finding from making the error function zero must minimize the error function.
So, just we need to compute $\frac{d}{dW} E(W)$ (derivative of the error function over $W$). In general derivative of a function in the form of $(zx+b)^2$ based on $z$ is equal to $2 \times (zx+b) \times x$. Similar to this case, you can do the same for matrices. Hence the derivative of each term in $E$ can be phrased as mentioned in the question, i.e., by getting derivative of each term of the sum in $E$ and rewriting in a matrix format (the mapping between your question and this example is $z = W$, $x = X_i$, and $b = -Y_i$). Note that multiplication of $2$ in each term of the result has been removed in the case, as it does not impact on the root of the equation as a constant factor.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is PPO a policy-based method or an actor-critique-based method?
as far as i understand there are 3 categories of Reinforcement algorithms:

Value-based methods (like DQN or Sarsa)
Policy-based methods (like REINFORCE)
Actor-critic-based methods (like A2C)

To which of those categories does PPO (Proximal Policy Optimization) belong to? As the name suggest it is based on policy optimization but I think it also uses an actor-critic-based structure?

PPO belongs to both category 2 and 3:

It has an actor-critic structure, meaning that it learns the optimal policy via the actor network, and the value function via the critic network.
Moreover, differently from SAC for example, the actor is learned via policy gradient (not Q-learning as in SAC). Therefore it is also policy-based.

Update: defining value-based, policy-based, and actor-critic agents - I'll provide an intuitive rather than technical definition.
In general, a reinforcement learning agent can be equipped (let's say) with one or more components such as a (figure from David Silver's lecture 1):

Policy, representing the agent's behavior function typically mapping states to actions;
Value function, i.e., a function or model that predicts the expected future reward;
Model of the environment that predicts its next state.

Now, not all these components are required (e.g., the env's model) and some of them can be implicit (like the policy, which can be extracted from the value function.) According to which is the "main" component, we can categorize agents into:

Value-based, which need only to learn a value function (either the state-value or state-action one). For such agents learning the value function is enough since the policy can be "extracted" from it. For example, from a Q-function is very easy to derive a policy that predicts the action associated with the higher state-action value. You can also have a policy from the value function itself, but a one-step model of the environment is needed since the future is involved. TD-learning and Q-learning are both examples of value-based agents. Moreover, we consider value functions the state-value, $V(s)$, action-value (or state-action) function, $Q(s,a)$, and the advantage function, $A(s,a)$.
Policy-based methods that learn directly a policy instead of first approximating a value function. The policy can be deterministic (i.e., a mapping) or stochastic (a probability distribution.) In both cases, a popular approach is to learn the policy's parameters by following the (deterministic) policy gradient, which induces a further specialization of this class of agents. Another way is to use evolutionary strategies (ES), or optimization algorithms. Classical REINFORCE and DPG are example policy-based agents.
Actor-critic agents, which merge both value- and policy-based methods by learning two models: the actor (policy) and critic (value function). Now the critic can be found having various roles, from variance reduction serving as a baseline (e.g., in A2C, PPO) but also as a way to learn a policy (e.g., DDPG, SAC, TD3).  The important thing is that we don't use the critic to derive an implicit policy.

Thanks for your answer Luca. So this means that there are also mixtures of the 3 basic RL algorithms (value-based methods, policy-based methods, and actor-critique-based methods)? I thought that every RL algorithm can only be assigned to one single category. Further is REINFORCE only a policy-based method and A2C only a actor-critique-based method or also mixtures?

But aren't actor-critic methods by definition also policy-based? I think that's not clear from this answer. Maybe a definition of policy-based and actor-critic methods is required here.

@nbro: Thanks for your comment. Actually I don't know. As far as I understand there are those 3 classes for RL algorithms. Apparently in REINFORCE there is no actor-critique structure while in PPO there is and both are policy-based methods. Actor-critique based methods combine value-based and policy-based methods as far as I understand

@PeterBe In classical REINFORCE there is no actor-critic structure, since the state's value is estimated directly by the empirical return (or rewards-to-go if you prefer.) If you then introduce a baseline, for variance reduction, you may turn it into actor-critic if the baseline is a learned model of the value function.

@nbro In general the definition is a little bit shaded. E.g., Silver's lecture 1 (see page 36) defines the actor-critic at the intersection of value- and policy-based methods. Instead, the spinning up RL's taxonomy lists both PPO, and A2C/A3C under "policy optimization", as a wider more general family than the policy gradient.

@LucaAnzalone: Thanks Luca for your comment and effort. But would you generally also classify RL algorithms into value-based, policy-based and actor-critic based methods?

@LucaAnzalone: Thanks for your comment. Any comments on my last comment? I'll highly appreciate every further comment from you.

@PeterBe have a read at my latest update. Hope it helps. :)

Thanks a lot Luca for your great answer. It is definitely helpful.

@LucaAnzalone: Thanks for your answer Luca. It's me again. In your very first sentence you write "PPO belongs to both category 2 and 3". But that is not true. PPO belongs to the Actor-Critic methods as far as I understand as it combines the value-funciton approach with the policy optimization approach

@PeterBe True: PPO is best classified as actor-critic, indeed, but it's also a policy-based method. I mean, this is a broader class than Actor-critic, which is more specific

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Why is the better policy defined with respect to all the states values being greater?
In Sutton & Barto (Section 3.6 - Optimal Policies and Optimal Value Functions), they say that :

Value functions define a partial ordering over policies. A policy $\pi$ is defined to be better than or equal to a policy $\pi^{'}$ if its expected return is greater than or equal to that of $\pi^{'}$ for all states. In other words, $\pi \ge \pi^{'}$ if and only if $v_\pi(s) \ge v_\pi^{'}(s)$ for all $s \in \mathbb{S}$.

My question is, why is the better policy defined with respect to all the states values being correspondingly greater instead of a combined metric of all state values of a policy?
If there is a policy that gives me the highest rewards on 99 out of 100 states, but gives me a lower reward on the last state compared to a second policy (which performs poorly on the other 99 states), would this first policy not be considered an optimal policy according to the definition above?

No, the answer of @foreverska is wrong, otherwise they would have said “better givena specific $\mu(s)$”. The reason is simply that given 2 policies, where one performs better than the other only in a subset of states, then you can create a strictly better third policy combining the two, where you consider the first one in the states where it outperforms the second one, and the other way around
in this case, the two initial ones are not in any specific order, but the third one (the combined one) has a order wrt the initial two

This is the answer I found in the University of Alberta's RL course as well. It also made me realize that my confusion was stemming from why an optimal policy (as defined above) should be guaranteed to exist at all.

@pew31 that's exactly from where it comes from, and i see how that might be confusing

If the subsequent policy is better in 99 out of 100 states this would seem to be better.  But imagine if the next policy in the iteration is also better in 99 out of 100 states but the one state is a different state than the first.  This third policy is worse than the original in two states (if the increase in the first bad state is less than the original decrease).  This pattern can be repeated until we arrive at a maximally bad policy.  A maximally bad policy is likely not greater than or equal to the first policy in this case and thus the chain of greater than or equal to must be broken somewhere.
More practically, the 99 out of 100 example does not specify how often that one state occurs.  If it is the state the agent finds itself in 99.9% of the time, this can quickly make a worse total reward for the agent.
This is why it is necessary to be better or equal to the previous policy in all states.  A policy which is better in 99 out of 100 states may achieve a higher reward for the agent and thus be considered better.  But the definition given by S&B points forward toward policy iteration and the guarantee that it achieves an optimal policy so it is a bit more strict.


text:
Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?
Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control):

A potential problem is that this method learns only from the tails of episodes, when
all of the remaining actions in the episode are greedy. If nongreedy actions are common,
then learning will be slow, particularly for states appearing in the early portions of
long episodes.

Could someone please explain clearly the above paragraph?

From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have:
If $A_t \neq \pi(S_t)$ then exit inner Loop (proceed to next episode), otherwise scale importance sampling ratio $W$ by $\frac{1}{b(A_t|S_t)}$.
Therefore here samples collected towards the end of episodes where the importance sampling ratio is large, contribute more to the estimation of the expected returns of the target policy. These high-weight samples dominate the update process especially if the behavior policy is more exploratory where $b(A_t|S_t)$ is very small, leading to learning primarily from the tails of episodes in the sense of return/value updates with greedy actions.
If nongreedy exploratory actions are common in the early portions of long episodes, the said inner loop would always exit before these early non-greedy actions, causing no feedback or learning about their credit assignment. For instance, if an agent went for a suboptimal long route in a large maze caused by early non-greedy actions, there's no learning to correct this due to the above demonstrated tail learning only nature. The optimal route learning would be very slow on average until some episodes happen to enter the optimal route by the behavior policy's early non-greedy actions.


text:
Is it easier to use back-propagation or genetic algorithms to teach an artificial intelligence?
I am making a very simple neural network for a school project, and I would like to know what the best and easiest way to "teach" a neural network would be. From what I know, backpropagation requires a lot of mathematical knowledge. Whilst Generational is gene-based and creating mutations doesn't. If there is another way to do it, that would be great to hear. I just want someone else's opinion and perspective before I start programming.

Training neural network by backpropagation doesn't require much mathematics, especially with the plethora of libraries available. With the libraries available, it would be more computationally efficient and tuning parameters would be a lot easier. This heavily contrast with Genetic Algorithms where not a lot of libraries are available and not enjoying as much learning resources. One thing about Genetic algorithm though is that you don't need a differentiable performance function and you don't need human data. But even in those areas, Deep Reinforment Learning algorithms, which still uses backpropagation, far exceeds the performance of evolutionary algorithms.

From what I know, backpropagation requires a lot of mathematical knowledge.

It's not a huge amount of knowledge. High school (USA) or A-level (UK) maths is enough to understand the principles. There is a lot of detail, and that may take time you don't want to spend just yet. If you use a pre-made library for the neural network though, you don't have to study this in any depth. Instead, you can think of the maths for the learning stage as a library function that you supply correct input/output pairs to, that you run multiple times and monitor results.
In addition, although a genetic algorithm is perhaps a simpler principle to explain and understand, getting the most basic version to work with a neural network task is quite hard in practice, and may take some time to tune.
Provided you are not wanting to build the network logic from scratch, I would recommend using a library either way (for genetic algorithms and neural networks combined for a small network, I would recommend the NEAT algorithm and a library that implements that). The libraries hide a lot of complexity of the maths involved, and there is still plenty to learn and understand about the principles of using machine learning to approximate a function.
Ultimately, what to choose depends on the nature of your project. You don't want to be over-ambitious, but also want to have something interesting in your write-up:

If your primary goal is to write a network from scratch, you should pick a really simple goal for the neural network - a typical toy problem is to learn to copy the XOR function. That will significantly limit the amount of maths you need to study to get it to work from first principles.

If your primary goal is to use a neural network to "do something cool", then you could use library code and tackle a toy problem such as recognising hand-written digits (the MNIST dataset).

One nice ting about using a genetic algorithm is that you can run it on simple simulated games, because you can base the GA improvements based on results from using the network, as opposed to needing to know correct input/output pairs for training. A controller for a simplified game, like flappy bird or for a model car driving around a track is possible in a small project.


text:
Is the Bellman backup unbiased?
This is comes from cs285 2023Fall hw3.

In my opinion, if $\hat{Q}$ is unbiased estimate of $Q$, then
$$
\begin{align}
\mathbb{E}_{D \sim P}[B_{D}\hat{Q} - B_{D}Q]
&= \mathbb{E}_{D \sim P}[r(s,a) + \gamma max_{a'}\hat{Q}(s', a') - r(s,a) - \gamma max_{a'}Q(s', a')]\\
&=\mathbb{E}_{D \sim P}[\gamma max_{a'}\hat{Q}(s', a')- \gamma max_{a'}Q(s', a')]\\
&= 0
\end{align}$$
So $\hat{Q}$ is unbiased estimate of $Q$ and the answer is yes.
Is this right?

Can you provide the link to the homework?

Given your provided definition of unbiased estimator at each input data, and since $\hat{Q}$ is an unbiased estimator for the true action value $Q$ at each state-action input pair, applying Bellman backup operator defined in your reference involving a max operator would result in biased $B\hat{Q}$ not converging to $BQ$ in expectation at each state-action input pair. The distribution of the maximum of $n$ independent i.i.d. random variables doesn't necessarily follow the same distribution, it may follow Gumbel distribution for example, which may not have same expectation as the original distribution, as pointed out by another answer.
Of course in reality TD off-policy Q-learning cannot ensure there always exists such an unbiased $\hat{Q}$ estimator at each state-action input.

No, $B\hat{Q}$ is not an unbiased estimator. Say that $Q(s, a) = 0$, and $\hat{Q}(s, a) \sim N(0, 1)$. Then $E[max_a \space Q(s, a)] = 0$, but $E[max_a \space \hat{Q}(s, a)] > 0$ (approximately 1.5387 if you have 10 different actions).


text:
What is the difference between feature extraction and fine-tuning in transfer learning?
I'm building a model for facial expression recognition, and I want to use transfer learning. From what I understand, there are different steps to do it. The first is the feature extraction and the second is fine-tuning. I want to understand more about these two stages, and the difference between them. Must we use them simultaneously in the same training?

Hello. Welcome to AI SE. Could you please provide the link the resource where you were reading that claims that "feature extraction" and "fine-tuning" are two different approaches to transfer learning? Because I don't think that's the case. I think these are just 2 stages of transfer learning, and maybe this would be the start of an answer to your question in the title, but I could be wrong.

Hello. Yeah I just realized that I was totaly wrong. Yes there are 2 stages in transfer leraning but should we use them in the same training or there is a difference ? Thank you for your reply.

It has been a while since the question was asked, but I came up with this article. It helped me to understand the topic. From the article:
Feature-based methods involve using the intermediate representations or features from a pre-trained model as additional inputs to a task-specific model.
Fine-tuning, on the other hand, involves modifying and retraining a pre-trained model to adapt it to a specific task.

The difference between the two approaches (feature extraction vs fine-tuning) is well explained here:
Fine Tuning vs Joint Training vs Feature Extraction
Also, this paper evaluate the performance one can hope to achieve with 2 sequence models (ELMo and BERT) with each approach:
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

Hello. Welcome to our community! Could you please provide more details and not just provide a link to an external source? We usually expect answers to be self-contained and links should only be used to provide additional details, unless the person was specifically asking for a reference/link.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Typically, in transfer learning, you have 2-3 stages

Pre-training: pre-train some base model $M_\text{base}$ on some "general" dataset $A$; note that you may not necessarily need to train $M_\text{base}$, but it may already be available e.g. on the web. During this phase, we extract (general) features or learn representations of the data, which can "bootstrap" the learning task with your specific dataset

Training:  You replace the last layers of $M_\text{base}$ (i.e. the classifier/regression part) with new layers to solve your task, then you might freeze the initial layers (e.g. the convolutional layers) that are assumed to contain the general extracted features that can also be useful for your task: let's call this model $M_\text{main}$; at this point, you train this partially frozen model $M_\text{main}$ with your dataset $B$.

Fine-tuning: after training, you could unfreeze some of the frozen layers in $M_\text{main}$, especially the ones closest to your new classifier, then train again

In all 3 stages, one could say that we're extracting features (because we're learning weights), but some people, I guess, will refer to the pre-training phase as the feature extraction phase. I think I've seen people call the training stage also the fine-tuning stage (and the previous version of this answer actually was referring to the training phase as the fine-tuning phase), but, in the end, these terms could be used inconsistently anyway, so the important thing is that you understand what's going on and keep context into account.
You can find more information about this topic here. Note that there may be other more sophisticated or simply different approaches to transfer learning.

Fine-tuning most often does not include freezing the previous trained layers. Actually it is more likely related to training the whole model for a few more iterations on the task specific dataset after pretraining on the task-unrelated dataset.

@Marcel_marcel1991 Well, during fine-tuning, you don't need to unfreeze all layers, but maybe you're right that some people may use fine-tuning to mean specifically when you unfreeze some of the layers. The TF tutorial says, for instance, "Also, you should try to fine-tune a small number of top layers rather than the whole MobileNet model." I will edit my answer to point that out.

@Marcel_marcel1991 If you look at the answer on Stats that the other answer links to, they are using the word fine-tuning to refer to something slightly different (i.e. learning new tasks or classes continuously by slightly changing the output layer), and it's in the context of continual learning, rather than transfer learning (as described in TF tutorial and in other papers). So, as I say now in my answer, it's possible people may be using these terms to refer to slightly different things.

I agree with you that there is no clear definition. I was only relating to that your answer assumes that you would always freeze the previous layers in fine-tuning, which is not the case. The edit now is fine to me.


text:
Why are the generator and discriminator designed differently in this example?
Why are the generator and discriminator designed differently in the example My First GAN of the coursera course: Build Basic Generative Adversarial Networks (GANs)?
Why didn't we use the same set of layers and activation functions for both generator and discriminator?
On what basis are the layer parameters and the activation functions decided in the generator and the discriminator?
The generator layer is as follows
> def get_generator_block(input_dim, output_dim):
>     '''
>     Function for returning a block of the generator's neural network
>     given input and output dimensions.
>     Parameters:
>         input_dim: the dimension of the input vector, a scalar
>         output_dim: the dimension of the output vector, a scalar
>     Returns:
>         a generator neural network layer, with a linear transformation
>           followed by a batch normalization and then a relu activation
>     '''
>     return nn.Sequential(
>         nn.Linear(input_dim, output_dim),
>         nn.BatchNorm1d(output_dim),
>         nn.ReLU(inplace=True)
>     )

The discriminator layer is as follows
def get_discriminator_block(input_dim, output_dim):
    '''
    Discriminator Block
    Function for returning a neural network of the discriminator given input and output dimensions.
    Parameters:
        input_dim: the dimension of the input vector, a scalar
        output_dim: the dimension of the output vector, a scalar
    Returns:
        a discriminator neural network layer, with a linear transformation
          followed by an nn.LeakyReLU activation with negative slope of 0.2
    return nn.Sequential(
        nn.Linear(input_dim,output_dim),
        nn.LeakyReLU(negative_slope = 0.2, inplace = True)
    )

In general both the generator and discriminator of GANs would consist of multiple convolutional or linear layers to capture complex patterns in the data. And the  specific code you provided seems to define basic building blocks for the generator and discriminator given input/output dimensions. The choice of the number of possible hidden layers and activation functions depends on factors such as the complexity of the data, the desired level of model capacity, and the trade-off between model performance and computational efficiency.
Regarding your question about the principle of different design choices for generator and discriminator, you need to start from their different functions. The generator typically starts with a low-dimensional noise vector and gradually transforms it into a high-dimensional output that should resemble real data. The architecture often involves deconvolutions (transposed convolutions) to upsample the data. Batch normalization (or better layer/instance normalization) and ReLU activations are commonly used to try to stablize and speedup training and introduce non-linearity. The final layer might use a different activation function such as tanh to match the desired or required output range.
The discriminator, on the other hand, takes an input (real or generated) and outputs a probability indicating whether the input is real or generated. It doesn't need to upsample or generate data thus it's comparatively much simpler to focus on classifying the input without any normalization optimization scheme requirement. Leaky ReLU activations are commonly used in the discriminator to introduce a small negative slope, helping prevent the vanishing gradient problem during training. The final layer might use a sigmoid activation to produce a probability score.


text:
If $p(s'|s,a) = 0$, would the reward the reward $r(s,a,s')$ be infinite?
In chapter 2 of Barto and Sutton's RL book, the four argument probability function $p: S \times R \times S \times A \to [0,1]$ is reduced to three arguments $p: S \times S \times A \to [0,1]$ as follows,
$$ P(s'|s,a) = \sum_{r\in R}p(s',r|s,a) $$
Therefore, the expected rewards for state-action pairs are defined as a two-argument function $r: S \times A \to \mathbb{R} $:
$$ r(s,a) = \sum_{r \in R} r \sum_{s'\in S} p(s',r|s,a) $$
Up to this I understood everything and have no problem. My problem arises when in the next line they defined the expected rewards for state-action-next state triples as $r: S \times A \times S \to \mathbb{R} $ where,
$$ r(s,a,s') = \sum_{r\in R} r \frac{p(s',r|s,a)}{p(s'|s,a)} $$
I do not know how to derive this and it seems very counter-intuitive for me. It is because what if the state $s'$ is unreachable? Meaning there are no arrows in the MDP from s to s' ($p(s′|s,a)=0$). Does it mean that the reward would be infinity? $r(s,a,s′)=\infty$? Or is it undefined for this? As in RL we always try to maximize rewards, taking an infinity reward will always makes it maximum (which makes no sense). It would be very nice if somebody can explain this equation with step-by-step derivation.

$$ r(s',s,a) = \sum_{r\in R}rp(r|s',s,a) $$. also assume that p(s'|s,a)>0. for conciseness some times notations are simplified. for instance as mentioned in the previous page of the book A is used instead of A(s).

You're right. I missed that by definition $p(s'|s,a)>0$.

Does this answer your question? Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true? It might not be an exact duplicate, but the questions are very similar and it might be a good idea to at least link them.

yes it does. Many thanks for referring to it!

Ok, so I was able to find the answer to this question by myself. So I'm sharing it with everyone. By the law of non-exclusive events, $P(A|B) = \frac{P(A,B)}{P(B)}$ where $P(B) > 0$. This can also be written as, $P(A|B,C,D) = \frac{P(A,B|C,D)}{P(B|C,D)}$ (just adding C,D as events that are already happened). In this case we have, $ p(r|s',s,a) = \frac{p(s',r|s,a)}{p(s'|s,a)}$ where by definition $p(s'|s,a) > 0$.

You've correctly figured out the most part of your confusion. The only thing I want to add is that by the finite MDP definition here it's certainly possible $p(s'|s,a)=0$ for some $s',s$. The reason you can avoid infinite reward lies in the implicit logic of conditional probability, you cannot arrive at a definite posterior updated $P(A|B,C,D)$ via your above equation when you already know $P(B|C,D)=0$. Since the event $B$ here is like a premise or assumption, and if it's already false (no chance to happen) then any derivation based on it can be true, that is, $P(A|B,C,D)$ cannot be definite which certainly is useless for any conceivable application and you have to exclude such priors in deriving expected reward above.


text:
Why should the agent bounce the ball back and forth on the same side of the screen in Atari Breakout?
The following is from page 17 of  "Michael Hu, “The Art of Reinforcement Learning: Fundamentals, Mathematics, and Implementations with Python”, Apress, 2023"
https://link.springer.com/book/10.1007/978-1-4842-9606-6

An example of good reward engineering is in the game of Atari Breakout, where the goal of the agent is to clear all the bricks at the top of the screen by bouncing a ball off a paddle. One way to design a reward function for this game is to give the agent a positive reward for each brick it clears and a negative reward for each time the ball passes the paddle and goes out of bounds. However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.

This part is not clear:

However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.

Why should the agent bounce the ball back and forth on the same side of the screen? Is there a reward in this case?

There might be some cases, say, once the agent clearing a brick obtaining a positive reward and then forever bouncing the ball on the same side of the screen would have a higher cumulative reward than trying to clear as many bricks as possible but then got many negative rewards which might be significant compared to positive reward. It's like a conservative game player, once winning a game initially then stops playing completely. Or it's like the case if getting a downvote of an answer is equal to getting an upvote in terms of scores, then many more people would be reluctant to answer any question.
Therefore appropriate reward engineering is crucial. In certain scenarios a poorly designed reward function might inadvertently incentivize the agent to adopt suboptimal strategies that maximize cumulative rewards without truly achieving the main objectives of the task.

Worth noting that the -ve reward for missing the ball with the paddle is far more immediate and obvious response from the game environment, when considering relationship between action and reward, than the +ve reward from hitting a brick which is significantly delayed from the action that caused the ball to go in the right direction


text:
Do I really need to do policy evaluation until convergence in policy iteration?
I am currently studying Sutton's book, and I learned that in policy iteration, policy evaluation is done until the value function converges, and then policy improvement is performed.
However, I have seen elsewhere that even without waiting for the value function to converge, and even performing policy evaluation only once, if policy iteration (evaluation and improvement) is repeated, the optimal policy will still be obtained.
I am confused by this. Even if I wrote a simple code for an experiment, I don't understand why policy evaluation needs to be done until convergence. Because I see that the results are the same whether policy evaluation is done 3 times, 2 times, or 1000 times before policy improvement when policy iteration is done repeatedly. Even repeating the process of performing policy evaluation only once and then improvement yields the same results. I don't know how to accept this.
Could you please help me?

I don't understand why policy evaluation needs to be done until convergence

It doesn't need to be, although the resulting algorithm if you cut short of convergence is not strictly policy iteration as it is usually defined. This freedom of choice (how accurately to measure values, when to make changes to improve policy) is the premise behind Generalized Policy Iteration (GPI), which is the supporting theory behind all value-based RL methods. In GPI, it is acknowledged that any combination of approximating current values and then improving policy based on the new approximate values is likely to converge to an optimal control solution eventually.
However, policy iteration with accurate valuation followed by policy improvement in separate stages is supported by a relatively simple and robust proof - the Policy Improvement Theorem, which is shown in the same chapter in Sutton&Barto.
Proofs of convergence for other methods that are based on GPI are harder (e.g. for Q-learning) or still open and unproven (e.g. for Monte Carlo control).
The book will get quite quickly to GPI from where you are in it. Keep reading!


text:
Equivalent of symbolic regression but for code instead of math expression
I'm already well versed with Genetic/Memetic algorithms and similar algorithms. I know about Symbolic regression, where some dataset is fitted through a math expression evolution, but I'm wondering, is there any name for equivalent algorithm but for Code instead of Math expression?
I looked around using keyword such as "Programming regression" or "Code regression" or similar but did not find anything.

Yes, it's called genetic programming.
I've done a bit of research on the topic in the past. If you use Python, you could use deap, which is quite easy to use. You may also be interested in the PushGP approach. If yes, you could try this library.

Yeah, I tried all of those already yesterday, but wanted to confirm if there was another name. Thank you.


text:
Why would balancing be so helpful when the imbalance is minimal?
I have a binary classification problem with a modest-to-none class imbalance (33% positive class-66% negative class). When I don't impose class balance, my XGBoost model produces no positive class predictions. When I downsample the majority class to balance the data, my performance gets really good (>0.9). Why would balancing be so helpful when the imbalance is minimal?

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

I would agree that tha observed results are unexpected. For such a radical change I would actually suspect a bug/omission in data processing for the imbalanced data set. It is not necessarily the rebalancing itself fixing the issue, but something done within the rebalancing processing that you should be doing anyway.
For example, not shuffling the data (which happens to be arranged [negative, postive]), using over 30% for cross-validation, and therefore being left with only negative labels in the training set.
Add some logs/tests giving dataset stats for your training data - only the training set - just before it is used. Libraries like pandas can do this relatively easily.


text:
Is there any advantage of genetic algorithm (or programming) over Neural Networks?
I am planning to switch from neural networks to genetic algorithms (GA) and programming (GP).
One of the main hassles of working with neural networks is that it requires a large amount of training data. Another issue with neural networks is that they require powerful computers for training, which I cannot afford.
Can genetic programming relieve me of these hassles?
I'm particularly interested in the protein structure prediction problem.

Your question is too broad. Please provide some clarity on the problem you want to solve. Explain your rationale for considering a GA or GP. Solutions based on GAs and GPs can be very time-consuming and may require a cluster of computers that are far more expensive than a retail GPU if you want to finish the problem quickly.

Could you give an example of a problem that you want to solve? Preferably one where you have rejected neural networks as too slow and data-hungry.

@NeilSlater, Protein structure prediction.

They are doing different things. NNs are usually used for classification tasks, whereas GAs are for optimising a configuration etc.

Did you research the use of GA or GP for protein structure prediction? If so, what did you learn? A good paper should compare its approach to some baseline or alternative approaches, which will likely include neural networks.

@BrianO'Donnell Just vote to close the post as "details details or clarity" or "needs more focus". You've been on this site for so long, you should already know this.

There are a few advantages of genetic algorithms (GAs) over neural networks using gradient descent for solving optimisation problems:

GAs don't require relationship between any objective function (fitness function) and learnable parameters to be differentiable.

GAs can alter hyperparameters of a model structure. This is related to the first point, but worth mentioning because it allows a GA like the NEAT algorithm to alter structure of neural networks on the fly.

GAs can be used to optimise models where the relationship between the models' outputs and end results are unclear, but those end results can be assessed. This is a property shared with reinforcement learning.

Unfortunately, this extra flexibility comes at a cost. Compared to gradient-based algorithms, GAs are often inefficient and slow. When a problem requires a model with a large number of free parameters, then GAs become impractical.
Regarding a requirement for lots of data and compute time;

Can genetic programming relieve me of these hassles?

Not usually. In some cases it might happen that a small non-differential model neatly encapsulates the problem and a GA or GP could find it. But for most hard problems where large neural networks have been applied, there are no known alternatives than gradient-based learning algorithms that require significant computation and lots of data.
You may still have alternatives for specific problems in practice. However researching a hard data-intense problem from scratch using a GA could well require orders of magnitude more compute resources than gradient-based methods.

I am particularly interested in the protein structure prediction problem.

There might be a role for GA or GP algorithms in searching for solutions to this problem. However, trying to frame the entire search as GA or GP is not practical. Instead a GA might be useful to direct model creation and testing.
There's no avoiding the significant difficulties and compute costs for research into the issue. At least not without a theoretical breakthrough in the core science of understanding how protein structures are created at the atomic level.


text:
What is the number of channels of input audio mel spectrogram?
What is the number of channels of input audio mel spectrogram? For example, in CV we always have 3 input channels on RGB picture. But what about audio?


text:
References for synthetic images generation from small datasets (~10-50 images)
I'm looking a references (papers / works) for synthetic image generation from small datasets. By small dataset, I mean 10-50 images.
I assume, that the best approaches should be based GAN (cGAN ?) or Diffusion Model.
Meanwhile I've found the following:
Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis
Image Generation From Small Datasets via Batch Statistics Adaptation

The references you stated indeed are the right way to go regarding small dataset image synthesis.
I'd research the space of few-shot image synthesis for what is out there, but something along the line of StyleGAN2 is a logical direction to look into.
In addition, I'd like to state specifically that training a synthesizer on only 10-50 images is always going to be very difficult. Is your goal going to be to make slight adaptations to the images? Or to synthesize completely new images? Image data is very high dimensional, and generalising high dimensional data from very few samples is near impossible, no matter what technique you employ.

I'm partially agree. It was correct few years ago, that it's very challenging to create synthetic data from 10-15 images. Recently there are works of creating an synthetic images even from single image. For example SinGan paper: https://tamarott.github.io/SinGAN.htm.

Did not know of that paper, nice addition!

It got best paper award at ICCV 2019.


text:
What's the difference between GPT3.5 and InstructGPT?
I read about the different model series in GPT3.5 here - https://platform.openai.com/docs/models/gpt-3-5
At the beginning of the page, it mentions to look at https://platform.openai.com/docs/model-index-for-researchers to understand the difference between model series InstructGPT and GPT3.5.
But, on that page, it says InstructGPT is a part of the GPT3.5 series. So, what's the difference between GPT3.5 and InstructGPT?

InstructGPTs are fine tuned GPT3.5 models. They have been tuned with different techniques to be good at following instructions and chatting. Examples of fine-tuning techniques are 1. Supervised fine-tuning on human demonstrations or 2. RLHF using PPO to maximise a reward model trained from comparisons by humans.

Hi! Thanks for your answer, this leaves me with the same question, though. If you see the first answer in this discussion: https://ai.stackexchange.com/questions/39023/are-gpt-3-5-series-models-based-on-gpt-3?newreg=d30affa2b8494f3faefd53868a57104e, the difference between InstructGPT and ChatGPT is confusing. Specifically, if the InstructGPT model does SFT, PPO, what does code-davinci-002 indicate?

See here https://help.openai.com/en/articles/6195637-getting-started-with-codex


text:
Is the re-parameterization trick necessary in the policy gradient method?
If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action.
Wouldn't this lead to the same issue that variational autoencoders face without the reparameterization trick, where the gradient cannot pass through sampling? Or is the reparameterization trick also used in the stochastic policy gradient method to address this issue?

For Policy-Gradient methods which require a differentiable sample from the action distribution, such as Soft Actor-Critic, then you are correct that it suffers from the same problem of requiring the gradient to be able to pass through these samples.
The common way around this is to consider distributions that can be sampled from using the re-parameterisation trick. The classic example of this is with a Gaussian distribution: suppose our network outputs mean and standard deviation $\mu_\theta$ and $\sigma_{\theta}$, respectively ($\theta$ are the network parameters), then we can note that $X = \mu_\theta + \sigma_\theta \times \epsilon$, where $\epsilon$ follows a Unit-Normal distribution. So, to sample from $X$, all we need to do is sample from the unit normal distribution, which does not require us to backprop through our network parameters.
The same can be achieved for discrete actions using the Gumbel-Softmax trick. For information on this, I would recommend this blog-post (and the references within).
For algorithms derived from the REINFORCE algorithm, where the policy gradient is given by $G_t \nabla_\theta \log \pi_\theta(a|s)$, then you can see that this is equivalent to maximising the (log) likelihood, scaled by the returns. All we need to do here is obtain the parameters of the distribution of our policy (e.g. mean and std-dev of a Gaussian) by doing a forward pass of the network for the current state, and then evaluating the PDF/PMF at the action we selected -- this does not require the sampled action to be differentiable.

I think I am still very confused about something: let's say for a non-actor-critic policy gradient method, we already have a stochastic policy. I don't understand how is the policy calculated w.r.t. the outcome, since the outcome is the realization of one possibility of the action from the distribution. In other words, from a particular outcome, the outside observer wouldn't be able to know this is a stochastic policy. How to compete the gradient w.r.t. this outcome then?

if the answer is monte carlo, then even of the stochastic policy is a discrete categorical distribution, it doesn't matter anymore isn't it?

It's not totally clear to me what you mean. If not using a PG method that requires sampling from the policy, the alternatives are value learning (we don't explicitly look at the policy here, just learn a Q-function for, usually, the greedy policy), or alternatively a PG algorithm that originates from the REINFORCE update. Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.

'Here, whilst we sample from the policy, we don't need to differentiate through the sampled action. We just maximise the likelihood (under the policy) of taking the action we saw, scaled by the returns.' but to update the model parameters which parameterize the policy, we still need gradient don't we?

Yes, but you don't need to differentiate through the sampled action. You just do a forward pass of your network to output the parameters of your distribution, and then evaluate the pdf/pmf at the action you chose and differentiate the output (remembering to also scale by the returns).

You can use the reparameterization trick only when the reward is differentiable with respect to the parameters.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
What is the purpose of fine-tuning a transformer?
What exactly is the purpose, since they can be prompted without fine tuning, although they are significantly worse than when fine-tuned?

Fine-tuning embeds task-specific knowledge directly into the LLM model weights, whereas prompting relies on the model's existing weights learned during its pretraining. A typical pretrained foundation LLM under causal or masked language modeling to learn contextualized embedding and prediction isn't specialized for any downstream NLP tasks. Cases such as high-stakes outcomes with specialized tasks (medical classification, Q&A, Chat), domain-specific tasks (legal contract generation, research paper summarization), or improved efficiency in specific workflows are better handled further by fine-tuning. If pretraining a foundation model is like teaching a student general knowledge by letting them read thousands of books and learn patterns in language, then fine-tuning is like training the same student specifically to pass an exam using past questions and answers, and possible continuous training of various specific tasks in the future.
A common example is the necessary multi-stage fine-tuning process of ChatGPT designed to align its foundation GPT model's capabilities with human preferences, safety standards, and Q&A task-specific conversational performance, in a continuous evolutionary fashion.

ChatGPT is built on OpenAI's proprietary series of generative pre-trained transformer (GPT) models and is fine-tuned for conversational applications using a combination of supervised learning and reinforcement learning from human feedback... Both approaches employed human trainers to improve model performance. In the case of supervised learning, the trainers played both sides: the user and the AI assistant. In the reinforcement learning stage, human trainers first ranked responses that the model had created in a previous conversation. These rankings were used to create "reward models" that were used to fine-tune the model further by using several iterations of proximal policy optimization... OpenAI collects data from ChatGPT users to train and fine-tune the service further. Users can upvote or downvote responses they receive from ChatGPT and fill in a text field with additional feedback.


text:
In Dyna Q, why is only one step learned during the first episode
In explanation of Example 8.1, Sutton and Barto's book says

Without planning $(n = 0)$, each episode adds only one additional step to the policy, and so only one step (the last) has been learned so far. With planning, again only one step is learned during the first episode, but here during the second episode an extensive policy has been developed that by the end of the episode will reach almost back to the start state.

My question is - why is only one step learned in the first episode with planning?
I agree with the fact that nothing is learned till the agent reaches the goal in both settings, but in the last "real step" when the agent reaches the goal, the planning steps if they choose the state and action adjacent to the recently updated square can learn a bit too.

You're correct that the planning agent could allow value function updates to adjacent steps even in the first episode. However, the state-action pairs immediately adjacent to the goal state (only state with positive reward) might receive some updates during planning, but earlier states lack sufficient path experience to propagate value reliably since in the first episode the agent doesn’t yet have any significant amount of experience of earlier state-action pairs leading to the goal.
In the second episode, however, the difference isn't in the real experience which is still bottlenecked at the final goal state with positive reward, but in the growing ability of the planning agent to chain simulated value updates backward using accumulated path experience. Each episode incrementally improves the agent’s model of state-action pairs leading to the goal state from its path experience, enabling deeper credit assignment backward through planning.

the statement in book which concretely says "only one step is learned" is incorrect right?

The book might have intended to simplify or emphasize the direct learning aspect from real grounded experience, but Dyna-style algorithms inherently allow multi-step planning, even after just one episode. So technically they're not wrong with the word "learned". A better phrasing might be "In the first episode, real experience updates the trajectory leading to the goal state in a model-free learning fashion, while planning enables further credit assignment across the trajectory and potentially beyond."


text:
When should I use a linear unit instead of sigmoid in the output layer?
In which types of learning tasks are linear units more useful than sigmoid activation functions in the output layer of a multi-layer neural network?


text:
Why use a neural network approach for linear regression if there is already an explicit solution
I  have noticed that many introductory materials on neural networks use linear regression to predict house prices as the standard first  example on neural networks. This seems to be a common practice.
But why use neural networks for a problem that already has an explicit solution, such as linear regression using the least squares criterion?
Linear regression has a well-defined formula for the best-fitting line (https://en.wikipedia.org/wiki/Simple_linear_regression#Example), so what is the benefit of applying neural networks in this case?

The explicit solution, the famous $\hat\beta_{OLS} = (X^TX)^{-1}X^Ty$ in statistics, is a piece of linear algebra that says what the best fit is, given a certain model. There is no commentary, however, about how good that fit is. The relationship between variables and the outcome could be curvy (nonlinear); there could be interactions between the variables, that the impact one variables has on the outcome depends on the values of other variables; the interactions could be curvy (interactions between nonlinear transformations of the variables). If you just stick a straight line (hyperplane for multiple linear regression) through the data, the fit might be quite poor and able to be improved upon.
There is an interesting paper out there that neural networks are just polynomial regressions, which is a particular form of linear regression that can be solved with the matrix solution, $\hat\beta_{OLS} = (X^TX)^{-1}X^Ty$. However, that requires explicit engineering of the polynomial features: which relationships are curved, which variables interact, and which interactions involve curvature. For better or for worse, neural networks do not require this. You just throw a bunch of neurons at the problem, with practitioners varying in how thoughtful they are about the exact architecture, and let the network figure out the relationship.
Being explicit in answering your question, neural networks offer a possible improvement in performance compared to more basic forms of linear regression (without interactions or nonlinear transformations of features), due to their high capacity for flexible modeling that the rigidity of hyperplanes do not allow. If you start allowing for complicated linear models involving interactions and curvature in the features, then linear regression could be more competitive, as Cheng et al (2018) argue.
REFERENCE
Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).

I know that when I teach students about fully connected layers, I always introduce the topic with linear regression.  There are a few reasons:

The general function applied in each dense neuron is $\sigma(wx+b)$. While the nonlinearity introduced by whichever activation is chosen for $\sigma$ makes this nonlinear, the heart of the learning is the parameters $w$ and $b$, or $\theta_0$ and $\theta_1$. This provides an excellent, simplified example to work through the gradient calculation using something like $MSE$.

The use of the linear regression also provides an excellent opportunity to reinforce the Matrix Composition Theorem. I.e., any series of linear or affine transformations must collapse to a single linear transformation matrix. This helps to develop the intuition for why a nonlinear activation is necessary.

With these main points introduced and understood, it's easier for learners to grasp the "why" of the design and the functioning of a dense layer without needing to rigorously work through an example by hand.

by your reasoning, there fact that a non linearity is there, is merely to avoid going back to N-successive-linear transformations. However, polynomial activations break the MC Theorem, yet MLP with polynomial activations are not universal approximators.

This is true.. but not the point of the post or the mention of activations. The question was asking why introduce linear regression to lead to neural networks, not why do we use the specific activations that we use.

It's just evidence that sometimes parallelisms are taken too far. NN and LR don't really share much...

The only grounded parallelism between NN and LR is interpreting the first N+1 layers of the NN as a learned preprocessing of the input.
The fact that LR has a closed form is more of a luck than a rule. You are interested in a statistic that is not the mean (such as the median)? Bye Bye closed form. Are you interested in constrained weight space? Bye Bye closed form.

I'm not sure I would call the fact that linear transformations are closed "Luck..." and it is in fact a rule or theorem that can and has been proved.

@DavidHoelzer "and it is in fact a rule or theorem that can and has been proved." what? a closed form of a model is now a theorem? no. it's more luck than skills. It all boils down to the fact that expectations are closed under linear trasformations. Logistic regression is arguably as simple as linear regression, and it does not have a closed form, that's why linear regression is more a luck than a rule that we have a closed form for it

@DavidHoelzer and i'm not saying we stubled upon it by luck, I'm saying that the fact that LinR admits a closed form, is arguably mere luck than a rule... hope I made it clear


text:
What is the number of all possible k clusters for N data points?
Assume that we have N data points and we divide into k clusters.
What is the number of all possible k clusters for N data points?

Is this a homework question? If yes, please, edit your post to add the tag [tag:homework]. For these questions, we shouldn't reveal the complete answer. That's why. Thanks for creating a post with a clear title, but please also include some context in your posts, like "what are your findings or ideas?"

No, it is not a HW question.

Is the value of $k$ fixed?

yes, it is  fixed.

Please, use also the [tag:math] tag then when your question is purely mathematical, and if it requires a proof, use the tag [tag:proofs]. If this is not a homework question, then where did you find it? Provide more context in your posts. If you just came up with this question, which I doubt, then you should say that. Also, edit your post to clarify that k is fixed because comments are temporary and can be deleted.

We denote the number as $S(N,k)$ and recall obvious special cases as follows.
$S(N,1)=1$: All points belong to one cluster.
$S(N,N)=1$: Each point is in its own cluster.
$S(N,k)=0$, if $k>N$: It is impossible to have more clusters than points.
For the more general case, refer Stirling number of the second kind.

In mathematics, particularly in combinatorics, a Stirling number of the second kind (or Stirling partition number) is the number of ways to partition a set of n objects into k non-empty subsets and is denoted by ${\displaystyle S(n,k)}$ or ${\displaystyle \textstyle \left\{{n \atop k}\right\}}$.

${\displaystyle \left\{{n \atop k}\right\}={\frac {1}{k!}}\sum _{i=0}^{k}(-1)^{k-i}{\binom {k}{i}}i^{n}=\sum _{i=0}^{k}{\frac {(-1)^{k-i}i^{n}}{(k-i)!i!}}.}$

Stirling numbers of the second kind obey the recurrence relation ${\displaystyle \left\{{n+1 \atop k}\right\}=k\left\{{n \atop k}\right\}+\left\{{n \atop k-1}\right\}\quad {\mbox{for}}\;0<k<n}$
with initial conditions ${\displaystyle \left\{{n \atop n}\right\}=1\quad {\mbox{ for}}\;n\geq 0\quad {\text{ and }}\quad \left\{{n \atop 0}\right\}=\left\{{0 \atop n}\right\}=0\quad {\text{ for }}n>0{\text{.}}}$

cinch handles the general case, but since you say that $k$ is fixed, there's a simpler approach:
Assuming each cluster must contain at least one element, we need to distribute the remaining $N-k$ elements. Each is assigned to one of $k$ possibilities, and since each such assignment is independent of the others, we can just multiply those together. So there are $k^\left(N-k\right)$ possible assignments.
(In addition to assuming fixed $k$, this also assumes that it matters which point is assigned to which cluster, not just what the cluster sizes are.)


text:
Why are PyTorch and TensorFlow the most widely used frameworks?
Even through there are several others, what's special about these two that make them so popular?

Questions about specific software are off-topic here, although we also have questions like "Why is Python popular in AI?", so we could also allow this question. Please, familiarise yourself with our on-topic page: https://ai.stackexchange.com/help/on-topic. But can you please provide more context? Have you ever used these frameworks, for example? What kind of answer are you looking for? We could simply say "Because they are the best". I don't think that type of answer would be extremely useful.

PyTorch dominates research due to its Pythonic flexible approach with dynamical computational graph support which is crucial for prototyping and quick development, while TensorFlow shines in production with its comprehensive ecosystem, scalability, both static and dynamic computational graph support, and multiple language support (Python, C++, Java, GO, C#, JS). Both as open-source frameworks, their adaptability, strong community, big corporation backing, support for GPU/TPU/distributed cluster, and powerful features make them the top choices for ML practitioners and researchers.
Other frameworks such as MXNet, Theano, and Caffe either face competition pressure from industry, lack community support, modern features, or the ease of use of PyTorch and TensorFlow.

Theano is an open source project primarily developed by the Montreal Institute for Learning Algorithms (MILA)... On 28 September 2017, Pascal Lamblin posted a message from Yoshua Bengio, Head of MILA: major development would cease after the 1.0 release due to competing offerings by strong industrial players.

A newer narrow-scoped framework called Google JAX is gaining traction for research but lacks the production tools and comprehensive features of TensorFlow and PyTorch.

Google JAX is a machine learning framework for transforming numerical functions. It is described as bringing together a modified version of autograd (automatic obtaining of the gradient function through differentiation of a function) and TensorFlow's XLA (Accelerated Linear Algebra). It is designed to follow the structure and workflow of NumPy as closely as possible and works with various existing frameworks such as TensorFlow and PyTorch.

I didn't check and only an extensive survey can provide good evidence for your claim, but, yes, it's most likely true that PyTorch and TensorFlow are among the most widely used libraries. There can be many reasons why they became successful, but one clear reason is that they are supported by big companies, Meta and Google.
However, PyTorch and TensorFlow are mostly deep learning libraries. There are other more general machine learning libraries that are also used a lot, like scikit-learn, pandas, or numpy, which is not strictly a machine learning library, but can be used for anything that involves matrix operations, etc. If you look at the 2024 Stack Overflow Developer Survey, you will see that numpy, pandas and scikit-learn are among the most widely used libraries and even come before PyTorch and TensorFlow, but we must also consider many people that answer to these surveys are software developers/engineers and not machine learning researchers, practitioners, etc.
In other AI areas, like evolutionary computation, other libraries are used more, like DEAP.
One thing we can say is that these libraries usually have a Python interface and are usually written in C or C++ under the hood to get performance.


text:
In some case, is training from random weight initialization better than fine-tuning?
I just trained my deep learning model with a transformer for a 4-classes multi-classification problem. However, some batches only have 3 classes, making the model biased for those 3 classes and making the model not confident to infer the other class.
I fine-tuned with retraining, but this time I removed that domain specific case that some batch only has 3 classes, leaving the training set and validation containing only 4 classes, and setting with higher learning rate such as 0.1 compared to the base model of 0.001.
But why is my fine-tuned model not good? Its accuracy is about 60%-80% compared to base-model that has 90% accuracy.

A learning rate of 0.1 is generally quite high for fine-tuning, especially if you're working with a pre-trained model. Fine-tuning a model usually requires a smaller learning rate (like 0.0001 to 0.001 for your case) because you don't want to disrupt the pre-trained features too quickly. You might also try using a learning rate scheduler to gradually reduce the learning rate during training along with regularization techniques.
Also fine-tuning a model works well when the pre-trained model's learned features are already close to being useful for your problem. If the pre-trained model was trained on a dataset that has significantly different class distributions or domains such as including quite a few 3-class batches in your case, a fine-tuning process might not work as expected when you switch to a more balanced or different dataset. If the fine-tuning dataset is too different from the original domain-specific data, the model may "forget" the domain-specific features it learned in the base training phase. This is known as catastrophic forgetting. You might need a full retrain with the class balanced training data.
Finally your base model might have overfitted to the specific dataset it was trained on including the domain-specific cases. When you remove that data, the model might no longer generalize well to the new data distribution. In general the base model should be a good pretrained general-purpose foundational model, and then fine tune the model on your domain-specific dataset.


text:
How can machine learning be used to create synthesized sounds of world instruments?
So far the usual way to recreate the sounds of these musical instruments is via physical modelling synthesis. But what about machine learning methods?

What are your findings so far?

DeepMind's WaveNet initially developed for speech synthesis based directly upon raw audio waveforms model has been adapted for generating realistic audio including musical instruments.

WaveNet's ability to generate raw waveforms means that it can model any kind of audio, including music... WaveNet is a type of feedforward neural network known as a deep convolutional neural network (CNN). In WaveNet, the CNN takes a raw signal as an input and synthesises an output one sample at a time... WaveNet greatly reduced the number of audio recordings that were required to create a voice model by modeling the raw audio of the voice actor samples.

You can further refer Engel et al (2017) "Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders" for WaveNet's application for musucal instrument sound synthesis specifically.

First, we detail a powerful new WaveNet-style autoencoder model that conditions an autoregressive decoder on temporal codes learned from the raw audio waveform. Second, we introduce NSynth, a large-scale and high-quality dataset of musical notes that is an order of magnitude larger than comparable public datasets. Using NSynth, we demonstrate improved qualitative and quantitative performance of the WaveNet autoencoder over a well-tuned spectral autoencoder baseline. Finally, we show that the model learns a manifold of embeddings that allows for morphing between instruments, meaningfully interpolating in timbre to create new types of sounds that are realistic and expressive.

There're also GANSynth (Google) which is an extension of NSynth and uses GANs to model audio data and generate instrument sounds with better temporal coherence. See Engel et al (2019) "GANSynth: Adversarial Neural Audio Synthesis".

Rather
than containing all types of audio, NSynth consists solely of individual notes from musical instruments across a range of pitches, timbres, and volumes. Similar to CelebA, all the data is aligned and cropped to reduce variance and focus on fine-scale details, which in audio corresponds to timbre and fidelity. Further, each note is also accompanied by an array of attribute labels to enable exploring conditional generation.

Finally OpenAI's Jukebox can also generate instrument sounds via VQ-VAE-2.

Hierarchical VQ-VAEs can generate short instrumental pieces from a few sets of instruments, however they suffer from hierarchy collapse due to use of successive encoders coupled with autoregressive decoders. A simplified variant called VQ-VAE-2 avoids these issues by using feedforward encoders and decoders only, and they show impressive results at generating high-fidelity images... Our first raw audio model, which learns to recreate instruments like Piano and Violin. We try a dataset of rock and pop songs, and surprisingly it works.


text:
What is not an applicaton of AI?
With the AI era, everyone says everything is AI, which doesn't seem to be the case. What is advertised as AI and isn't?
For example, (1) Google's PageRank algorithm is called AI, is it really AI? (2) Is a precise spinal column operation by a robot AI?
In summary, I'm looking for areas that claim to be using AI but, ultimately, aren't actually AI and it just marketing.

You may wish to look into the difference between AI and machine learning, which is a subfield within AI. AI is very broad.

@David Hoelzer, do you have any references where I can learn more about the broad scope of AI?

This question is similar to: What are the minimum requirements to call something AI?. If you believe it’s different, please [edit] the question, make it clear how it’s different and/or how the answers on that question are not helpful for your problem.

This question is basically asking what AI is, so I think it's a duplicate of the linked question or this: https://ai.stackexchange.com/q/7838/2444. I think both should be in the duplicate questions, although they are slightly different. Or even this https://ai.stackexchange.com/q/7624/2444.

Anyway, I think your question is actually a bit different than the others, but still an answer requires people to define AI.

It's difficult to answer your question because different people use or assume different definitions of AI. Nowadays, many people think that AI is just language models (e.g. tools like ChatGPT and Gemini), but it's not. AI is also not just machine learning.
So, to answer your question, we first need to define AI. We already have several questions that attempt to address this, so I'll not repeat the info there.

What are the minimum requirements to call something AI?
What is artificial intelligence?
What is the most general definition of "intelligence"?

I gave an answer to each of the questions above and clearly I recommend you read my answers.
Anyway, off the top of my head, I can't think of an area where people say they are using AI but they are not.
PageRank is an example of an algorithm that some people will categorize as AI. I cannot say that this is right or wrong. It really depends on your definition. It's a clever algorithm, but so is binary search or quick sort, and not many people call them AI. We could say that every clever algorithm is AI, but then you need to define clever.
I think it's important to keep in mind that AI is more about agents, goals and environments, not just algorithms. An agent, like us humans, will ultimately use algorithms to achieve goals in an environment.


text:
Can the limiting distribution depend on the initial distribution?
I am a bit confused about the definition of limiting distribution in Markov chains.
My understanding is that it represents the behavior of the chain in-the-limit. That is, I start from the initial distribution $\mu_0(s)$ and repeat $\mu_1(s) = \mu_0(s) P$, $\mu_2(s) = \mu_1(s)P, \ldots$ and at some point the distribution will not change anymore. This seems also what this question/answer says.
However, all definitions I find say that the limiting distribution does not depend on the initial distribution. But I can easily make a non-irreducible chain with two communicating classes like the one below, where the limit will converge but depends on the initial distribution.

So, can the limiting distribution depend on the initial distribution? If not, how can I find it?
This answer shows a few methods and one is what I am already doing below, i.e., a simple loop to simulate the chain. But, if I run it I converge to different distributions depending on the initial one.
import numpy as np
np.set_printoptions(precision=3, suppress=True)

P = np.array([
    [0.0, 0.5, 0.0, 0.5, 0.0],
    [0.0, 0.0, 1.0, 0.0, 0.0],
    [0.0, 0.5, 0.5, 0.0, 0.0],
    [0.0, 0.0, 0.0, 0.5, 0.5],
    [0.0, 0.0, 0.0, 1.0, 0.0],
])

MU = [
    np.array([0.3, 0.1, 0.2, 0.4, 0.0]),
    np.array([1.0, 0., 0., 0., 0.0]),
    np.random.rand(5),
]

for mu_t in MU:
    mu_t /= mu_t.sum()
    for _ in range(1000):
        mu_t = np.dot(mu_t, P)
    print(mu_t)
    print(np.dot(mu_t, P))
    print()

I know that an ergodic chain has only one unique limiting distribution and that my example is not ergodic, so I am considering the general case.

all definitions I find say that the limiting distribution does not depend on the initial distribution

The proof of this trait of the limit relies on the asumption of ergodicity.
You can define an MDP where there are non-interacting splits in trajectory (i.e. a branch in trajectory with no way of reaching outside of a subset of states), with two or more recurrent loops, plus an initial distribution where it is possible to reach at least two split points. This is essentially multiple ergodic MDPs in a branching structure. In that case, within each split, the limiting distribution is independent of the starting distribution.
In your example case, everything ends up in B & C or D & E. The ratio of times spent between BC and DE depend on the starting distribution. Within those groups, the ratios of time B:C and D:E are fixed and converge independently of any starting distribution.

Can a limiting distribution independent of the initial state exist in a non-ergodic chain? Because I usually read "a limting distribution is the limit of the distribution, independent of the initial state". And then "if the chain is ergodic, the limiting distribution exists and it's unique". The definition of limiting distribution seems unrelated to the one of ergodicity, and one that could exist in non-ergodic chains. But you said it depends on ergodicity, which confuses me (makes sense in terms of independency of the initial state, just doesn't seem to fit the literature? Any ref?).

@Simon as you said, ergodicity gives you guarantees that it exists and that is unique… in your example, it exists but it’s not unique, and furthermore depends on the initial state

@Alberto Right, so the limiting distribution, by definition, can be not unique, right? Because I often read "if it exists, it is unique" in its definition. That's why I was confused.

@Simon the fact that the limiting distribution depends on the initial state implies that the couple MC+initial state has a limiting distribution, though you probably cannot prove that it is always the case... generally speaking however, a limiting distribution exists if and only if it does not depend on the initial state (by definition)

Indeed the limiting behavior of a non-ergodic Markov chain which is not irreducible with multiple mean positive recurrent communicating classes depends on its initial distribution, since apparently the state space can be decomposed into multiple communicating classes thus the limiting distribution if exists depends on which class is activated by the initial distribution and typically there's one limiting distribution per recurrent communicating class if all states within the said communicating class are aperiodic.
In summary without irreducibility the limiting distribution if exists is not unique, and without aperiodicity a mere stationary distribution may not be converged as a limiting distribution as illustrated in the answer to your recent post here. There's conceptual difference between a limiting and stationary distribution of a MC.

Thanks! Your reply helped understanding Neil's better. I think both answers are useful, I accepted Neil's just because he replied first.


text:
Why are these 2 terms canceled out in the ELBO derivation for diffusion models?
I was looking at the derivation of the ELBO for diffusion models from this website and I'm a little bit confused as to why they cancelled these two terms out.

Where you see this derivation in your linked website? which equation number?

37 in the show me the proof part

Indeed this sounds very confusing until you expand the product $\Pi$ terms as outlined below:
$$\frac{p_{\theta}(x_1|x_2)q(x_1|x_0)p_{\theta}(x_2|x_3)q(x_2|x_0)...p_{\theta}(x_{T-1}|x_T)q(x_{T-1}|x_0)}{q(x_1|x_2,x_0)q(x_2|x_0)...q(x_{T-2}|x_{T-1},x_0)q(x_{T-1}|x_0)q(x_{T-1}|x_T,x_0)q(x_T|x_0)}$$
Therefore now it's obvious that the cancelation is from all alternating $q$ terms with only one conditioned variable $x_0$  in the numerator and denominator, respectively, except the first and last terms $\frac{q(x_1|x_0)}{q(x_T|x_0)}$ still remain. That's why in your reference the next step in show me proof section there appears one additional middle term $+\log \frac{q(x_1|x_0)}{q(x_T|x_0)}$. The author perhaps should expand like here to help reader's clear comprehension considering he's already written so many detailed steps.


text:
Should I use the same learning rate when I compare the performance for different language models?
If I'm using the same datasets  and using different language models, such as BERT, DistilBERT, Electra, etc., and my goal to compare the performance, such as the accuracy, macro f score, etc. should I used the same learning rate and number of epochs to make comparison fair, or is it allowed to have different learning rates for each? For example, if BERT provides the best result with a learning rate 2e-5 and Electra provides the best results with 1e-6, can I register both results and use them?

Since learning rate is a hyperparameter, using different learning rates or epochs for each model is not only allowed but generally encouraged as long as the hyperparameter tuning process such as grid search is consistent across all models. Different models have unique architectures, parameter counts, and training dynamics, which can affect how they respond to specific learning rates. Comparing models using the same fixed learning rate may disadvantage certain models, as they might not reach their optimal performance under those conditions.
Finally it's important to keep other factors constant such as the training dataset, evaluation metrics, data splits, preprocessing steps, etc. Of course, if resources are permitted, you can also run all models with the same learning rate and training setup to explore how they perform under identical conditions which could potentially provide additional insights.

Thank you for your answer, I appreciate your help. I found some research paper who use fixed learning rate to compare between models for that i'm wondering why they do that or is it  a must to compare in a fair situation?, also do you think is it better to use learning rate schedualer in this situation?

@Balira They must have some other specific reason which you may not notice or they may not disclose. Most optimization of ML models cannot control (global) learning rate, for example, Adam as the most populate optimizer of ANNs could be said to be a parameter-specific implicit learning rate scheduler which is not controlled by any external global learning rate scheduler.

Thank you cinch, you really provide me with important points


text:
Why are ergodic MDPs also communicating?

An MDP is ergodic if the Markov chain induced by any policy is
ergodic, which means any state is reachable from any other state by
following a suitable policy. [Source]

The part after "which means ..." is the definition of communicating MDP. Why does "the Markov chain induced by any policy is ergodic" implies "any state is reachable from any other state by following a suitable policy"?
Other papers/books define an ergodic MDP just with "if the Markov chain induced by any policy is ergodic", but they all say that an ergodic MDP is also communicating. Why ergodic → communicating?

Why does it hold? what? nothing has to hold, they are defining what is an ergodic MDP

Why does "the Markov chain induced by any policy is ergodic" implies "any state is reachable from any other state by following a suitable policy"?  @Alberto

it's not an implication, it is a definition "MDP is ergodic $\rightarrow$ any state is reachable from any other state by following a suitable policy"

@Alberto Then what is the difference with communicating MDP? "Any state is reachable from any other state by following a suitable policy" is the definition of communicating MDP. Also, Puterman's book defines ergodic MDP just with "if the Markov chain induced by any policy is ergodic", then saying that communicating MDPs are also ergodic (thus, the implication I don't understand).

i think that the communicating case is "there exists a policy that can move from state $i$ to state $j$ in a finite set of steps", while for ergodic MDP "state $i$ can be reached by state $j$ under any stationary policy", so it's a much stronger assumption

@Alberto I think "any" is not right. Ergodic means 1) that there exists A policy that "connects" every two states (communicating) + 2) every policy induces an ergodic Markov chain. But not that EVERY policy does it (unichain MDPs do that for EVERY policy). Anyway, my question is that all definitions seem to imply that 1) is implied from 2) and it's not part of the definition.

If all states in an irreducible Markov chain (MC) are ergodic, then the chain is said to be ergodic. So once an ergodic MC is induced by a policy, then by above definition all states are irreducible. And irreducibility and communicating here are defined as:

A Markov chain is irreducible if there is one communicating class, the state space.

Two states are said to communicate with each other if both are reachable from one another by a sequence of transitions that have positive probability. This is an equivalence relation which yields a set of communicating classes.

Since ergodicity implies irreducibility which means the existence of one communicating class as the whole state space, therefore any state is reachable from any other state within the said state space by following the said induced policy.

Are communicating and irreducible synonym then?

@Simon not exactly. Communicating only refers to two states which is a more basic concept, while irreducibility is for the whole chain when there's only one communicating class for all states as emphasized in my above answer. And the concept of communicating class is resulted from an equivalence relation of communicating states in the sense of set theory which is also mentioned in my answer. So if there's only one communicating class for all the states of a chain, then it implies all states are mutually communicating with each other. Hope this clarifies.

There are inconsistencies in the definition of an ergodic Markov Chain while de concepts of irreducibility, aperiodicity and recurrence are well grounded.
Thus, I think it is better not to employ this term too much and keep the word "ergodic" for the well-known theorem.
According to the paper you cite,

An MDP is ergodic if the
Markov chain induced by any policy π is both irreducible
and aperiodic, which means any state is reachable from
any other state by following a suitable policy.

So what can we say about that?
Let $\mathcal{X}$ be the state space.
If the chain is irreducible, then whether all states are recurrent, or all states are transient.
But if $dim(\mathcal{X}) < \infty$, then the second case is impossible.
Thus if the chain is irreducible and $dim(\mathcal{X}) < \infty$, then the chain is recurrent.
Now, If the chain is irreducible and recurrent, there exists a unique (up to a normalisation factor) stationary measure $\mu$.
Also, if the chain is recurrent, irreducible and aperiodic, then there is a $n_0$ such that for any $n>n_0$, $P_n(x, y) > 0$ for any states $x, y \in \mathcal{X}$, where $P_n$ is the $n$ steps transition probability kernel. This means any state $x \in \mathcal{X}$ is reachable from any other state $y \in \mathcal{X}$.
Sources: [1]
[1]: J-F. Le Gall - Intégration, Probabilités et Processus Aléatoires (2006)


text:
Is there any correspondence between physical objects' shape and activity occurring in a neural network?
Many years ago I saw an experiment which involved a lab primate whose brain was monitored as it watched a simple rectangular object as it moved. And sure enough, there were regions of cells in its brain that seemed, IIRC, to sequentially fire in some manner that corresponded to the rectangle. I do not recall if it was merely speed or if there was actually a rectangular region that "moved" (that is, different cells would fire in adjacent regions).
So my question is whether or not this sort of thing might be observed in some kinds of artificial minds. If, for example, you asked it to display an object as it rotated would there be some set of artificial neurons that would be allocated to represent the object and rotated in the same sense as described above, that is, artificial neurons would gradually "fire" in adjacent regions during the rotation?

Since ANNs are inspired from various types of biological neural coding schemes, it won't be too surprised to see the similar activation saliency map between them. One reference is Li et al's (2022) Artificial Visual System for Orientation Detection Based on Hubel–Wiesel Model.

The Hubel–Wiesel (HW) model is a classical neurobiological model for explaining the orientation selectivity of cortical cells... To realize a straightforward and efficient quantitive method and validate the HW model’s reasonability and practicality, we use McCulloch-Pitts (MP) neuron model to simulate simple cells and complex cells and implement an artificial visual system (AVS) for two-dimensional object orientation detection... Computer simulations show that the mechanism-based AVS can make accurate orientation discrimination and shows striking biological similarities with the natural visual system, which indirectly proves the rationality of the Hubel–Wiesel model.

Another example could be Hinton's CapsNet.

A capsule is a set of neurons that individually activate for various properties of a type of object, such as position, size and hue. Formally, a capsule is a set of neurons that collectively produce an activity vector with one element for each neuron to hold that neuron's instantiation value (e.g., hue)... For low level capsules, location information is "place-coded" according to which capsule is active. Higher up, more and more of the positional information is rate-coded in the capsule's output vector... Capsnets use neural activities that vary with viewpoint.

Thanks. My guess is that the sort of neural nets you mention are not the typical kind? I mean, the sort of "linear" multilayer networks? I will look at the links, thanks.

@Jeff Your question doesn't restrict to MLPs, further since modern MLPs are fully connected with continuous output (unlike the original McCulloch-Pitts neurons) which cannot be similar to the special case you described in OP as that's clearly some kind of efficient sparse neural coding. So you probably won't see the match you desired in MLPs, but more rare neural networks such as Hinton's CapsNet or even the good old binary bipartite Ristricted Boltzmann Machines (RBMs). Hope this clarifies.

it does. very interesting stuff.


text:
Are pure math topics like functional analysis and differential geometry useful for machine learning and AI?
I'm a math major with a burning passion for pure mathematics (to the point of obsession). However as I graduate I wanted to work in an industry that best utilise my math skills which brings me to my question.
Does working in machine learning, deep learning and AI requires advanced math skills such as differential geometry and functional analysis? How does knowing such math topics helps in the work of a machine learning and AI practioner?

Apart from common math, any advanced math would be potentially helpful for an AI practitioner, especially for research positions related to deep learning which still has quite a few unproven theoretical conjectures albeit many seemingly convincing empirical results have been accumulated over the years. For example there's a famous manifold hypothesis and information geometry of statistical manifolds which is related to differential geometry, and today's learned representations by deep learning are often assumed to be on a manifold, but the precise geometry and topology of these learned manifolds are not fully understood. Questions such as under which conditions and for which types of data this hypothesis holds and how manifold curvature/connectivity/other geometric properties affect learning, remain unresolved:

Machine learning models only have to fit relatively simple, low-dimensional, highly structured subspaces within their potential input space... The argument for reasoning about the information geometry on the latent space of distributions rests upon the existence and uniqueness of the Fisher information metric. In this general setting, we are trying to find a stochastic embedding of a statistical manifold.

In information geometry, the Fisher information metric is a particular Riemannian metric which can be defined on a smooth statistical manifold, i.e., a smooth manifold whose points are probability measures defined on a common probability space.

We find that the resultant information metric does not describe the full geometry of space but only its conformal geometry

Manifolds such as those found in graph-based or geometric data (e.g., 3D point clouds) are not easy to represent with conventional neural network architectures, which are designed for Euclidean data. The development of deep learning techniques that can model data on these non-Euclidean manifolds, such as geometric deep learning, is still in its early stages, and many questions remain about how best to handle such data.
Functional analysis is generally not a core skill for most machine learning roles outside of highly theoretical positions. For example if you want to fully understand and apply universal approximation theorem or various optimization methods you may need a good knowledge of functional analysis and approximation theory. A recent interesting post in this site requires this kind of knowledge. Since you described yourself as a math major with a burning passion for pure mathematics (to the point of obsession), maybe you can try to write a non-trivial answer for that post.


text:
Are there any attempts to make translation models support ancient/extinct languages?
Are there any attempts to make translation models support ancient/extinct languages like Sumerian, Ancient Greek, Old Norse, Cornish, etc.?

Please, next time, share with us your findings so far. I'd encourage users to do a quick google search before asking this type of question.

I am not aware of any attempts at building machine translation for extinct languages specifically, but I would see this as a sub-field of machine translation for low resource languages. The techniques applied to low resource languages should also be applicable to extinct languages, as those are, naturally, low resource languages.
For an overview over machine translation for low resource languages, see Haddow et al., 2022: Survey of Low-Resource Machine Translation. This could be a starting point for you research.


text:
How would you go from 1 to k hidden layers in Cybenko's result that neural networks are universal approximators?
Cybenko showed that if $\sigma$ is a sigmoidal, continuous function, then for any $\varepsilon > 0$, for any continuous function $f: [0, 1]^d \to \mathbb{R}$, there exists a function of the form $g:x \mapsto \sum\limits_{i = 1}^n a_i\sigma\left( \langle w_i, x \rangle + b_i \right) + b$ such that $\forall x \in [0, 1]^d, \left| f(x) - g(x) \right| \leq \varepsilon$.
So each continuous function on a compact set can be approximated by neural networks with $1$ hidden layer, with activation function $\sigma$ for this layer and identity for the output layer.
My question is: how can we deduce from this result that for any fixed $k \geq 1$, any continuous function on $[0, 1]^d$ can be approximated by neural networks with $k$ hidden layers, using $\sigma$ (and identity at the end) ?
Basically, how to go from one hidden layer to $k$

Welcome to this community! Nice question.

Just set all the next layers to map the identity.

Either construct them  saturating the weights, or consider that each layer is a 1-hidden-layer MLP which is a universal approximator, so it can learn the identity

To extend Cybenko's famous result from a single hidden layer to multiple hidden layers, you essentially need to show that any continuous function on a compact support is approximately decomposable into "smaller" continuous functions (not trivially the identity function) on a compact support recursively up to any fixed $k$ as you specified, each of which is already proved to be approximated universally with a single hidden layer.
The Stone–Weierstrass theorem can be helpful here.

In mathematical analysis, the Weierstrass approximation theorem states that every continuous function defined on a closed interval [a, b] can be uniformly approximated as closely as desired by a polynomial function... The Stone–Weierstrass theorem generalizes the Weierstrass approximation theorem in two directions: instead of the real interval [a, b], an arbitrary compact Hausdorff space X is considered, and instead of the algebra of polynomial functions, a variety of other families of continuous functions on ${\displaystyle X}$ are shown to suffice

Since continuous functions with a compact support can be universally approximated by the algebra of polynomials or similar families which are Lipschitz continuous primitive recursive functions, they could be further approximately represented by composition of primitive recursive functions. But this may not be always possible and there might be some limit for $k$ as such recursive non-trivial decomposition may not be always able to continue indefinitely, and bear in mind that many practical advantages of MLP over a fat single hidden layer are empirical findings without rigorous proof so far from deep learning of feature-rich highly nonlinear functions which are intuitively approximately decomposable into many layers.