text:
How to classify positional time series data?
I am working with a data set that includes (x, y, z) coordinates and timestamps of human movement. The input data looks something like:
[{
    "movement_type": 1,
    "path": [
        {
            "x":  1.5959584221177756,
            "y": -0.02057698369026184,
            "z": 1.1674611568450928
        },
                {
            "x":  1.5959584221177756,
            "y": -0.01429012417793274,
            "z": 1.1671339273452759,
        },
        ...
    ],
    "timestamps": [
        1666898619.132143,
        1666898619.135477
    ]
},{
    "movement_type": 2,
    "path": [
        {
            "x":  1.5959584221177756,
            "y": -0.02057698369026184,
            "z": 1.1674611568450928
        },
                {
            "x":  1.5959584221177756,
            "y": -0.01429012417793274,
            "z": 1.1671339273452759,
        },
        ...
    ],
        "timestamps": [
         1666898619.363774,
         1666898619.370507
    ]
}]

The ultimate task is: given a stream of x/y/z and timestamps can we classify a subset of the stream as one of the movement types in the training data?
There is no guarantee that the movements are done at the same velocity or in the same space. Because of this, I imagine I will need to process the data to focus on more on the difference between successive points, rather than absolute positions.
I'm specifically looking for guidance on how to start thinking about this problem. Is this a time series classification problem? Any suggestions on how to encode the data or features to extract? I'm sure there is some precedent but I am having trouble finding a starting place. Thank you in advance.

A good question. In short, yes, it is a time series problem. One simple way you can start thinking of it is translation.

Given one dimension and time, how would you recognize patterns? You will have some benchmarks and if any of those are met you will say okay because the position moved from [(1m, 1sec), (2m, 2sec), (1m, 3sec)] it is a certain kind of known pattern.

Given two dimensions and time, how will you recognize patterns now? Exactly the same way, but now with (x, y, t).

So, for n dimensions to be analyzed, we have n+1 variables. While it might not be absolutely necessary in this case to need a continuous representation of your dataset, you can consider these independent points in time as one continuous dataset in the 3+1th hyper dimension that in this case will be representing time.
Since you are the owner of the dataset (or at least the question) I believe you know all this already. Now, you can basically create (n-1) vectors that represent the translation of the point b/w ith and i+1th data-point. You can now run the dataset on a seq-to-seq model and call it a day!
But wait! There exists a simpler way! What is it? It's a multi-class binary classifier. You just need to think about converting each chunk of the dataset as one data point, and for that, you need to find a representation of that chunk (assuming each of these chunks is labeled). One way would be to extract features from each chunk of the dataset. You might be looking at the vector sum of 2-d projections, distance traveled, time taken, etc. Once you have a rich feature set you can perform the regular Data Engineering steps before training the classification model.
Hope this helps.

Thank you so much for the response. I'll be exploring your suggestions and may reach out with further questions :)


text:
Comparison of the two alternative forms for the KL divergence
On page 468 of 'Pattern Recognition and Machine Learning', what does 'the same variables given by the product of two independent univariate Gaussian distributions' mean?
The PDF says,

The green contours corresponding to 1, 2, and 3 standard deviations
for a correlated Gaussian distribution p(z) over two variables z1 and
z2, and the red contours represent the corresponding levels for an
approximating distribution q(z) over the same variables given by the
product of two independent univariate Gaussian distributions whose
parameters are obtained by minimization of (a) the KullbackLeibler
divergence KL(q||p), and (b) the reverse Kullback-Leibler divergence
KL(p||q).

but I don't understand the explanation clearly.
Which is the same variables?
Which is the two distributions?
What is the product? Does it mean the probability density function?
Thanks.

Can you please ask a specific question? "I don't understand the explanation clearly." is not a question and does make clear what you don't understand. Please, once you've decide what your specific question is, put it also in the title. Thanks.

I added specific questions  such as 'same variables' and 'two distributions'. Thanks.

The KL divergence is just not symmetric, and so changing $q$ for $p$, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution.

In the first plot, you see the forward KL, $KL(q\ \|\ p)$, that encourages the approximating distribution $q(z)$ (the one you topically learn) to cover the mode of $p(z)$: this is called mode-seeking behavior, which tries to match the most-likely point of the distribution $p$.
The other plot shows, instead, the reverse KL, $KL(p\ \|\ q)$, that tries to match the support of $p(z)$, however loosing the mode in the approximating distribution.

To recap, the forward KL tries to put probability mass in the area of the distribution where the probability of samples is higher (in this example, the area where the std is 1), while the reverse KL tries just to cover all the distribution's support placing some non-zero probability everywhere: that's why the second, bigger, red circle covers the most of the green contours.
Update:

Which is the same variables?

The (random) variables $z_1$ and $z_2$ define the "space" of values that you can observe, e.g. you're given a dataset of samples taken from $p(z)$, i.e. the green distribution, where $z \in \mathbb [0,1]^2$. Basically, the approximating distribution, $q(z)$, is defined on the same space where the variables $z_1$ and $z_2$ lie.

Which is the two distributions?

I think Bishop assumes $q(z)$ to be defined as a mean field approximation, which should correspond to $q(z) = \prod_{i\in\{1,2\}} q(z_i; \mu_i, \sigma)$ where $\mu_i$ and $\sigma$ are the parameters you want to learn to approximate $p(z)$. Basically, you consider two tractable distributions, e.g. a Gaussian, and learn the parameters of each "simple" distribution. Then, to get the final, more complex, distribution you perform the mean field assumption which is about doing a product among each $q_i(z_i)$. Also I put $\sigma$ and not $\sigma_i$ since from the figures the variance seems to be independent from $z_i$. For example, the mean field assumption does not enable you to learn the correlation between each $z_i$, so this may explain the same variance across dimensions.

What is the product? Does it mean the probability density function?

I think a partial answer is above. Anyway, is a product over the pdf of each $p_i$ and $q_i$ to yield, respectively, $p(z)$ and $q(z)$. But in the case of $p(z)$ you also account for the
correlations when doing the product, instead in $q(z)$ independence is assumed.
In the end, you can define $p(z) = \mathcal N(\mu,\Sigma)$ where $\mu=[0.5,0.5]$ and $\Sigma=\begin{bmatrix} \sigma_{11} & \sigma_{12} \\ \sigma_{21} & \sigma_{22} \end{bmatrix}$, where each $\sigma_{ij}$ (for $i=1,2$ and $j=1,2$) tells you the correlation between variable $z_i$ and $z_j$ that has the effect of shrinking or expanding and rotating (more importantly) the green Gaussian. Instead, $q(z)=\prod_i \mathcal N(\hat\mu_i,\hat\sigma_i)$, since is an approximation, has no such capabilities (e.g., rotation) and so you get a perfect circle, smaller or larger according to the KL you minimize to learn the parameters: $\hat\mu_1 = \hat\mu_2 = 0.5$ and $\hat\sigma_1 = \hat\sigma_2 = \hat \sigma$ or $\hat\Sigma=\begin{bmatrix} \hat\sigma & 0 \\ 0 & \hat\sigma \end{bmatrix}$.

@diffusionstable I'm glad the answer helped you. I've edited it trying to answer the other questions too. Have a look and let me know!

Thank you. It's a bit complicated, so I'll try to read it slowly!

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Sum of gaussian distribution
I assume the sum of the Gaussian distribution is 1. However, the actual calculation in JavaScript yields a value greater than 1 in JavaScript.
Is this code wrong?
gaussian_distribution.html]
<script>
m=1; // mean
s=1; // std
sum = 0;

// Gaussian distribution
function f(x){
    return (1/(Math.sqrt(2*Math.PI * (s**2)))) * Math.exp(((x-m)**2)/(s**2))
}

// Run
for (x=0; x<3; x+=0.01){
    sum = f(x);
    //console.log(sum)
}

console.log(sum)
</script>

Yes, that code is wrong. There are 3 issues. When you say sum, you really mean integral, which is the area under the curve of the normal distribution, and is indeed 1. Correct code, which outputs 1 (almost) is below:
m=1; // mean
s=1; // std
sum = 0;

// Gaussian distribution
function f(x){
    return (1/(Math.sqrt(2*Math.PI * (s**2)))) * Math.exp(-(1/2)*((x-m)**2)/(s**2))
}

// Run
for (x=-10; x<10; x+=0.01){
    sum += f(x)*0.01;
    //console.log(sum)
}

console.log(sum)

The changes:

The code is performing numerical integration, which graphically is represented by adding a number of small rectangles under the curve. The area of these rectangles is equal to length*height. In this case, height is the function value at a specific x-value, and width is the step in x-values we take. When we add up all of the rectangles across every x-value in a certain range (from here -10 to 10), we get an approximation to the integral between those bounds. I have altered the line sum = f(x) to read sum += f(x)*0.01, to represent both the fact that we are adding up each rectangle (= to +=), and that we care about the area (adding *0.01).
Your gaussian distribution function is incorrect. The gaussian or normal distribution has a factor of -1/2 in the exponent, which I have added.
Your bounds were wrong. Strictly speaking, 1 is the area under the gaussian curve from negative infinity to infinity, across all x values. However, by looking at a graph of the gaussian (here), you can see that the area is negligible at most x-values, except from roughly -2 to 4. By choosing bounds of 0 to 3, you cut off a large section of the area. My bounds of -10 to 10 were arbitrarily chosen, and really it does not matter as long as the bounds are at least -2 to 4, which contains 99.7% of the area.

My code outputs 1 (almost) as expected.
2 small things:

You can have a normal distribution with any mean and standard deviation you want, but the standard way is mean=0 and standard deviation=1, as this results in a curve nicely centered. You can see this by adjusting mean and standard deviation values here.

My code outputs not 1, but 1.000000000000003. This small error has 3 factors contributing to it:

Floating point arithmetic, which this program uses, is inexact, and subject to rounding errors

Numerical integration is also not exact, and only an approximation of the true integral

We have not actually integrated across all x-values, merely from -10 to 10, which would also introduce negligible error.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
How to expand reconstruction error to mean squared error in Variational AutoEncoder?
How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$?
[reconstruction error]
$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$
[mean square error]
$\mathbb{E} \Big [\big(x - p_\theta(q_\phi(x))\big)^2  \Big]$
[MSE pseudo code]
def reconstruction_loss(y, t):
    # MSE y:predicted value
    #     t:true value
    return square(y - t).mean()

Is the reconstruction error just an idea and is actual formula the MSE? Is that all?
I mean, why does MSE (or BCE) comes from reconstruction error?

I see 2 problems/questions here (reconstruction to MSE and reconstruction to BCE). Could you please focus on 1 problem at a time? You should create a separate post for the 2nd question. Thanks. You should also provide more details about why you want to do this and why you're stuck at doing it or what your ideas are.

I edited the question clearly. Thank you.

Thanks. Can you please now provide the source of the MSE you're showing us? This isn't from the original paper. Something doesn't seem to be right in that formula.

Yes. I Added pseudo code of MSE.

Yes, the formula is not consistent with the pseudocode. The parentheses are not correctly placed. You need to square the subtraction according to the pseudocode

In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should be able to derive something interesting. See equation 12 from the VAE paper. You can check out the definition of the Gaussian pdf on Wikipedia. If you're stuck, you could start with the simple regression problem and, if necessary, watch e.g. this Hinton's video.

If you model the distribution $p_\theta$ as Gaussian then:
$\displaystyle p_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $
$\displaystyle \log p_\theta(x|z) = -log \sigma - \frac{1}{2}\log 2\pi -\frac{1}{2} \frac{x-\mu}{\sigma}^2$
If you decide that your neural net will only produce the mean $\mu$ and you will use a constant $\sigma$, then the derivative of $\log p_\theta$ is the same as the derivative of $MSE$, but scaled with $\sigma$.
Now, given that this is the most common choice of model (Gaussian with constant $\sigma$), people just optimize the $MSE$ and adjust their learning rate accordingly.

Is that $ - \log\sigma - \frac{1}{2} \log 2\pi - \frac{1}{2} \Big( \frac{x-\mu}{\sigma} \Big)^2 $ ?


text:
In the figure of Stable Diffusion, when does the switch part change?
In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?


text:
About cosine noise schedule in Diffusion Model
Could you provide the explanation of Figure 4 from the paper at Improved Denoising Diffusion Probabilistic Models?
(1) The paper says, "the end of the forward noising process is too noisy,and so doesn’t contribute very much to sample quality". But if the goal is to have an image with only noise, why is it problematic to have a lot of noise?
(2) The paper says, "a model trained with the linear schedule does not get much worse (as measured by FID) when we skip up to 20% of the reverse diffusion process". But why is it the reverse diffusion process? Shouldn't the deterioration be related to the forward  process?
(3) Also, why is the training process relevant?
It seems to me that the explanation of this section means 'linear noise schedule is better than cosine one'.
Could you explain, please?

I will suppose that you already have understood how diffusion models work. Some good resources are this blog and the DDPM paper.
If we look at Figure 3 of the paper, we see that in linear schedule the image are almost purely noise in the last quarter.
During sampling, we usually perform the same number of steps the model was trained on, but in the linear schedule the reverse process will just turn some random noise into some other random noise for 1/4 of the time.

If those steps are irrelevant in terms of image quality (estimated using FID), it means that we are just wasting time (and a bunch precious energy to run GPUs)
The choice of the noise schedule used during training, i.e the degree with which the image is gradually turned into noise, will affect the performance (measuring the quality of generated images during sampling) of the diffusion model. If the performance at test time does not change much when skipping training steps, this means that the model has learned nothing for a fraction of his training steps, so the authors aim to find a schedule where each training step positively impacts the image quality at test time.
Another proposed solution of this paper is in the learned variances $\Sigma_\theta(x,t)$. The original paper sets $\Sigma_\theta(x,t)= \sigma^2 I$, while this paper proposes an interpolation
$$\Sigma_\theta(x,t)= \exp(v \log \beta_t + (1-v) \log \tilde{\beta_t})$$
In figure 8 they show that with this modification the number of sampling steps can be reduced without losing image quality (measured by FID). All but $\Sigma_\theta(x,t)= \beta^2 I$ obtain the same performance in 400 vs 4K denoising steps.

Thank you for reply. I understand about the Figure 8. On the other hand , does Figure 4 mean that the cosine schedule gets much worse when skipped up to 20%?

In the case of Figure 4, does it mean cosine schedule is not good?

@diffusionstable no, the cosine schedule is good because intuitively every denoising step is useful to produce a better image, whereas in linear schedule many steps do not contribute in improving the final result.

I'm sorry to append another question, but since a higher FID value indicates lower accuracy, regarding the FID figures, it is written that even if the reverse diffusion process is skipped by 20%, the linear schedule does not deteriorate. What does this mean? (It seems the linear schedule is good.)

@diffusionstable A Higher FID indicates a lower image quality. Skipping steps while mantaining the same FID is a nice property during sampling, but not during training. If some (up to 1/4th, in this case) of the steps were not useful during training, the model will likely produce lower quality images, compared to a model trained with a schedule where each step is useful for learning. You can observe this by looking at table 1, where the linear schedule achieves a higher test FID compared to the cosine schedule.

Intuitively, the noise schedule shall be smooth and progressive enough to be easily approximated by the NN.  In this context, a linear schedule is plausibly too "easy", and hence prone to overfitting.
In contrast, the modulation of the cosine schedule provides a slightly more challenging task to the NN (yet remaining smooth and gradual), so that when the reverse process is learnt, it is more robust.

Thank you for relay. I understand the overfitting. I see.


text:
Do I need residual block in a transformer model if vanishing gradients don't exist?

In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^3 will be visite not l’Afrique itself because visite gives more context to l’Afrique. So when all the V values are summed A^3 will contatin more of visite embedding than l’Afrique embedding. So my question is this theoretically if vanishing gradients did not exist would a residual block still be required for a transformer network to work because it would need to maintain the information of the original word x^3 and combine that with A^3 since A^3 has more information about X^2 than X^3 ?

Please, edit your post to format the formulas with mathjax.

Also, please, put your specific question in the title. Thanks.


text:
How exactly is backquery supposed to work in this situation?

Context: This code is based on a 3 layer fully connected neural network trained on had written numbers 0-9. This back query code will then take in an output value of 0.99,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01 and then its run backward through the network to get the pixel values at the beginning of the network to see what the defention of a 0 is to the network.
So my question is after the inverse sigmoid is applied amd that vector is multiplied by the vector of the transposed weight matrix how is that supposed to give me the activation values from the layer previous because if I do a dot product between two matrices $W*X=Z$ and then transpose $W.T*Z$ that does not give me X ? So then how could back query be useful ? It clearly is useful cause when I run the code it shows me the networks idea of a 0 but cant piece together how it works in my head.
Here is some additional context

It would be better if you provided the actual code rather than the screenshot, if people needed to refer to it.


text:
How Xavier Initialization formula works
In the research paper on Xavier initialization what is the purpose of putting $n_{in}$ and $n_{out}$ under 2 and added them it just says as a compromise but it is not exactly a harmonic mean or an average it seems random ?

More of an extended comment but it felt worth giving as an answer.

Equations (10) and (11) give you the estimates $\mathrm{Var}(W^i) = \frac{1}{n_i}$ and $\mathrm{Var}(W^i) = \frac{1}{n_{i+1}}$ respectively. Equation (12) is the harmonic mean of these estimates.
As pointed out here, one simple way that Glorot and Bengio could have arrived at this "compromise" (without thinking about harmonic means) is to add together equations (10) and (11).
The paper "Delving Deep into Rectifiers" by He-Zhang-Ren-Sun (PDF), which introduces Kaiming initialization, argues that it shouldn't actually matter if one uses $\frac{1}{n_i}$ or $\frac{1}{n_{i+1}}$ (or any number between them, like the harmonic mean). Here's the argument: suppose that we pick $\mathrm{Var}(W^i) = \frac{1}{n_i}$, so the signal variance is constant through the forward pass. Then over the backward pass, the variance of the gradients scales by a total of $$\prod_{i=1}^{L-1} \frac{n_{i+1}}{n_i} = \frac{n_L}{n_1},$$ where $L$ is the number of layers. For most typical neural network designs, this will be within an order of magnitude of 1; also, it won't change if the network gets deeper. So even if the gradients scale somewhat over the backward pass, the scaling isn't exponential and we don't expect vanishing/exploding gradient problems.
As you hopefully realized, what's in the screenshot is just a calculation of variance, and we still have to choose a distribution. People typically use the uniform or normal distributions with mean 0 and the given variance. The 6 in the numerator of the formula for the "Xavier uniform" distribution is just there to give a distribution with the variance in (12).

wow that was very helpful cleared up a lot of questions I had

Xavier initialization scheme with harmonic mean of in/out info flow rates is more conservative than He, even if the gradients variance scaling isn't exponential, in deep layers it may still very likely become large enough to cause sigmoid/tanh activations exploding or vanishing. He scheme is more appropriate for the more common Relu activations since Relu is more stable to become saturated comparatively.

@cinch I think that the important difference between He and Xavier initializations is that the output of a ReLU has roughly half the variance as the output of a sigmoid/tanh, and so He et al. initialize the weights with roughly twice the variance. I would point you to p. 1030 of the He et al. paper (paragraph starting "We note that it is sufficient to use...") for the argument I gave in #3, which applies equally well to non-ReLU activations.

@PaulVanKoughnett can you please explain the part about how "this will be within an order of maginitude of 1 also it wont change as network get deeper"

@Stef Sure. Say we have a fully-connected network with 5 inputs, a 20-dimensional hidden layer, and 10 outputs. In this case the ratio is 2. So during backpropagation, the variance of the gradients will double -- but this isn't actually a severe enough issue to lead to exploding gradients. Moreover, if we make the network deeper by adding more hidden layers, the ratio doesn't change. You should contrast this to an improperly-initialized network where every layer is scaling the gradient variance by 2 -- in this case the gradients would start to explode once the network became deep.

@PaulVanKoughnett   So basically you can pick either 1/ni or 1/ni+1 and you will have no variance increase on the way forward and then on the way back ward you will have some increase or decrease in variance. But if you don't pick either then variance will explode or vanish both directions a lot ? Is that about right ?

@Stef Yeah, that's right -- but more generally, if you pick numbers around the same size (e.g. the harmonic mean), then you have controlled variance change on the way forward and back, so you'll still be fine. (Also worth emphasizing that the whole argument only applies to the initial state of the network -- definitely possible for gradients to start vanishing later on in training.)

Wait I thought it was talking about all training as a whole not just the initial state ?

@Stef AFAIK there's no guarantee that training will preserve whatever statistical conditions you impose at initialization.

Note your case of numerator $2$ is chosen for normal Xavier initialization, there're other choice such as $6$ for uniform Xavier initialization. So the heuristic intuition behind the smaller number $2$ is related to applicable deep feedforward neural network architectures. When initializing weights for activation functions like sigmoid and tanh the normal Xavier initialization technique is suitable because these functions have derivatives that are relatively small compared to the linear activation functions such as relu. For inputs that are far from zero, the derivative of the sigmoid or and tanh function becomes very small or approaching zero to cause vanishing gradient or activation saturation problems, therefore the variance of initialized weights ought to be scaled smaller to keep activations within a reasonable range regardless of the layer sizes.
In summary the formula here is somewhat heuristic and there might be cases where other methods, or even different forms of the Xavier initialization, work well for specific architectures or datasets.

Could it also be that if one layer is much bigger than the other then that means the input or activation is being looked at in a bunch of different ways in the bigger layer so the variance of the weights decreases rapidly with the formula they used indicating that maybe you don’t need that much variance if you have a ton of neurons because the input or activation is being looked at in a bunch of different ways

See the Xavier paper: "For tanh networks, the proposed normalized initialization can be quite helpful, presumably because the layer-to-layer transformations maintain magnitudes of activations (flowing upward) and gradients (flowing backward)... we have used the same number of units per layer. However, we verified that we obtain
the same gains when the layer size increases (or decreases) with layer number". Thus your intuition about the relative layer size is incorrect, the desirable thing for DL weight initialization is to maintain variance from above across layers such as Var(z)~=Var(X).

In Xavier initialization, the key factor is the number of inputs and outputs in the layers and not so much the method of randomization. The goal is to maintain the variance in bounds that enable effective learning with various activation functions.

So in froward prop if x has variance var(X) and W has a variance Var(w)=$1/n_{in}$ when you do forward prop for a layer and then compute the variance for z will it be $n_{in}*Var(x)*1/n_{in}=Var(Z)$ so then by defualt $Var(z)=Var(x)$ or does the variance not directly multiply ?     (I am only considering keeping the variance the same in the froward prop so I can simplify the reasoning I know I need to take into account the backprop variance as well)

Now you're correct. The ith layerwise variance is derived by equation (5) in the same paper, so to achieve above variance stability goal equation (10) which is same as your expression must hold for each layer. Hope this clarifies and helpful.

and so the equation $2/(n_{in}+n_{out})$ is using the harmonic mean to find a proper balance so that the variance doesn't get explode or diminish on the way forward or backward ?

Obviouslya your specifically concerned formula is not exactly harmonic mean of nin and nout but resemble harmonic mean, as explained in my answer it's mainly derived empirically to strike a balance between avoiding vanishing gradients (if var(z) is too small) and preventing exploding gradients (if var(z) is too large). Comments are already too long. Please accept and close the question if you're satisfied.

The variance formula derived in the paper is independent of the choice of distribution to initialize the weights from. The 6 in their uniform distribution is just to produce a distribution with variance 2/(n_in + n_out). Additionally, the "compromise" they give is the harmonic mean of 1/n_in and 1/n_out, the variances that would be selected by (10) and (11) respectively. (I am as confused as OP about why they chose this "compromise" in paprticular.)

@PaulVanKoughnett Eq (12) happens to be the harmonic mean of 1/nin and 1/nout (not nin and nout). Generally harmonic mean is suitable for rates where both 1/nin and 1/nout can be interpreted as in/out flow rates of a fully connected layer, while arithmetic mean generally more suitable with normally distributed data and is always >= harmonic mean. As I explained in my answer for sigmoid/tanh activations the initial weights should be smaller, that's the only 'intuition' here, other than this, as you see one of the final conclusions of the paper admitted there's still many myths in the scheme.

Yes, that's what I said. The intuition about rates is interesting, though -- is there any justification in thinking of these numbers as flow rates? By the way, I found the discussion in the Kaiming initialization paper a little more understandable than in this paper. It implies that just using variance 1/n_in or 1/n_out shouldn't affect anything for common network designs (in the other direction, the variance will scale by a reasonable constant).

@PaulVanKoughnett It's very rough analogy with incompressible fluid dynamics, stable training is like to keep the incompressible fluid in all the weight pipelines flow forward and backward for their respective passes. Variance of weights are like flow rates, if harmonic average rates are not followed between layers it cannot keep its incompressibility on the 2-way for a long time which is necessary for DL, thus unstable. But these are very speculative together with special physics knowledge and shouldn't further confuse the public here unnecessarily, better left for private discussions.


text:
Deep Q Networks v Monte Carlo Tree Search in Alpha Zero
Recently I've been studying how Deep Q Networks work, and as I was reading I just assumed that game engines like Alpha Zero use Deep Q Learning to choose actions. But as I was reading the Alpha Zero papers and I saw that they were using Monte Carlo Tree Search with a Neural network in the back to provide policy and value predictions. My question is why didn't the Alpha zero team use a deep Q learning approach if it is capable of building complex representations of the state. What are the advantages to using a Tree search algorithm for this particular circumstance and in general what are the different situations where one would prefer to use Deep Q learning and where one would go for a Monte Carlo Tree approach.
My current theory is that Deep Q Learning approaches require dense reward environments and games like chess don't really place emphasis on mid game rewards. The only thing that matters is to win the game whether by a lot or a little.

DQN and MCTS (+ NNs) are algorithms of two distinct RL flavors:

DQN is model-free RL: it's able to learn the value of actions without having access to the underlying MDP (i.e. environment's model), so that you can later retrieve a ($\epsilon$-greedy) policy.
MCTS is a planning technique in model-based RL. You can use it only if you have access to the model of the MDP (or to a learned approximation). Having the true model is quite beneficial, because you can plan in advance meaning that you can reason about which action is the best by seeing what would happen in the future.

DQN is able to do a 1-step lookahead in the future by looking at the action-value of the next state-action pair, i.e. $Q(s',a')$ - there are also extensions to $n$-step learning, anyway $n$ is relatively small (because that whould increase variance a lot) - instead MCTS can explore the full tree of interaction (where you have all the possible state-action combinations) until termination of the episode. Indeed, exploring the full tree is always impractical and MCTS does this in a randomized fashion (thus the name Monte Carlo) to approximate the value.
AlphaGo/Zero uses also a value network to provide a further estimate of the next state value, $V(s')$, which is combined with the classical notion of value provided by MCTS itself, then it has also a policy network that is learned to predict the actions found useful by MCTS.

My current theory is that Deep Q Learning approaches require dense reward environments and games like chess don't really place emphasis on mid game rewards.

Partially true, I mean exploration technique and reward shaping can help with sparse rewards, but the point is that having access to the underlying model makes everything easier to learn. Consider that DQN has to implicitly learn the model (actually it learns an association between state-action and value), and after learning it cannot change its beliefs if the state distribution shifts for example. Instead, MCTS can even adapt to changing distributions because you plan your action in advance, so without experiencing the actual environment.

My question is why didn't the Alpha zero team use a deep Q learning approach if it is capable of building complex representations of the state.

Their value network can, in principle, learn complex representations of the state too. It is used as a heuristic to improve the search of MCTS. In principle, I think is possibile to use a Q-network too but one has to change the heuristic that MCTS follows to judge actions which guides the exploration of the game tree.

I think it can be such difficult to use DQN to play Go.Since Go has about 361! states.This results in most of the obtained Q(s',a') being sparse in all cases, which makes the convergence very, very slow.


text:
Can DQN find optimal path while avoiding random obstacles?
Can an agent trained with a DQN algorithm in a grid world, avoid obstacles (randomly appearing during the run time) and still find the optimal path to finish a task? The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible.

Yes, DQN can do this, but if you want optimal behaviour, you will not be able to use a really simple representation of the state, with only the position of the agent tracked. In order to predict expected returns, and take an optimal path, the agent needs to be told somehow where these obstacles are.
In this kind of scenario it is common to use some "image" of the grid that the agent is navigating, with walls, obstacles and the agent changing the value at the grid point they are occupying. Then the agent can use a convolutional neurtal network (CNN) to approximate Q values. You don't have to do this, any representation that tracks all changeable positions between episodes should work. However, the grid view of the environment is reasonably generic, and would cover many possible versions of the environment (e.g. you can change number of obstacles, add moving obstacles and other grid elements, all using a similar representation and starting agent model).
The training session will need to include new random locations on each episode, so that the agent can learn to generalise how it should deal with them. It will take longer for the agent to learn this generalisation, than if it learned a static maze.

Thanks for your answer. I see what you mean. I can make all information on the grid world accessible by the agent except the obstacles, cause they are really generated in real time. Then maybe I have to look into solution for partially observable RL solutions?! At the moment I will try to train for many episodes so it can see all the possible locations of the obstacle but then the target locations (locations to be visited) also changes in each run. So it becomes way more complicated

@Mamora if the agent is allowed to see the obstacles then you can put them on a grid. This is a lot like the DQN Atari agent. If the obstacles have to be invisible to the agent and it needs to learn their position in real time by bumping into them, then I might have some suggestions - ask a new question and give more details of your problem

@Mamora if the agent is allowed to see the obstacles then you can put them on a grid. This is a lot like the DQN Atari agent. If the obstacles have to be invisible to the agent and it needs to learn their position in real time by bumping into them, then I might have some suggestions - ask a new question and give more details of your problem

@Niel Slater, the obstacle is only visible when the agent is close to it. It can see one step ahead. Thanks, I will ask a new question then :)

@Niel Slater this is the new question with more details about the problem: https://ai.stackexchange.com/q/41112/73668


text:
How is OpenAI embeddings obtained
There is OpenAI embedding API https://platform.openai.com/docs/guides/embeddings. How is this embedding related to the GPT3.5 transformer model architecture? Is it the vectors learned from the input embeddings part, which is the summation of word embedding and position embedding, at the end of model training?
Any tutorials or guidance will be helpful.

Please, one question per post and focus on the theoretical aspects, otherwise, this could be off-topic.

Thanks for the suggestion. Edited the question.

It generally represents the output from the last layer of the encoder architecture. However, multiple methods are used to obtain the embeddings by designers. Some take the embeddings from a specific hidden layer, some average out the outputs from multiple layers, some get the embedding from the last layer.

Thanks! This is helpful. Are the optimization goals for 'model used for inference purpose' and 'model used for embedding purpose' the same?

Yes, kind of. So, for embeddings, you use the encoder part of the architecture, but for inference, you also need the decoder part. The training process or optimization goals mostly are the same for both Transformers and BERTs.

I can't seem to find GPT3.5 amongst the models used for embeddings so it's hard to say how these embeddings are obtained.

I share a similar curiosity. I'm pondering whether they employ a Siamese network architecture, akin to what is utilized in Sentence Transformers. The Sentence Transformer incorporates additional layers post-BERT, which are trained on a corpus of analogous sentences and necessitates a corpus comprising pairs of similar sentences. It's quite plausible that they may not be employing a Siamese architecture. Unfortunately, there doesn't seem to be an abundance of details available on this matter.


text:
Ignoring aspects of text embeddings, e.g. making the embedding topic-agnostic
Imagine a large set of text embeddings (e.g. by OpenAI model), created on user inputs in a natural language interface (e.g. a semantic search app), which we want to cluster on some "non-topic aspect" of the text (e.g. clarity of query). As expected, clustering by default results in different topics being grouped together.
What are the best practices to ignore the topics when clustering if we define topics as whatever some dimensionality reduction process favours (e.g. the clusters we get initially)?
Alternatively, is there some documented/researched process of training additional models on top of these embeddings (or adding layers to existing models) to boost these different aspects of the embedded text, or at least to achieve this "blindness" to certain aspects of the embedding (with some additional train data)?
Of course, I am assuming that these other aspects are also contained in the embedding, they are just less expressed.

An option that I have thought of before is to label the embeddings as good (basically take any public database good enough that has labeled data without grammatical errors).
Once you have these, you can find the top k eigenvectors by performing PCA with low variance, and there you can say with a certain confidence that these correspond to having fewer grammatical errors.
NOTE: Due to the black-box nature of Neural Networks, you can never certainly say that the eigenvector with low variance in this case has to correspond to grammar.


text:
To what extent are neural networks stable across multiple training runs?
Quick question about LLMS (and gradient descent in general): we search the space of neural networks by gradient descending in order to minimize one explicit function but what seems to be happening is that in the course of minimizing this function, the neural network automatically picks up several other skills (like building a world model...?). I imagine there is a lot of randomness in this process so to what extent are the extra skills picked up "fixed".
In other words, if I train models multiple times on the same data using the same loss function, to what extent are the resulting neural networks similar in performance (out of the training set, say)? Does the answer to this question matter much on what the loss function is and how much training has taken place?
Given the proliferation of LLMs with not too dissimilar behaviour, I expect the answer to the above question is positive (maybe in the limit that the training time tends to infinity). But theoretically, this is a little surprising to me that there is an "almost unique" minimizer of the given loss function. Do we have a good theoretical framework for explaining this?

If you want determinism make sure you program it in
A machine learning model will be deterministic to the same extent as any other computer program.
It is entirely based the stability of the inputs you put into it.
If you initialse a network to random weights you have potentially introduced some non-determinism. You can get it back by using a known seed and a known random number generator (library behavior can be system dependendent.
Likewise if you put the same input data in you should get the same results.
I suspect this is not the question were asking or the answer you were looking for so...
Consider the fitness landscape for the problem
A learning system is trying to find a good solution to a problem.
For example you are trying to minimise the error between the output of your model and actual results.
The complete space of possible solutions gives you a fitness or energy landscape.
Your learning is trying to navigate this to find solutions and avoid local minima.
So if you now add noise to this system. Use a different subset of the possible input data, different initial weights or a different learning algorithm the fitness landscape is still broadly the same so you should get a similar answer.
If you change things too much of course this is no longer the case.
Maybe a interesting set of input data adds a new peak or trough to the landscape.
Maybe a tweak to your learning algorithm enables it to escape a particular kind of local minimum or get trapped in another.
This is probably more the answer you were looking for.
It terms of a theorhetical model they probably exist but require a lot of maths and good citations which others may be able to provide.

Thanks, the second part is definitely more of what I am looking for. I understand the theoretical justification but I am wondering what experiements have been done along these lines/theory developed based on these experiments.

There's this paper on the efficacy of random initialization on model performance:

The conclusions are that even if the variance is not very
large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.

Part of what you said also reminds me of the lottery ticket hypothesis, motivated by the idea that you can prune a large portion of a NN's parameters while keeping the same accuracy, but resulting sparse architecture is difficult to train from scratch. The idea is that certain subnetworks in a NN may happen to be initialized s.t. they are particularly effective for some task.


text:
Backpropagation with CrossEntropy and Softmax, HOW?
Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the softmax activation function to Zs, for example, O1 = softmax(Z1)), and Ys be the target values (which are 0 or 1 because in this example we are dealing with classification problems and using one-hot encoding). E is the sum of the neuron's loss using the CrossEntropy loss function.
Let's say our neural network has 2 neurons, and Y1 = 1 (so Y2 = 0). What is the derivative of E with respect to Z1 and the derivative of E with respect to Z2?
After calculations, I came to the conclusion that the value of all derivative of E with respects to Zs(Z1 and Z2) should be equal, becasue they are all equal to O1-1 ( since Y1 = 1 as i said), so am i right or wrong?(and why)

The derivation of the softmax function is a bit tricky because except the other common activation functions(sigmoid, relu...) all the Zs values impact each other. It is because when you are calculating the value of the softmax function at the denominator you are summing all the exponentials of the layer's values. For short, one input value(which could be Z1 in this case) of the softmax function has an impact on all output values(which are Os in this case).
So, you need to calculate separately for each output value(Os) with respect to on of the input values(Z1 for example) and add them to get the derivative of E wrt(with respect to) Z1. Or in other words you need to calculate the derivative of O1 wrt Z1, then derivative of O2 wrt Z1 and when you add the two values you will get the derivative of the E wrt Z1(comes from calculus).
Now let me show you the derivatives of E wrt Z1 and Z2 won't be same.

No it's not the same. First the derivative of the cost function is taken with respect to the weight and not the input. This is usually done using the chain rule of calculus. To calculate this using the chain rule we take the derivatives of the activation of each neuron(I.e the derivative of the activation function which takes in input from the previous layer and is parameterized). So the derivative of the cost function with respect to the weight is dependent on the position of the weight and the activation functions that came before it(from the previous layers). If their activations differ then their derivative isn't the same.
So the derivatives of E with respect to the weights of Z1 and Z2 are different.


text:
Samples from a reverse diffusion process with cosine noise schedule blow up
I have implemented a diffusion probabilistic model, and I am finding some of the model behavior unexpected.
When I draw samples from an untrained reverse diffusion process with 20 denoising steps using a cosine noise schedule, I find that the elements of my final denoised samples have a magnitude of around 400 (this number increases into the 1000s if I reverse diffuse for more steps). Is it normal for the magnitude of the final denoised sample to scale with the number of diffusion steps in this way?
I have given a MWE below where I have replaced the predicted noise (provided by a neural network in a denoising diffusion probabilistic model setting) with actual noise (I don't think this matters, as you can set the predicted noise to 0 and the same effect is still observed).
def cosine_beta_schedule(T, s = 0.008):
    """
    cosine schedule as proposed in https://proceedings.mlr.press/v139/nichol21a/nichol21a.pdf
    """
    t = np.linspace(0, T, T + 1)
    alphas_cumprod = np.cos((t / T + s) / (1 + s) * np.pi / 2) ** 2
    alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
    betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
    betas_clipped = np.clip(betas, a_min = 0, a_max = 0.999)

    return betas_clipped

import numpy as np

T = 20 # number of diffusion steps
betas = cosine_beta_schedule(T)
alphas = 1. - betas
alphas_cumprod = np.cumprod(alphas)
alphas_cumprod_prev = np.hstack([1., alphas_cumprod[:-1]])
sqrt_one_minus_alphas_cumprod = np.sqrt(1. - alphas_cumprod)
posterior_variance = np.clip(betas * (1. - alphas_cumprod_prev) / (1. - alphas_cumprod), a_min = 1e-20, a_max = 1)

# reverse process sampling loop (Algorithm 2 of https://proceedings.neurips.cc/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf)
x_t = np.random.randn(1)
for t in np.arange(T)[::-1]:
        noise_pred = np.random.randn(1) # THIS IS A SURROGATE FOR A NEURAL NETWORK THAT PREDICTS THE NOISE ADDED TO DATA
        posterior_mean = 1 / np.sqrt(alphas[t]) * (x_t - betas[t] / sqrt_one_minus_alphas_cumprod[t] * noise_pred)
        posterior_variance = betas[t] * (1. - alphas_cumprod_prev[t]) / (1. - alphas_cumprod[t])
        x_t = posterior_mean + np.sqrt(posterior_variance) * np.random.randn(1)


text:
Justification of Scaling in Classifier-Free Guidance in Diffusion Models
Background | Classifier-Free Guidance Derivation

To summarize the derivation of Classifier-Free Guidance, looking at this paper (Page 21.), we can write classifier guidance as: $$\nabla_{x}\log p\left(x_{t}\mid y\right) =\nabla_{x}\log\left(\frac{p\left(x_{t}\right)\cdot p\left(y\mid x_{t}\right)}{p\left(y\right)}\right)
    =\nabla_{x}\log p\left(x_{t}\right)+\nabla_{x}\log p\left(y\mid x_{t}\right)-\nabla_{x}\log p\left(y\right)
    =\nabla_{x}\log p\left(x_{t}\right)+\nabla_{x}\log p\left(y\mid x_{t}\right)$$
and we can amplify the guidance by adding a factor $\gamma$:
$$\nabla_{x}\log p\left(x_{t}\mid y\right)=\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\nabla_{x}\log p\left(y\mid x_{t}\right)$$
Then to derive classifier-free guidance all we do is rewrite the first equation we saw:
$$\nabla_{x}\log p\left(y\mid x_{t}\right)=\nabla_{x}\log p\left(x_{t}\mid y\right)-\nabla_{x}\log p\left(x_{t}\right)$$
and substitue it into the 2nd equation we saw to get $$\nabla_{x}\log p\left(x_{t}\mid y\right)=\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\left(\nabla_{x}\log p\left(x_{t}\mid y\right)-\nabla_{x}\log p\left(x_{t}\right)\right)$$
and this is classifier-free guidance.
We can also rewrite it to be $$\nabla_{x}\log p\left(x_{t}\mid y\right)=\gamma\cdot\nabla_{x}\log p\left(x_{t}\mid y\right)-\left(\gamma-1\right)\cdot\nabla_{x}\log p\left(x_{t}\right)$$
The Question

It seems reasonable to me that as we can amplify and play with the guidance factor, we could also add a factor $\beta$ to play with how much we go in the direction of $\nabla_{x}\log p\left(x_{t}\mid y\right)$ and in the classifier guidance get $$\nabla_{x}\log p\left(x_{t}\mid y\right)=\beta\cdot\nabla_{x}\log p\left(x_{t}\right)+\gamma\cdot\nabla_{x}\log p\left(y\mid x_{t}\right)$$ and in classifier-free guidance get effectively free control over both terms: $$\nabla_{x}\log p\left(x_{t}\mid y\right)=\gamma_{1}\cdot\nabla_{x}\log p\left(x_{t}\mid y\right)-\gamma_{2}\cdot\nabla_{x}\log p\left(x_{t}\right)$$
I wanted to know why adding this extra control over the direction we move in doesn't help, why does anything we choose besides $\gamma_{1}=\gamma_{2}+1$ yield bad results? Adding this extra factor $\beta/\gamma_{1}$ seems as intuitive and arbitrary as adding the $\gamma$ factor at the derivation of Classifier Guidance (at the beginning).
Is there a deep explanation or at least intuition behind why shouldn't this work?

There is an intuitive reason if you look past the logarithm and the score.
The fact is that classifier-free guidance is really a geometric average between the conditional and unconditional probabilities. Basically,
$$p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma}$$
Why is this valid? In the limiting case, consider a scenario where $x$ is actually independent of $y$, i.e. $p(x \mid y) = p(x)$. In that case,
$$
p(x \mid y) \propto p(x \mid y)^{\gamma}p(x)^{1 - \gamma} = p(x)^\gamma p(x)^{1 - \gamma} \propto p(x)
$$
It is consistent.
However, if you consider arbitrary scaling factors $\gamma_1$ and $-\gamma_2$, you lose the geometric consistency. Effectively, what is going on is:
$$
p(x \mid y) ^{\gamma_1}p(x)^{-\gamma_2} \propto p(x \mid y)^{\tau}
$$
Where $\tau$ is some constant. The limiting case where $x$ and $y$ are independent makes this obvious, you get $p(x)^{\gamma_1 - \gamma_2}$. So as long as $\gamma_2 \neq \gamma_1 - 1$, you are sampling from some $p(x \mid y)^\tau$ (which is usually strange stuff way off the data manifold), while you really want to just sample from $p(x \mid y)^1 = p(x \mid y)$.

You probably could express the guidance score in that form, but it's important understand the intention behind how things are presented.
In this context, we only really care about the relative magnitude of the conditional and unconditional terms of the guidance score. When controlling the generation of images with class conditioning, it wouldn't matter how large the conditional term is in isolation if it's overshadowed by unconditional term.
By writing the guidance this way, we think about the score in terms of which signal (either conditional or unconditional) is dominating the overall feedback to the model. This "level of domination" only needs one knob to express ($\gamma$).
If you were writing this paper, you could add an extra scaling term around the guidance score, adjusting the overall magnitude of the feedback and achieve the same degrees of freedom as your suggested formulation. Looking at the original classifier-free guidance paper, it seems like there is a similar hyperparameter that you could adjust, although it's not exactly the same.
However, you could just as reasonably omit this term as (1) it would require extra space to explain, but it (2) doesn't change the formula significantly, and (3) is an adjustment that readers could reasonably arrive to themselves, if they do end up wanting this extra flexibility.

First of all, I must again state that, from my experiments using huggingface models, using any parameter other than $\gamma_{1} = \gamma_{2} + 1$ yields awful results (not even a normal picture), so I'm not sure your answer is correct.
Secondly, what extra scaling hyperparameter are you talking about in the original CFG paper?

@snatchysquid What values were you trying that worked/didn't work? Can you also link to the models that you were using/where you were making the changes?

I used for example the model in the following colab notebook and simply change the Classifier-Free Guidance scale. choosing even a difference of 1.25 (instead of 1) between $\gamma_{1},\gamma_{2}$ resulted in either a blank (brownish) image, or a blank image with a small figure that sometimes matches the prompt (e.g. a banana riding on a skateboard and wearing a hat)


text:
Why use learning rate schedules if weight updates automatically decrease when approaching local optimal?
Andrew Ng said in his slide that:

However, there are numerous types of 'learning rate schedules' in TensorFlow that change the learning rate profile as training progresses.
If it's true that these adjustments are redundant then why do we need learning rate schedules?

In one hand you have the fact that if the initial step-size is a bit too optimistic, decay will at some point lead to a step-size that works
However, there is a much more theoretical reason for it.
Indeed if you consider a non-smooth function, it's necessary sometimes to converge to decay your stepsize...
As an example, consider $f(x) = |x|$, with an initial guess of $x = 2.5$ and a stepsize of $\alpha = 1$
Well, the gradient of such function is $1$ if $x > 0$ and $-1$ if $x < 0$
Now, apply GD: $x = x - \alpha \nabla_x f(x) \rightarrow x = x - \nabla_x f(x)$, and considering that the gradient on $x = 2.5$ is also 1, becomes $x = x - 1$
Therefore, you will start from 2.5, then next time go to 1.5, then again to 0.5, and at that point, you'll jump to -0.5, and from there jump back to 0.5
As you can see, you will never converge to the optimal $x = 0$, except if you decay your stepsize

The keyword here is can. Deep learning models have millions of parameters making the loss landscape incredibly complex. Although everything in the slide still applies -- GD will take smaller steps as you get closer to a local minimum and you could theoretically converge using a fixed learning rate -- you almost always need extra tricks for training a neural network.
For example, you might want to set the initial learning rate of your model higher and decrease it as training progresses because the loss landscape for neural networks has many local minima and you don't want to converge prematurely into a suboptimal one. (Although, even this is a huge oversimplification of deep learning optimization.)
Contrast that with a much simpler (e.g., linear) model. You won't see learning rate schedules being used to train these models because the loss landscape will be very simple (in fact, for a 1D linear model, it'll look exactly like the diagram in the slide). In this situation, the learning rate schedule would probably be redundant.


text:
What technique is used for training Large Language Models like GPT?
I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models.
Deep learning, generally, can involve training with supervised learning with labeled datasets which makes sense.  But I also encounter references to unsupervised learning: specifically, predicting the next word in a sequence. The concept of unsupervised training with next-word prediction is unclear to me. To me, it feels like GPT is trained with supervised learning with a labeled dataset: is this correct?

These models are trained with unsupervised learning, but it's probably more intuitive to describe it as self-supervised learning.
Your confusion makes sense: the next-token prediction objective, which trains models to predict the next token given the previous token, is basically just doing supervised learning with the inputs being the prefix of the text and the labels being the next token. But, here, the labels are created by us, from the same corpus that the input is from! So the model is supervised by the data itself: "self-supervised" learning.
Also, to be clear, self-supervised learning is unsupervised*, you don't really have a separate set of labels---we're making them ourselves.
* I realize that these definitions can be controversial/ambiguous. I'm basing my usage on these: 1, 2, 3, 4. You could alternatively say that self-supervised learning is a separate category altogether (e.g., these slides). But, all things considered, you probably don't need to worry too much about definitions as long as you understand the underlying concepts---people will know what you're talking about either way.

since these labels are derived from the input data itself the pre-training phase is considered self-supervised learning, which is a subset of supervised learning, as the name would suggest. Calling this process unsupervised is an error, but many people still call it unsupervised. Terminology in every field is commonly misused because people make mistakes and have different levels of understanding of the concepts. For example people often misuse link function and the inverse link across different scientific papers.
so you are correct, self-supervised learning is a form of supervised learning because we have a target value; the one thing that defines supervised learning
edit: ok according to IBM self-supervised is a form of unsupervised learning. So maybe the answer isn't so clear. my thinking keeps coming back to autoregressive models, e.g. like ARIMA, which are considered supervised.
image the following scenario. you have stock prices Y and you use previous Y data to predict the next time step. If you did this using ARIMA I think most people would define this as a supervised task. the target value represents a perfectly accurate ground truth (e.g. the stock market price is known for a fact without any doubt). This is the case despite the target not being defined using explicit external labeling.
now if we use an LSTM or GPT, we call this self-supervised, and suddenly its an unsupervised model? the only thing we changed is the model type which shouldn't factor in to if a task is supervised/unsupervised.
the below link clarifies this more and also ellaborates the confusion around this term.
source

Though self-supervised learning is a technically a subset of
unsupervised learning (as it doesn’t require labeled datasets), it’s
closely related to supervised learning in that it optimizes
performance against a ground truth.


text:
Implementing an RL agent on a variable action space
I've been trying to design and implement a DQN agent as an AI opponent option for a math game I've been working on.
The game is a turn-based grid game, where players alternate making moves and combining numbers to reach a target score. The grid consists of n by n numbers, which can only be used once, whereafter they get replaced by None and can't be selected anymore.
I have a method called get_legal_moves that creates the action space, and the state vector is declared as follows: [player_score, opponent_score, self.target_score, self.grid_size, self.agent_player, self.turns_taken, grid/hashed_state].
The part I'm having trouble with is that the grid is filled with randomly generated numbers, and the action space is constantly changing.
I don't have much experience with ML, so converting the features to tensors and having the agent properly interpret it has been a bit of a problem. All of the elements of the state vector are integers, but the grid is a list of lists of integers, or alternatively, a hashed variation.
I skimmed monte carlo trees and other solutions, but I don't really know where to start. I'd like some advice on how to go about building the agent to begin with. I keep hitting tensor matching errors and problems with the action space getting smaller or larger.

Hi @X496 and welcome to AI Stack Exchange! If possible, editing this question to have 2-3 paragraphs might help readers digest your question more easily, and it might lead to a faster or even better answer. Great question, and we hope to see more of your questions in the future!

I will try to answer your questions as best I can, potentially building on some of your ideas. Please note that I'm making certain assumptions about how to design the action space because I don't have a complete understanding of how your game functions. Nevertheless, I hope this will aid you in constructing your RL agent.
How to ingest 2d grid data to the network.
I'll start with what I find to be the simplest question: how to prepare the grid values for ingestion by the Neural Network.  For this, you would want to vectorize your grid values into a single vector, or in other words, flatten each cell value into a separate and independent feature in the network. Another possibility is to use a convolutional layer specifically for the grid data. As a general rule, if your game has some type of spatial component, it is highly recommended to use convolutional layers. These layers require fewer weights to train and tend to perform better. That's the reason why models like the Original DQN and AlphaGo use Convolutions for their game states.
Handling a Variable Action Space
When creating a Neural Network, the number of nodes in the input, hidden, and output layers remains constant. You could attempt to construct a model that changes its number of outputs or actions, but in my opinion, this would be an overly complex solution. A much better approach will be to list all possible actions in your game (legal and illegal), then apply a mask function that assigns extremely negative values (i.e., $- \infty$) to illegal actions. This means that if the action selection is a greedy one, that will make you sure that those actions are never going to be selected. If you're using an epsilon greedy strategy, you also need to ensure that the randomly selected action does not have a $- \infty$ Q value.
For enumerating all possible actions you must count all the possible actions in the game. If your game consists of a n x n grid I assume that the action space also increases by n. Additionally, you need to take into account the possible numbers each cell can contain, this can be expressed as a set F representing the random numbers that can have each cell.
Size of the Action Space
So as I mentioned earlier, I'm assuming that your action space has a cardinality of $|F| n^2$. For each cell in the grid, you have the possibility of putting any element of the field. Remember that the Agent will have to be trained on a fixed combination of these parameters. This means that for each of these combinations, you will have to make another model with another output parameter. But this doesn't mean that you cannot use the same hidden weights for training on multiple models. This technique is called model pretraining or transfer learning and this has been demonstrated to be very useful in many applications.
Another thing you might need to consider is the size of your action space. You will start to notice that the size of the action space can start increasing very rapidly as you increase the size of your grid. This is because it grows in a quadratic way. Meaning that if the grid is a 4x4 will have 16 cells but if it is a 10 x 10 it will have 100 cells of different options and that number will be multiplied by the elements of possible numbers that can have each cell. Just to give you an idea, the game of GO is a 19 x 19 board leading to a 19 x 19 = 361 action space. As a way of comparison, the Game of GO is considered to be one of the hardest games to master for an RL agent. That's why I will try to limit the size of your grid. Although it always depends on how the dynamics of the game work, and how sparse the reward can be, I think that it works as a very good comparison for your game because of the fact that both are grid-based games with similar properties. But again take this with a grain of salt because, in the end, you are one that really knows how your game works.
In the case that you consider that the action space is too big, you can always use methods of action samplings. Similar methods are used in models like AlphaTensor or Sampled Muzero. In the case of the AlphaTensor model, the action space of the model is approximately 5^12 =  244,140,625 different actions. Although this can easily be a little too far from your action space, it has been demonstrated by these papers (1 2) that sampling a subset of all your action space can be a useful approach to handle big action spaces.
Using Monte Carlos Tree Search
Finally, to address the question related to using other solutions for solving this RL problem. I will talk more about Monte Carlos Tree Search approaches just because I have more experience with these models. I will also recommend you search for feedback on other RL models like PPO or other policy gradient methods. In the case of MCTS, I already reference a lot of models that use MCTS(AlphaGO, AlphaTensor, MuZero). And there's a pretty good reason why. MCTS is an excellent model to use when your state space is too big and you want to implement a planning component. Like in the game of GO, where you have an adversarial player and your goal is to outsmart him seeing various moves ahead of your current state. This is true because MCTS is designed to make beneficial selections by striking a balance between exploration and exploitation. If you're interested, I recommend this lecture by David Silver (Lead Researcher of the AlphaGO and AlphaZero papers).  But again MCTS works as a Game Tree Search that is able to solve the problem of exploration vs exploitation in a decent way. In some cases, equally good results can be achieved by simply tuning the hyperparameters of an epsilon greedy strategy with DQNs. Thus, it's important to experiment and choose the algorithm that performs best for your use case. Given said that, by the description of your game, I will recommend you try MCTS for the sake of learning and researching.

Welcome to AI Stack Exchange. This is a great first answer!


text:
Empty space detection
I'm looking for a TensorFlow model detecting empty spaces on the images. I need to add my company logo to this empty area so there shouldn't be any faces or objects in this area.
Also, I would be thankful for information if the is any existing solution for doing that (can be paid)

It sounds like you're looking for a foreground/background segmentation model (also called image matting). There are several ways to do this.
Here's one option for a Tensorflow model: https://github.com/adumrewal/imageMatting
Paperswithcode is a website compiling paper implementations along with their benchmark scores. You can find several tensorflow-based models on their Image Matting leaderboard.
However, if using Tensorflow isn't 100% necessary, then I'd go with something like PaddleSeg instead, which provides more well-tested implementations for applications like this, whereas running research repositories can often be a bit frustrating.


text:
How does the linear layer step work and what should I do at the end of the LSTM?
So basically I've read some text about LSTM, and luckily they mentioned the linear layer step at the end of the LSTM Process. However, they didn't explain how it works or what I would need to convert the hidden state into understandable and useable information. To be clear I do not want a Pytorch implementation of this. I am making my LSTM in C on arduino. If you don't know how to code in C or it is too time-consuming to make an example, a simple explanation without too much math detail, but enough that I get the basic idea of what to do is just as good. If you don't know how to explain it in a rookie friendly way, the article I found, which I mentioned above, does a pretty good job. The name is "LSTM Networks | A Detailed Explanation" by Rian Dolphin and can be found on towardsdatascience.com. I normally would give the link, but the last time I did that, people got mad.
Thank you so much if you respond
Edit: Wait, is it just another ANN on top of it.

I’m voting to close this question because this is a "write this code for me" post and seems to indicate the OP has not done minimal research first.

If you need to get an output from the LSTM as a vector in $\mathbb{R}^N$, and the size of the hidden state is $1 \times M$, you matmul with a learned $N \times M$ matrix. This is the linear layer.
For example, if the problem is classification and you have $N$ classes, you might use the linear layer to project the hidden state to size $N$, and then softmax to get logits.


text:
Why the gradients produced by the soft targets scale as 1/T^2 in knowledge distillation?
In the paper "Distilling the knowledge in a neural network", it mentioned "the magnitudes of the gradients produced by the soft targets scale as 1/(T^2) ", but it has no mathematical derivation. So how the term 1/(T^2) is calculated?

This scaling factor is derived in Section 2.1 of the paper. $\frac{1}{T^2}$ is a coefficient for both equations (3) and (4) when calculating the derivative of the cross-entropy loss $C$ wrt a logit $z_i$. These two equations are made under some assumptions, though: the temperature $T$ has to be high compared to the logit magnitudes (for equation (3)) and that the logits need to be zero-meaned (for equation (4)).
The way you get from (2) to (3) using the temperature magnitude assumption is with the approximation: $\log(1+x) \approx x$ for small $x$. Let $x$ be $\frac{z_i}{T}$ which means:
$$\log(1 + \frac{z_i}{T}) \approx \frac{z_i}{T}$$
$$1 + \frac{z_i}{T} \approx e^{\frac{z_i}{T}}$$
Apply this to the numerators and denominators to complete the derivation.


text:
Match two paragraphs of text
I'm building a friend finder app and I need to match people based on a paragraph of text. Here is an example of what I mean:
Person A:
I love walking and going to the beach, I also love reading and keeping active. I'm very allergic to dogs, so I don't have any pets and have no intention of having any. I used to swim in college as a d1 athlete. My favourite movie is Finding Nemo, I hate horror films..."
Person B:
I'm a dog lover with 3 Labradors. I'm an extremely active person who loves to swim many days out of the week. I love going to the theatre and watching movies in IMAX. My job is an civil engineer, but I like to code games in my spare time..."
Match: 56%
What is the best way to calculate a score that accurately matches the similarity of interests/hobbies etc. between the two individuals?
I have looked at OpenAI embeddings API and storing them in a Pinecone database for retrieval. I have also looked at creating a 'Profile' using some kind of classification model that would give each user a score for various traits such as {active: 0.3, social: 0.9, ...} and then comparing results.
Many thanks for any help.

This sounds like a good project. The question is also reasonable but it does seem a bit like you are asking the community to do your homework for you. Perhaps you could indicate some of the ideas you have to start with.

Hi @BruceAdams
I'm definitely not trying to get someone to do my homework! This was an extracurricular activity that doesn't even count to my final grade.
You are right, I should have added more context to some of the ideas I have explored already.
I have looked at OpenAI embeddings API and storing them in a Pinecone database for retrieval.
I have also looked at creating a 'Profile' using some kind of classification model that would give each user a score for various traits such as {active: 0.3, social: 0.9, ...} and then comparing results.

It would be better to edit the question to say that. You are also contradicting yourslef by saying "for my MSc project" vs extracurricular.

You might also consider trying to match the text with "dating relevant" questions. Look at okcupid or similar for some examples.

I have followed your advice @BruceAdams. I don't believe I am contradicting myself as what I am working on will be looked at by my tutor (as part of my MSc), but it's completely optional and I'm doing it for fun + learning! Thanks

There are several ways to do this. The most straightforward would be to encode the two paragraphs as vectors (also called text embeddings) using a pretrained language model.
The idea is that the vectors representing the paragraphs encode the "semantic content" or the underlying "meaning" of the text, and the vector encoder model was trained such that you can use a distance metric like dot-product or cosine-similarity to measure this "semantic distance" between text. The sentence-transformers library has several implementations and tutorials for this.
One caveat is that the type of similarity captured by the models may not apply to your specific use case as they may be trained on a different distribution of data (these models are often trained for information retrieval purposes). In this case you may want to fine-tune the model or choose a model that was trained similarly to your target use-case.
Another similar method is to use cross-encoders. Instead of encoding each paragraph separately then comparing them with a cheap metric, cross-encoders are models that take both paragraphs as input and output a similarity score. This may give you better accuracy (again, depending on how it was trained), but it comes at the cost of performance since you need to do an expensive forward pass for each pair of sentences.
I would also consider a more interpretable method of first extracting "properties" from each description, then comparing each of them separately. For example, it'd be reasonable to assume that most personal descriptions have hobbies, favorite movies, etc. You can try extracting spans of text corresponding to each of these properties separately, then use one of the methods described previously (or even just basic token-matching) for comparisons.

Hi @Alexander, thanks for taking the time to write this out. There is lots for me to unpack and get started in the right direction.


text:
Are LLM hallucination necessarily a bad thing?
I've been pondering over this for a while now: is the so-called hallucinating necessarily a problem in LLM or in AI in general? What it stands for anyway, maybe the model is trying to crunch so much information into the answer we just cannot comprehend. How could it be the model's fault since it knows only what we teach them after all?
I am trying to cover LLM hallucination here as a whole considering it as a general term many of us have read about as a known problem with LLM. For further reading and understanding different kinds of LLM hallucination I recommend to check this link and this link.

I'll promote this question with slightly provocative thought; should we call hallucinating concluding instead, if the model is stable and does not add randomness to the data it has been taught with?

For reference was it Facebook who set two LLM chatbots to chat with each other, it didn't take long they created their own language no one understood anymore. Maybe we should.

Hi Harism! Thanks for the question. It would be helpful if you provide a definition for 'hallucination'. This is helpful for (1) clarity and unambiguity of your question and (2) for people who are not very experienced on the topic.

@RobinvanHoorn I will edit my question now but your request for clarity and unambiguity is not necessarily fulfilled but I do my best.

I wonder if its a bit about intuition for humans. One doesn't have a clear basis for it, but it somehow might make some sense.

Although "hallucination" is discussed as if it were a failure mode of token-predicting LLMs (which are the core of all the recent large models), this is not really any different to normal behaviour. LLMs are not designed to output correct information, they are not trained specifically to do so, and there is no loss function or reward signal during training that encourages truth (although some fine tuning or human interaction reinforcement learning applied afterwards may adjust this to some degree).
LLMs do not have inherent filters for detecting true or accurate text output. Instead they output text according to the probability that the model determines that it should appear, based on training material plus the text so far, including any "pre-prompt" setup plus the text supplied by user and it's own output so far.
The extent that LLMs output true and accurate statements is limited to:

The probability that a true statement might appear in all similar text from the training data.

The ability of the model to generalise from the training data and predict accurately.

For some AI systems, ability of added monitoring and control code models to detect and correct issues when they occur. These secondary models are typically far more limited than the LLMs, but may help.

For some LLMs there may be fine tuning that helps the core model provide truer or more useful output.  This cannot be done at the same scale as the base training however, because it requires far more human effort per training example.

In all cases, there is limited "understanding" of what the model outputs, in terms of real world relevance and accuracy. Mainly this is achieved because the majority of training data makes logically correct and accurate statements that the model will learn to predict approximately. It cannot refer back to other modes of knowledge though, or have any way to experience the world directly. So sometimes the text predictions produce nonsense that is well written otherwise.
In my experience, this nonsense happens very easily indeed. Just ask a model an expert level question on any subject. The LLM will produce text that on a first look seems reasonable, but an actual expert in the subject will tell you it makes no sense at all in a "not even wrong" kind of way. This is why LLM answers are not accepted on most Stack Exchange sites.

Interesting, I went to try expert level subject question on Google Bard and you are correct, it started to hallucinate things like I work for Google AI which is total nonsense. However it got some things correct also like the work is available GitHub despite my question was about YouTube video I've made. I'd say it was close to 40/60 percent Bard got right. Pretty impressing I'd say since I asked something about my own work no one else knows.

@harism: Of course it doesn't "know" anything, it outputs text that has a kind of consistency, and when it comes to guessing things you have done, just like someone writing realistic fiction, sometimes the words match reality.

It depends on the goal of the model.
For a model that generates a story plot based on an input prompt, hallucination might even be an welcome feature.
For a model that is meant to stand in for a subject matter expert and answer topical questions, yes, hallucination is a terrible thing.


text:
Why use exponential and log in Positional Encoding of Transformer
This code snippet is from here under the section named "Position embeddings".
class SinusoidalPositionEmbeddings(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dim = dim

    def forward(self, time):
        device = time.device
        half_dim = self.dim // 2
        embeddings = math.log(10000) / (half_dim - 1)
        embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
        embeddings = time[:, None] * embeddings[None, :]
        embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
        return embeddings

The code is implementing Positional Encoding that was introduced in Transformer model.
I don't understand why we have to use torch.exp and math.log. Is this related to scaling or non-negative issue?
Many thanks ahead!

This problem had been explained in the blog: https://medium.com/@hunter-j-phillips/positional-encoding-7a93db4109e6

The current code just implements the now-standard expressions for positional embeddings given in the original transformer paper (Attention is all you need).
In section 3.5 of this paper they suggest the formula
$$PE_{(pos,2i)} = \sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
$$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
It turns out to be easier (or more efficient?) to implement in code using the standard maths identities $x = \exp \log x$ and $\log x^a = a \log x$
$$\frac{pos}{10000^{2i/d_{model}}} = \exp\left(\log(pos) - \frac{2i}{d_{model}} \log(10000) \right)$$
Hopefully you can now see that this corresponds to the code you have given.
But perhaps you are interested in the question as to why the positional encodings are of this fairly whacky form. My understanding of the main intuitions are that sines and cosines interact really nicely with translation due to their periodicity, and by using exponentially spaced 'frequencies' one can extract signals for interactions at a large number of different 'length scales'; see this excellent blog post and associated links for further explanation. But as I understand it, the main reason this clever idea is so widely used is because it works so well in practice.


text:
Are popular songs considered outliers to AI
I was thinking about making an adversarial network to generate popular music, so one AI which generates and then two others which detect whether the song is AI generated and the expected view count of the song. The AI would thus learn to generate songs which are both realistic and popular.
However I came to a dilemma: Would popular cover songs be flagged as being AI-generated (not realistic)?
It seems like the most popular songs (Old town road, Wrecking ball, Gangnam style, crazy frog...) are all significantly different from normal music, and so would be flagged as AI generated. This would make it very difficult to generate music which satisfies both realism and popularity, since the most popular songs would have low realism scores...
Does anyone know methods to handle this sort of issue?

I think answer to your question lies in the way how ChatGPT prompt is handled to begin with. I am not too familiar with LLM but I am sure it cannot generate same lyrics each time I ask the same question. Or new lyrics at all - just answer some existing ones best it can.

You are talking about generating the lyrics or the composition?

The concept of "music similarity" isn't very clear cut -- I'm not sure that I personally would popular songs outliers. Artists take inspiration from existing music, so musical styles, production techniques etc. from popular songs should spread more than unpopular songs. I think this problem will come down to the data that you're analyzing/training on and the way you choose to encode/generate the songs.
GANs in particular have a well-documented problem of mode-collapse, where the generator over-optimizes for a single discriminator, leading to a lack of diversity in generations. Considering the diversity of different genres, styles, instruments, etc. in music, having a model capable of handling a diverse dataset is particularly important, especially if popular songs do turn out to be outliers.
For example, consider the MNIST dataset. The generator might notice that 1s are really easy to generate, so will generate realistic, hard-to-detect 1s. After a couple iterations, the discriminator will start classifying all 1s as fake. But now, the generator can simply move onto the next easiest digit to generate, as the discriminator is just checking for 1s. In other words, while real-world data has many modes, because of the nature of the "game" being played, GANs will collapse to only a few. See the linked websites for some possible solutions to this.
You may also want to consider using a different modeling method. Models like variants of RNNs and Transformers are often used (see: Table 3) as they were designed for sequential data. Given its success with image-generation, diffusion models have also been popular recently. However, GANs still have their advantages, like being faster due to their non-autoregressive nature.

I'm not sure what you plan to do, but I think that you are misunderstanding both what an outlier is and how GANs work.
An observation being an outlier is a property that's relative to the expected underlying distribution of the data. When an observation emerges that is very unlikely to have been drawn from a random variable with the same distribution as the rest, that observation is labeled an outlier. Imagine you came up with a numerical ranking for the quality of songs, and you were studying the entire field of music. Here, the sample space is the set of all possible songs, and the measurable space your variable is mapping this space into is a subset of the image of your ranking function.
During your study, you observe that this variable is distributed in a certain way: Most songs have a mid ranking, only a few songs have a very high ranking, and only a few songs have a very low ranking (let's assume this is the case, even though it's probably not). An observation that is extremely low, so low that you wouldn't expect to see a song with such a low ranking in a million years, could be considered an outlier. Likewise, a song with an exceptionally high ranking could also be an outlier. This could indicate that something happened that broke your ranking measurement (for example, a cat sat on the keyboard while your ranking algorithm was rating a song).
However, let's now imagine that we are limiting our study only to good songs, so that our sample space is no longer that of all possible songs. Maybe even consider only the set of exceptionally good songs. Could a song with an exceptionally high ranking be an outlier here? No, having a high ranking in this subset is the norm, so unless it was so incredibly high that it again destroyed even the distribution of exceptionally good songs, it wouldn't be considered an outlier. An average song, however, which wouldn't have been an outlier in the first case, would be an outlier in this second case.
So if the only thing you show to your algorithm is good songs, how are good songs going to be outliers?
That being said, this shouldn't matter to you because the discriminator of a GAN isn't concerned with distinguishing the quality of a song but with distinguishing original and artificial samples. The only samples the discriminator sees are the samples the generator is trained with and the generator's creations. An outlier here could be a song that, even though it is natural, has features that make the discriminator inappropriately think it is artificial (which I doubt a lot would mean the song is 'too good').


text:
Is there any Deep RL method that is based on value function approximation of Post-decision States
I am trying to construct an RL algorithm for managing a fleet of vehicles to maximize profit. As far as I know, the Sequential Decision Process can be decomposed into the following pic:

My current method is based on value function approximation of post-decision States $S_t^x$. But without the technique of Target Network and Experience Replay of DQN, my model is easily trapped at the local minima and can not take advantage of the earlier exploration steps just like DQN. Are there any deep RL algorithms that are based on post-decision states?
Here I enclose one paper for managing robots using the post-decision states: A Comparative Analysis of Neural Networks in Anticipatory Transportation Planning

It might be a good idea to briefly describe these "post-decision states" and the technique you're currently using to solve your problem. Is it something known in the literature or something you came up with? Moreover, it might also be a good idea to provide more info about your specific problem. "managing a fleet of vehicles to maximize profit." is not very clear (at least to me).

Hello @nbro, I enclosed a paper using post-decision states.


text:
Number of units in Final softmax layer in VGGNet16
I am trying to implement and train neural network model VGGNet from scratch, on my own data. I am reproducing all the layers of the model. I am having a confusion about the last, fully connected softmax layer.
In the research paper by Simonyan and Zisserman, the last layer of VGGNet has 1000 neurons with sotmax function. As I understand this, this layer can classify data into 1000 classes. However, my data has only 4 classes. So should I add a layer with only 4 units instead, or should I add a 1000 unit layer, and add a 4 unit layer on top of it?

You should do the first one: add a layer with only 4 units instead of the 1000 unit layer.
You can think of the first $N-1$ layers as a feature extractor, converting the high-dimensional image to a (relatively) low-dimensional dense vector, and the $N$th layer as a linear classifier over that dense vector. The goal during training is then to learn a feature extractor such that the output is linearly separable with respect to the classes.
Because you changed the number of classes, you need to change the size of the output layer.


text:
Is size of trained model on disk a good measure of model complexity?
I am writing a research paper on my own custom CNN model for image classification. I am comparing my model architecture with pre-trained architectures, like DenseNet121 and InceptionV3.
I want to compare the size of my model with the pre-trained models. I know the following metrics for judging the size of a model:

Number of Parameters
Number of Layers

However, I was also thinking about comparing the trained model size on disk. As trained DenseNet121 is taking 200MBs and my model is taking less than 20MBs. Therefore, my model is better for embedding in a smartphone application.
Is that a good comparison, or am I missing some important point?

Although it might be a good indication, there might be additional parameters influencing the size of the file. For example, the precision of the weights, are their float64, float128 etc. This significantly influences the size of the save file. Similarly, weights and layers might not necessarily be indicative of the complexity of a forward pass which is the part necessary for smartphone applications (i imagine). One architecture might do additional (complex) operations in the forward pass, that might make runtime on a smartphone application slower.

The title of your question asks about model complexity. Yet the body of your question talks about this metric as being useful for embedded systems like a smartphone, which have more limited memory.
I think the answer depends on which of those two questions you actually mean:

As a measurement of how useful the model may be on a limited system: Yes, this is very useful knowledge! However, I caution you to look at floating point precision of both models, and note that in your comparison if they are different.

As a measurement of model complexity: No, this is not a good metric. Smaller disk space does not imply that the model is less complex. For example, one could imagine a KNN taking up enormous disk space, but we would likely not consider that as complex a model as a large (but not as large!) neural network.


text:
How can a Regression based Neural Network learn class thresholds?
I understand that to solve multilabel classification problems, we can use the softmax activation function in the output layer of the neural network. The softmax function outputs probabilities of each label, and the label with highest probability is then predicted as the target label.
However, I just saw in a research paper that the authors used regression function instead of softmax function, in output layer. The paper says:

Because regression classification can automatically adjust
classification thresholds based on data distribution to maximize classification performance

I do not understand how can the model learn classification thresholds by itself?
Are these thresholds part of the neural network architecture? Are these thresholds trained like weights of layers?
This is the link of the paper: https://www.sciencedirect.com/science/article/abs/pii/S016816991931556X

please link the paper

I have added the link to the paper

First thing to notice, is that the assumptions on the target don't match the ones of multi-classifications: in particular, in multi-class classification, it's generally assumed that any other class outside the target one, is equally bad.
Instead here, it's clear that this is not true:

Given an input with target "Healty Apple", predicting "General Apple Scab" is not as bad as predicting "Serious Cedar Apple Rust"... in other words, the class order counts.
In order to capture this property, they decide to use classification regression.
About the automatic threshold, they don't say anything about it on the paper, so in my opinion what they do is to adjust them to improve performance.
On top of my head, one way is by predicting the regression score for the training set, and then fitting 6 Gaussian distributions (like a naive Bayes model), and adjusting the threshold by moving them so that they best fit the result.... or you can just plot them with different colors and check where the colors lies

Oh alright, I understand now that they are using classification regression because class order is important in this case.
Can you please also elaborate a little on using 6 Gaussian distributions for adjusting the thresholds

@DawoodAhmad it's just an idea, but an even easier one might be to do a forward pass of the whole training set, for each class see where the median element is in the number line, and then adjust the threshold separating the space equally


text:
Model-based learning in continuous state and action spaces
I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but they do not explicitly construct the model to simulate the environment, through policy gradient.
The closest literature that builds a model that I found is on continuous state space and finite action space, which is slightly different than what I aim to do. Furthermore even in this case the main problem that I face is, transition probability is assumed to give you the probability of obtaning next state, which does not necessarily have to be non-zero for instance when the transition probability is non-atomic transition probability.
I will appreciate if someone atleast points me relevant literature in this direction.

I don't see any question here. Can you please ask a specific question?

but they do not explicitly construct the model to simulate the environment probably because it's not that trivial... you can approximate it with variational models, but still you will have a very hard time building a model that works in the continuous space (model in the RL setting, thus a env simulator)

but maybe something like this might give a glance https://arxiv.org/abs/1509.02971 (however, not theoretical definition of a continuous MDP is given)

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, $p(s'\mid s, a)$, and the reward model, $r(s,a,s')$:
$$p(s',r\mid s,a)$$
In principle, you can learn it just by plain supervised learning in which given a state-action pair $(s,a)$ the prediction targets are the next state, $s'$, and reward, $r$. Once you learn a suitable model you can use it to replace your actual environment, or even to plan (i.e., query the learned simulator instead of the true env to see what would happen next.), and then use the usual model-free RL to learn a policy.
In your case I think that the most effective model-based algorithms are the ones based on either world models or dreamer. These approaches learn a generative model (e.g., a VAE) to mimic the response of the true env, and after, or at same time, also learn a control policy.
The thing you want to keep in mind is that the learned simulator is always flawed, meaning that at each timestep it will make a (small) error that cumulate with time. So there are other approaches in which the simulator is used for $K$ steps at most, or that consider the uncertainty in the predictions such that to account for the mis-modelling of the learned model.