text:
In multi-class classification, how accurate can the model be if there's class imbalance?
My dataset has essentially multi-classification problem, where I have the treatment failure (0), cure (1) and relapses (3) of patients that are associated with a series of covariates (~100 different features, SEX, BMI, HIV, RNA Signatures etc.). In this dataset, I have data 100 patients and 80 are considered cured, 10 failure and 10 relapse.
Is there a problem that I only have little data for the treatment failure? How accurate would my treatment failures predictions be if my I have so little data to train and test it?

You are right. This is a broad question, but I'll try to provide some general direction. Let's go through the whole pipeline from  data collection to model selection, training, and evaluation. These are just basic things to consider. Choose  and experiment based on your dataset. With limited data, it will be tough to generalize, but here is checklist:
1. Data collection: I'm not sure if it’s possible to collect more data in your case. If feasible, keep collecting data for the 'failure' and 'relapse' class.
2. Resampling techniques:

Undersampling "cured" and oversampling other classes to balance class distribution. (Link1, Link2).
Or use hybrid approach(both oversampling the minority class and undersampling the majority classes)

3. Loss functions:

Check out weighted loss: higher penalties for misclassifying examples from the minority classes. Thiss will encourage the model to focus on learning their characteristics.

4. Modeling: I believe Tree based models and ensemble methods tends work better on  imbalanced data.

Tree based: such as Random Forests and Gradient Boosting
Ensemble methods: combining multiple models (e.g., bagging or boosting) can improve performance and generalization.

Class-specific evaluation metrics:
Apart from the regular confusion matrix, evaluate your model's performance using metrics like precision, recall, F1-score, and AUC-ROC for each class, not just overall accuracy

Do experiments. Find the best approach for your specific data. That's the way to learn!

It would be great if you could accept the answer if it has answered your question.


text:
How big the context can be using HuggingFace models?
I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically timpal0l/mdeberta-v3-base-squad2.
As I could see in the examples, it is necessary to provide a small text (context) and a question, which the model will respond to by extracting information from the context and using training on the structure of the language. This is my working code so far:
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, AutoConfig, DefaultDataCollator
import torch

tokenizer = AutoTokenizer.from_pretrained("timpal0l/mdeberta-v3-base-squad2")
model = AutoModelForQuestionAnswering.from_pretrained("timpal0l/mdeberta-v3-base-squad2", return_dict=False )

# From portuguese: My cat is called Helena. She is fat and has spots all over her body.
little_text = "minha gata se chama helena. Ela é gorda e tem manchas pelo corpo."

# From portuguese: who is helena?
question = "quem é helena?"

inputs = tokenizer.encode_plus(question, little_text, add_special_tokens=False, return_tensors="pt")

input_ids = inputs["input_ids"].tolist()[0]

text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
answer_start_scores, answer_end_scores = model(**inputs)

answer_start = torch.argmax(
    answer_start_scores
)
answer_end = torch.argmax(answer_end_scores) + 1

answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

print(f"Question: {question}")
print(f"Answer: {answer}\n")

------------------------------------------------
Question: quem é helena?
Answer: minha gata # From portuguese: My cat

Everything went well. The model is able to perfectly answer questions about the small text provided as context.
What I couldn't understand is: when I provide a much larger text to serve as context (from a file, for example) the time for the response increases considerably. So every time I go to ask a question about the given text I have to wait a long time to get the answer.
So my question is: Am I doing this correctly? My intention is to create a chatbot to help answer questions about a specific system (a kind of help bot), so I simply copied all the text from the online help and pasted it into a single file but I believe that this method only works for small texts because at every question I must tokenizer.encode_plus(question, little_text ...
How big the context can be?

When working with large texts as context, the time for the response can indeed increase significantly. This is because the model needs to process and understand the entire context before generating an answer. The larger the context, the more time it will take for the model to encode and analyze it.
▰
To avoid reprocessing the entire context for each question, please preprocess the context separately and encode it once, storing the representation for later. Then, for each question, you can encode only the question and pass it along with the preprocessed context to the model for inference.

Can you please show where I must change my code to achieve this? Asked to GPT but the answer was a terrible mess.

@MagnoC Don't rely on ChatGPT for this kind of questions. ChatGPT is only relatively good at generating new random text that is in a way related to some text that you pass to it. ChatGPT is can be bad at providing info about facts and especially specific facts (remember that it only produces the most likely tokens and it doesn't really reason like us). Having said that, models usually have a fixed context, including ChatGPT (which should have like 4096 context length). How long is your text?

Anyway, this question seems to be specific to a specific pre-trained model, so it seems to be off-topic. See our on-topic page: https://ai.stackexchange.com/help/on-topic. We focus on the theory.

Ok. Thanks anyway. But @RevolucionforMonica gives me a solution that is generic for this KIND of network and not for this specific model because any model can be used with this code ( I think )

Context and Model Overview:
Model Capabilities: The timpal0l/mdeberta-v3-base-squad2 model, seem to be based on DeBERTa V3, utilizes relative position embeddings and is capable of processing very large sequences. While it doesn't have a strict token limit, practical considerations such as memory and processing time can impact its performance with extensive texts. Refer this issue on GitHub.
Processing Time: Longer contexts lead to increased response times due to tokenization and embedding overhead. Character-based tokenizers contribute more to this overhead compared to efficient word/subword-based tokenizers, which handle semantic units more effectively.
Strategies for efficiently handling large context:

RAG Approach: Implement Retrieval-Augmented Generation (RAG) by segmenting your large text into smaller, retrievable passages. Use a retrieval-based method to select relevant passages based on the question and apply the QA model specifically to these smaller chunks.

Compression and Optimization: Where feasible, compress or summarize the context without losing crucial information. Techniques like paraphrasing can help maintain context relevance while reducing computational load.

Context Management: Preprocess and store the tokenized representation representations of your large context to avoid repeated tokenization. This preprocessing step can significantly reduce latency for frequently asked questions.

Thanks. Seems to me RAG is the choice now I can understand it a little more. But I'm facing a dilemma about how to choose the best chunk size. If it is too big, I have performance problems. If it is too small, I loose semantics ( one chunk of text may need informations from another chunk ). So I'm learning how to partition my text efficiently

I would use overlap chunking. Try various combinations of chunk_size and overlap_chunk_size.

The Attention computation is contrains by square of the contexte. Tokenize is mandatory and avoid to tokenize / detokenize too many time can help. Your RAM or VRam is alsoo limitation as memory size is... square of the context.

Thanks, but my question was edited by @nbro and lost the meaning. I do not care how big the context can be. My question was about how can I work with big contexts without need to wait a life for the answer. The key was given by "Revolucion for Monica" with the word "preprocess"

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Can we use the capabilities of quantum computing to enhance neural networks?
Given that qubits can exist in a superposition, meaning a qubit can represent probabilities of every possible state, what if we leverage this idea in artificial neural networks? Specifically, in scenarios where we require a softmax layer for multi-class classification, could we use the power of qubits in the output layer to reduce the number of nodes required to measure the probabilities of a prediction belonging to a particular class?

There's already a discipline called quantum machine learning.
The paper Challenges and opportunities in quantum machine learning focuses on quantum neural networks and quantum deep learning (according to the abstract), but feel free to look for other resources.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
Where does the TD formula for tic-tac-toe in Sutton & Barto come from?
In section $1.5$ of the book "Reinforcement Learning: An Introduction" by Sutton and Barto they use tic-tac-toe as an example of an RL use case. They provide the following temporal difference update rule in that section:
$$
V(S_{t}) \leftarrow V(S_t) + \alpha[V(S_{t+1}) - V(S_t)] \tag{1}
$$
However, in the chapter on TD methods they state that the "simplest TD method makes the update":
$$
V(S_{t}) \leftarrow V(S_t) + \alpha[R_{t+1} + \gamma V(S_{t+1}) - V(S_t)] \tag{2}
$$
Does any one have an explanation for where $(1)$ comes from? I can't see it to be equivalent to $(2)$.
I am also wondering if anyone has a way of relating this update rule to dynamic programming. Is this supposed to be some sort of approximation of value iteration when the environments dynamics, $p(s', r | s, a)$, is unknown?

I wish I could answer this but I'm struggling with the notation a bit as well.  If gamma is 1 and the rewards are sparse (0 for non winning moves, 100 for winning, etc) I can see the similarity between the equations.

The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward is at the episodic terminating state such as tic-tac-toe. Equation (2) is a more general form applicable to both episodic and continuing tasks, considering future rewards through discounting which is the one typically used in RL TD learning methods.
As for the connection to value iteration of dynamic programming, it seems not much since value iteration update rule is based on Bellman optimality operator of the Bellman optimality equation (BOE) whose convergence is based on contraction mapping fixed point theorem, while the convergence of TD learning is based on Dvoretzky’s convergence theorem of stochastic approximation.

Thanks for the answer! Is there any similarity to DP at all? In Sutton&Barto they state in the section on DP that other methods try to mimic DP without the assumption of complete knowledge. Surely it must be more similarites than the common goal of finding an optimal policy, or?

Of course their goal to find optimal policy is similar while DP is usually model-base given your transition dynamics thus it's more like planning with precision and often better convergence, and TD is model-free method so you cannot use transition dynamics and thus more trial and error training needed.


text:
Is there an algorithm capable of telling knots and links apart?
I'd like to create some algorithm that can tell apart knots from links.

A knot is made up of one line, a link with multiple. As input, we expect images as above.
Is there an algorithm capable of classifying the images?

How many of these examples do you have?

assume as many as you need

This answer assumes that knots are one continuous path and links are multiple linked ;) paths.
The part that makes your examples tricky is that the images are 2D representations of a 3D thing, and "over/under" is implied by breaks in the path.
The computer vision algorithms that will help you are skeletonization, and analysis of skeletons. Here's a sketch of an algorithm. Steps 0 and 1 may or may not be necessary, and will depend on how variable your images are.

Preprocess your image if necessary (threshold).
Compute the skeleton of your image if necessary
Find and label connected components of skeletons (image A below)
Find tips of connected components (image B)
Say two components are the same if there are tips close enough to each other.
If there is one connected component, it's a knot, otherwise it's a link.

I ran this method on your image to produce images A and B. Here's a similar algorithm in python that looks to have similar functionality.
Step 2

Step 3

Step 4

Huge thanks! What a great answer, insightful, informative. :)

Glad you found it helpful :D


text:
Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment?
Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment? For example a function G(s) where it takes in the state as input and outputs an action based on it? This would help when the expert doesn't know how to define the actions right?

Are you asking if the network can from it’s environment determine a list of actions automatically and take one?

Sort of, like based on the environment it generates an action without needing the human to define any actions (eg. Moving right or left, velocity values, etc.)

I'm not sure I fully understand the use but I'll hazard at something real quick and see if it helps.
If the agent has a perfect knowledge of environment dynamics then the policy can deal with states instead of actions.  The Agent in $S_1$, knowing it has transitions available for it to $\{S_2, S_3, S_4\}$, can determine it would like to transition to one of connecting states irrespective of the underlying action to get there.  This would be the $V_\pi(s)$ value iteration scheme.
In many real world cases the dynamics of the environment are not known (or not known exactly) so the agent cannot directly pick it's next state.  But the actions are often known and so the agent can repeatedly sample from a list of actions in states to get a feeling of the quality of an action in a state ($Q_\pi(s,a)$).  There is no need for every action to have an effect mind you.  If an action in a state is "unsupported" it can simply lead back to the same state.  The environment must map an action to a transition but the human doesn't need to define what they mean.

Thank you, this was a good answer, but I do have 1 question, if for example the current state is separated into the input and output state, and the output state remains inside the environment dynamics without being coded into the observation state, if ideally the value iteration scheme model has pefect knowledge of the environment dynamics, does that mean it knows to what state it should transition the input state to make the output state == goal state / max reward?

Value iteration is the process by which we estimate the value of being in a state (based on the reward for getting there and discounted rewards of any subsequent states).  The policy which arises from value iteration is that for every state we choose the next state which has the maximum total reward.  Assuming our estimations of all states are near perfect, this policy can be followed from starting state to terminal in a path which gives max average reward.

So if ideally the true optimal values are known / perfect knowledge, even if the reward structure is sparse (recieves -1 for every state when goal isn't reached, +10 if goal state is reached), it will always know the path to the goal state in the policy value iteration function?

Yes, so under value iteration the value of the state is the reward for reaching that state plus the discounted reward of next state(s).  Lets say the discount rate is .50.  If there is a -1 reward for the current state and 10 for the next, the value of the state is 4.  The state leading to this state would have a value of 1.  Compare this to a path which does not lead to a reward of 10 which would just be sums of discounted -1s and thus have lower value than the one that does.  It forms a sort of pheromone trail to a goal.

Oh I understand now, so basically if the policy value iteration function knows the true optimal values, it can find the optimal path since it already "knows" it.

Hey uh actually can we get in touch? It's fine if you don't want to, I'm having some problems with my DQN model and I'm afraid if I post in AI StackExchange it's gonna get taken down for whatever reason, or i'm gonna get downvoted to oblivion

Yea, AI StackExchange is for theory not programming.  I'd suggest posting it on StackOverflow with the correct tags.  I'm notoriously bad at reading other people's code so even if I can't help maybe someone else can.  Link it here I guess at the risk of it being marked as "discussion" but I should see it.

I was actually thinking about on a platform like Discord maybe? Yeah, I got a bad reputation there for apparently posting questions not related to topic

Oh yeah and 1 question more, would the output layer of policy value iteration function in a Neural Network be continous or discrete? Would I have to set up individual states, or can it be a continous range of states?

Hey you still there? I'm sorry if I went too far, I can settle in with StackOverflow if you're not comfortable with Discord

I don’t know that I remember my discord login.  There is an RL community on there IIRC.

DQN + continuous output isn’t common.  The more common approach is a policy gradient method like Actor Critic, A3C, PPO, etc.

I mean like for the policy value iteration function, not a DQN

And ok then, how about you create a new Discord account?


text:
Is there an algorithm that produces a uniform distribution over the set of trajectories with maximum reward sum?
I am not an expert in reinforcement learning. I am applying it to my field of study.
I am training a model such that given a state, it predicts the probability of taking an action for every action possible. As such the model can be described as M: S -> P where P is the set of all probability vectors over A, the discrete action space. S is the set of all possible discrete states.
The model takes actions stochastically to create a finite trajectory (I do not choose the action with maximum probability, instead, I sample trajectories). Multiple trajectories have the same reward (summed over the state-action units in the trajectory).
Define G to be the set of trajectories with maximum reward sum.
If I train such a model using any mainstream algorithm (Q-learning, actor-critic, etc.) the model will learn, loosely speaking, to maximize the probability of a trajectory from G. It will not learn a uniform distribution over the set G. Instead, probability will concentrate on a specific trajectory. This is not an exploration problem as far as I know, since even if the model explores every possible trajectory, probability will still concentrate on one trajectory. And it is not inherently a problem in reinforcement learning, it is specific to my use case.
I would very much appreciate any references that tackle this problem. I am searching for an algorithm that produces a uniform distribution over the set G and a negligible probability for all other trajectories.

Are you attempting to build a model which finds all (or many) optimal policies in an environment?

@foreverska yes, in a sense. All these optimal policies should be summarized by one model that, in a sense, samples uniformly from one of them.

Q-learning will indeed learn a trajectory, as they are all equivalent, and usually the resulting policy of a Q-learning algorithm is a greedy policy, which selects:
$$
a = argmax_a Q(s,a)
$$
so, no matter what you do, the final policy in your case has to be deterministic.
In the actor critic scenario, this is not obvious. Indeed, if two actions $a_0$ and $a_1$ lead to trajectories with the same expected return, when you update the actor using:
$$
\nabla J(\theta) \approx \mathbb{E}[G_t \nabla_\theta \ln \pi(a|s)]
$$
you can clearly see that both the actions will be reinforced equally
However, you have to consider that the overall objective of RL is to maximise:
$$
\mathbb{E}[\sum \gamma^tr_t]
$$
In order to that to happen, going back to the previous scenario, any linear combination between the policy that always selects $a_0$ and the one that always selects $a_1$, will be an optimal policy, as the all lead to the same solution (solution meant as expected sum of rewards)
A very simple example of this, is by considering the trivial case where you have 2 bandits (think about slot machines) that give the same expected return $C$: your policy return will be $V = p(a_0)C + p(a_1)C$, clearly, any combination of the two probability lead to the same value (obviously having $p(a_0)C > 0, p(a_1) > 0, p(a_0) + p(a_1) = 1$)
However, luckily for you, there is an easy fix if you want the policy that not only maximizes the expected sum of rewards, but it's also as stochastic as possible, that is to introduce an entropy bonus
$$
\nabla J(\theta) \approx \mathbb{E}[G_t \nabla_\theta \ln \pi(a|s)] + \mathcal{H}(\pi)
$$
And this solution, will select the policy $\pi$ that also maximises the entropy, thus making it as stochastic as possible, thus distributing equally the probability mass across actions that lead to trajectories with the same expected return

Thank you. Your answer led me to this research paper that addresses my problem perfectly. For future readers: https://arxiv.org/pdf/1812.05905.pdf

@Alberto: Your last paragraph is wrong. Adding an entropy term destroys the optimality and the policy you get would generally not optimize expected return. Whether this is a good heuristic for the specific use-case or not is a totally separate question, but you cannot claim optimality. CC @moeasal

I think that the entropy bonus needs to be weighted carefully, and potentially down over time. It may be useful initially to discover different equivalent trajectories, but it will also cause a preference towards action distributions that are not optimal when the entropy bonus is close to scale of the rest of the objective. Also, are you sure adding this bonus to the gradient works? I think that's equivalent to maximising the squared entropy measure - probably ok, but a little odd

The entropy measure is probably some log sum over the probability distribution anyway, so this weirdly ends up looking a bit like L2 regularisation

@NeilSlater yes it's the sum of log terms, though I'm not sure how you can end up with an L2 regularization...

I mean it has the "rough feel" of L2 weight regularisation, except due to positive direction is about  largest distribution (as opposed to smallest individual numbers), and also of course it is not weights being impacted but output probabilities. And like regularisation, it's a balance between two potentially opposing objectives

@NeilSlater i mean i hardly see appearing any squared term, but you can see it as something like adding $-||p(a)-1/N||^2$, thus giving a bonus the closer you are to to uniform ($N$ being the number of actions you can perform)

@user21820 i agree with you that you are adding a small bias, however you can pretty clearly interpret it as a Lagrangian of the constrained problem which maximizes the entropy s.t. the policy value is the maximum

@NeilSlater sorry, only now i saw your first comment… yes it works and it has been done many times, for example in the paper of SAC (though sure, it needs a scalar carefully chosen)

The very fact that you need a constant weight on the bonus term already tells you that it cannot be correct. The closer the weight to zero, the closer to optimality. You can't have both optimality and preference for stochastic trajectories! Sure, we may want some trade-off, but you need to be clear about that.

@user21820 not sure how you can be so sure about it: assume we find the optimal policy without entropy with value $V^$, then problem $\max \mathcal{H}(\pi) s.t. V(\pi) = V^$ is for sure well defined, AND feasible (though maybe intractable) AND the solution is that the OP is looking for, but the "the fact that you need a constant weight on the bonus term already tells you that it cannot be correct" goes 100% against the concept of Lagrangian (Lagrange multipliers) in constrained optimization, so I'll suggest you to give it a look :)

@user21820 after looking at that, I then suggest you take a look at the paper of PPO, where they use the Lagrange multiplier form of the loss when talking about the KL version of the objective, and show that with a dual descent mechanism,e you can be very close (note that the "constant" work in this comment section has been used only by you, I've never said that it needs a fixed value)

I know the Lagrange multiplier technique, which is not at all the same as what you are doing. Optimizing entropy after optimizing the return value of course can be done, but you cannot do both simultaneously by just arbitrarily adding two terms. You are misunderstanding my use of the word "constant"; what I said was exactly in line with what @NeilSlater said, namely that you need to weight the two terms and gradually adjust the weights. You also agreed that you need 'carefully chosen' weights, and "very close" ≢ "optimal". My point is just that as written in your post it cannot be optimal.

@user21820 https://arxiv.org/pdf/1801.01290.pdf proof B.1... i would say that is pretty close...

I am not strictly convinced that Q-learning should necessarily learn a singular optimal path.  Given infinite exploration, Q-learning should estimate the reward of all actions.  In a state where two actions have the same reward the tie should be broken randomly (giving all equal paths the same probability of being followed).  The remaining actions would have zero probability of being followed given a perfectly deterministic maximizing agent.
In practice we do not explore infinitely or our estimators do not learn infinitely.  This might account for the perception that it finds a singular near optimal path.  Often the problem space is too large and our estimators too opaque to inspect what has actually been learned.  I would challenge to experiment with tabular Q-learning on a multi-armed bandit where more than one arm has equivalent rewards.
So I guess my answer would be that the issue is unlikely to be with the learning method but with the learning hyper-parameters (learning rate, epsilon annealing rate, etc).

thank you so much for the insight. would you say the same about actor-critic? and does this problem have an official name so that i can find literature discussing it?


text:
How to perform inference pn a mixture of Gaussian models?
Let's say you have a model, which consists of two Gaussian graphs, which is great for making predictions on multi-modal data. With this you can create the following probability function for each y.
$
    Pr(y \mid \lambda, \mu_1, \mu_2, \sigma_1^2, \sigma_2^2) =  \frac{\lambda}{\sqrt{2\pi \sigma_1^2}} \exp \left[ -\frac{(y - \mu_1)^2}{2\sigma_1^2} \right]
     + \frac{1 - \lambda}{\sqrt{2\pi \sigma_2^2}} \exp \left[ -\frac{(y - \mu_2)^2}{2\sigma_2^2} \right],
$
In this case you will need a model, which predicts $\theta = \{λ, μ^1, σ^2_1, μ^2, σ^2_2\}$.
This will result in the following loss function, when you use the negative log-likelihood.
$L = -\sum_{i=1}^I\log\left[\frac{sig[f_1[x_i, \phi]]}{\sqrt{2\pi f_3[x_i, \phi]^2}} \exp \left[ -\frac{(y_i - f_2[x_i, \phi])^2}{2f_3[x_i, \phi]^2} \right]
     + \frac{1 - sig[f_1[x_i, \phi]]}{\sqrt{2\pi f_5[x_i, \phi]^2}} \exp \left[ -\frac{(y_i - f_4[x_i, \phi])^2}{2f_5[x_i, \phi]^2} \right]\right]$
In the book "Understanding Deep Learning" it was mentioned that making inference on such a model is difficult, as there is no closed form for that and you would need an iterative approach. But I'm very confused about that statement, since I thought that the best y value, for a given set of parameters is always the most likely. Therefore I would think that you could simply do the following:
$
\hat{y} =  \underset{\mu}{\text{argmax}}\left[Pr(y \mid \lambda, \mu_1, \mu_2, \sigma_1^2, \sigma_2^2)\right]
$
Because normally if we would use one normal distribution to approximate the probability for y, we simply can use the predicted $\mu$ from the $f[x, \phi]$ function
In the example above I simply want to figure out which $\mu$ value leads to the highest probability. My questions is why I can't simply do this.
Is it because when we have two Gaussian distributions where the $\mu$ of these is close to each other, that the maximum probability is not given by neither of those.

For a mixture of N Gaussians, the mode (aka the point with the highest likelihood) is either in a mode or connected to a mode (referring to the ones of the single Gaussian distributions mode are aggregating for the mixture).
Given this observation, you just need to initialize N points in the modes of the models, run Gradient Ascent on them, and check at the end which point is the one with the highest likelihood

Here you can see that even if the $\mu$ values are very close that the position at which y can be maximized is not one of the available $\mu's$.
You can see clearly for a normal distribution that the position at which the highest probability $\mu$ is.
$\min[(y - \mu)^2] = 0$
Here you can see directly that for $\mu = y$ the probability can be maximized, if $f[x, \phi] = \mu = y$
Unfortunately the same does not apply for the mixture of gaussian model, since there is no close form that describes that parameter.
Therefore for more complex gaussian mixtures, with more components you may need an iterative approach to find the y that brings the highest probability.


text:
How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm:

How are these terms equivalent, and how did $s$ get replaced with $S_t$?
As far as I understand, the first term is the the marginal expectation of the function

over $s$, and hence can be written like the second term.
Also, I'm assuming the proportionality constant gets absorbed in the step size and hence we get rid of the proportionality sign.

re: the proportionality becoming an equals, you should read like this: the gradient is proportional to the RHS on the top line, and this is in turn equal to the RHS on the second line. So, the gradient is still proportional to the RHS on the second line, the equals is just meaning that it is equal to the top line RHS. To see why it is equal, just not that we are just re-writing the sum as an explicit expectation (which it is, because $\mu(s)$ is a distribution).

Your intuition is correct as in your reference just above this equation they explain as:

Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $\pi$; if $\pi$ is followed, then states will be encountered in these proportions.

And the definition of the distribution $\mu(s)$ is also defined at the bottom of the previous page of your quoted section.

Thanks for the explanation. But what is S_t here and why did we substitute s in that expression?

The added subscript $t$ is to express the parameter update rule of the next equation (13.7) in the same page at any generic time step. There's exact definition linking rv $S_t$ and any generic state $s$ in the beginning of this chapter $\pi(a | s,\mathbf{\theta})=\text{Pr(}A_t =a | S_t =s, \mathbf{\theta}_t =\mathbf{\theta})$. Hope this clarifies your remaining confusion.

In general, suppose we have a discrete random variable $X$. The expectation of $X$ is defined as
$$\mathbb{E}[X] = \sum_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$
where $\mathcal{X}$ is the support of $X$.
Given that $\mu(s)$ is the state distribution, it is equivalent to say that
$$\mathbb{E}\left[\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right] = \sum_s \mu(s) \left(\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right) \; ;$$
since $\mu(s)$ is the state distribution and $S_t$ is used to denote the state as a random variable.


text:
Why is this RL derivation right?
This comes from the paper, Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review.
I don't know the why the following derivation is true. The paper only briefly explains this.
Can someone help with a detailed deducing process?

The paper only briefly explains the quoted derivation is mainly because it's only standard application of the policy gradient theorem (PGT) for the proposed maximum entropy RL here where you can move the gradient operator inside the expectation statistic acting on the log policy distribution $q_{\theta}$. The only difference with the standard PGT is that here the reward term at each time step is added to a $\theta$-dependent Shannon entropy term, thus apparently you have to take additional gradient of the entropy term.
Framed this way the innermost summation terms including $-1$ of the second step is just the expansion from the RHS of the first step based upon entropy definition $\mathcal{H}$ for the policy and taking derivative of $\mathcal{H}$ wrt $\theta$ as also explained by the authors immediately after your quoted equations, where please note they omitted parentheses. They also explained the last line follows by noting that the gradient estimator is invariant to additive state dependent constants at each time step $t'$, and thus replacing $−1$ with a state-dependent baseline $b(s_{t′})$, which is very similar to the standard REINFORCE with Baseline method for reducing variance during Monte Carlo sampling.


text:
Does Machine Learning focus on discriminative AI while Deep Learning also focus on generative AI?
I know that Deep Learning is subset of Machine learning
But is it correct that classical ML algorithms mainly focus on implementing Discriminative AI while DL algorithms implement both Generative AI and Discriminative AI?

Today you subtly changed your question body from 'only' to 'mainly' about classical ML's focus on Discriminative AI. As you correctly listed before most known methods in any classical ML textbook are based on discriminative models such as svm, knn, logistic/linear/Poisson regressions, etc, thus apparently classical ML models mainly focus on discriminative models, compared to the very few classical generative models such as naive Bayes, GMMs, HMMs as I already mentioned in my yesterday's answer.

It's not accurate to say that classical machine learning implements only discriminative AI while deep learning implements both generative and discriminative AI. Both classical machine learning and deep learning have methods for both generative and discriminative tasks. For example, the classical naive Bayes for classification and some NLP tasks and GMMs used for clustering and density estimation are typically considered as a generative model.

In the case of discrete inputs (indicator or frequency features for discrete events), naive Bayes classifiers form a generative-discriminative pair with multinomial logistic regression classifiers: each naive Bayes classifier can be considered a way of fitting a probability model that optimizes the joint likelihood ${p(C,\mathbf {x} )}$, while logistic regression fits the same probability model to optimize the conditional $p(C\mid \mathbf {x} )$.[17]... Discriminative classifiers have lower asymptotic error than generative ones; however, research by Ng and Jordan has shown that in some practical cases naive Bayes can outperform logistic regression because it reaches its asymptotic error faster.


text:
If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?
I'm working with a fully connected neural network with input 32x32x3.
The  architecture includes a dense layer 32 + ReLu activation, then another dense layer 64 + ReLu Activation,   followed by a dense layer 32 + ReLu Activation,  and finally, a dense layer of 10 neurons with softmax activation
I have a homework question:

If we had to choose between Uniform(0,1) and Uniform(-1,0), which one would you expect to work best and why?

After searching the internet and using ChatGPT, I have reached different conclusions that are not consistent:
1-  All my inputs are positive, and if I choose Uniform(0,1), my z = W.X + bias will be >0, causing my ReLU function to be linear. Therefore, Uniform(0,1) is considered worse.
2-  All my inputs are positive, and if I choose Uniform(-1,0), my z < 0, leading to most of my neurons being deactivated after using ReLU. Hence, Uniform(-1,0) is considered worse.
I don't know which one is correct, or if either of them is correct, and I would like help to clarify this doubt.

It is not the input to first layer you need to worry about, but the output from the hidden layer to the next layer.
No matter how the inputs and weights are arranged, after passing through ReLU in the first layer then all outputs will be zero or positive. Then in the following layer, if all weights and biases are drawn from a negative-only distribution, all sums will be below $0$ and all ReLU activations will be $0$. From that second layer onwards, no useful data is propagated, and backpropagation will return $0$ gradients, meaning the network is stuck in that state.
So the negative-only weights distribution will break a ReLU-based multi-layer network.
The positive-only network will start like a linear-only network as you suggest, but incorrect values will still work with back-propagation, because there is a non-zero gradient, and the network can learn. It should start to learn where to use negative weights.


text:
What is the name of the reward function that utilizes the rewards of the next n steps?
I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a simulated car that has to drive through a series of goals without collisions.
I read chapter 7 of the Reinforcement Learning Book by Sutton and Barto: http://incompleteideas.net/book/the-book-2nd.html
In chapter 7 they describe n-step bootstrapping which uses the rewards of the next n-steps and a value estimate of the nth step to update the value of the current step t. This describes extensions to reinforcement learning algorithms, I would like to use an approach inspired by the n-step bootstrapping.
I would like to use a related approach to smooth my reward function. Unfortunately I am not sure how that approach is called and if it has already been talked about in research.
I am searching a name or some research regarding the following reward function:

Reward(st, at) is simply the reward encountered at timestep t. I would then hand the SmoothedReward to the RL algorithm (PPO).
Can you please direct me to research about something like my smoothed reward function?

n-step bootstrapping is a way to approximate the expected return, and has the theoretical reasoning of being an estimate (combination of sampling and stats sumary) of the true value. What do you intend your "smoothed reward" being used to substitute (are you simply using your reward sum instead of the reward), and what is your thinking for justifying its use in PPO?

I would hand this smoothed reward to the PPO training algorithm instead of the simple Reward(s_t,a_t).
(Yes simple this discounted reward sum instead of the standard reward)
I think this could be useful in my problem since rewards that are caused by the current iteration might only be awarded a bit later.

Your function could be called the truncated return - i.e. the sum of rewards up to some time step in the future.
It would be unusual to perform reward shaping by taking the orginal reward from an environment and replacing it with the truncated return. It definitely breaks some things, so is not a general solution. The broken things include:

Off-policy value estimations. The truncated returns would include results from future decisions that the behavior policy made, but the target policy would not make. By "squashing" the return into the new observed reward, you would prevent use of any approach, such as importance sampling, to correct for this mismatch.
An environment with important rewards before step $n$. These early rewards will be under-represented in the new full returns (i.e. returns calculated on your transformed environment), and this could change which early actions were optimal, changing the environment so that the new agent that is optimal in your reward-shaped environment would make different decisions to the optimal agent in the original environment.

In comments you would use the calculated truncated return from the original environment as the reward signal observed by the PPO algorithm (effectively in a transformed or reward-shaped version of the environment). This is not necessary, because PPO already calculates and uses estimated returns from the observed rewards. This is true for all RL algorithms - they all take the observed return, or calculate an approximate expected return, or sometimes a value derived from the return, such as the advantage function. PPO uses a variation of the advantage function.
Effectively, other that the above two problem cases above, what you would be doing for most on-policy algorithms is multiplying each reward by a factor $\sum_{k=0}^n \gamma^k$, because the same immediate reward will be associated with every state, action pair that came before it, going back n steps. So, this should still work, but I don't think it will transform a problem to make it easier to solve for the agent. It is not much different, if you avoid the problem scenarios, from changing the learning rate.
Some RL agents do use the truncated return internally. It can be used for eligibility traces for example.

Thanks for the explanation. I see your point explaining that this smoothing is not necessary for any RL algorithms, I just thought it would help.
Thanks for the remark concerning off-policy value estimation, I didn't think of that.


text:
Do different camera angles affect the performance of the deep learning model?
I'm working on a project to build a face recognition system and I have a question: do different camera angles affect the performance of the deep learning model?
For example, in CCTV, training data and testing data are taken using different camera heights (assume, for training data, the camera height is 150 cm and, for testing data, the camera height is 200 cm, so the camera angles also can be different). Is this possible to implement? Because when I tested the model with testing data, the results were not that good and it always had false recognition.

Yes, changing camera height can affect the performance of models in applications like face recognition. Because it will change the face-capturing angle and size of the face captured. But do not worry, it is a common issue which we face on a regular basis.
To resolve, you need to do some work mentioned below:

The main thing for face recognition is face detection. For the face detection model, you need to do augmentations like angles, lighting, resizing, and rotation.

You also need to use a model to extract a few points of the face like both eyes, both the start and end points, and nose points. Based on that data, you need to do a warp affine to match the ideal face. Thus, all the data is in normalization format.

Choose a neural network like arc face which can work with pose invariant features with angular marginal loss.

Keypoints are augmentation of the data, as the preprocessing step adds warp affine for face normalization and uses a model that can handle pose invariant features.

Okay, thank you for the explanation. One more question. If I want to build the model for face recognition using this methods, without using another model to extract few point of the faces, just label them manually using LabelImg, is it still possible to do it? @Hiren

Yes, @Naay you can use model architectures like MTCNN, blazeface or retinaface similar. The method you mentioned is used for Detection only you need to use multi-tasking Vision Models as I mentioned.


text:
In the PPO objective function, does the expectation do a weighted average over the batch?
I am currently working on implementing Proximal Policy Optimization (PPO) for a reinforcement learning task, and I have a few questions regarding the objective function and probability calculations.

In the PPO objective function, $$L(\theta) = E\left[\min \left(r(\theta) \cdot A, \text{clip}(r(\theta), 1 - \epsilon, 1 + \epsilon) \cdot A \right)\right]$$ does the expectation $E$ do a weighted average over the batch? If so, how are these weights determined?

When calculating the ratio function $$r(\theta) = \frac{\pi_{\text{new}}(a|s)}{\pi_{\text{old}}(a|s)}$$ would $r(\theta)$ be a single value or a whole array with probability distribution length?

I appreciate any clarification on these points, as it would greatly help me in understanding the nuances of PPO and implementing it effectively.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

One question per post. Split this post into multiple ones.

Sorry for putting all my questions in one post.
After careful searching, this is the answer to all the questions:

The expectation notation means mean average.
$r(\theta)$ does element-wise division of new policy and old policy probability distribution.


text:
Why do we subtract logsumexp from the outputs of this neural network?
I'm trying to understand this tutorial for Jax.
Here's an excerpt. It's for a neural net that is designed to classify MNIST images:
from jax.scipy.special import logsumexp

def relu(x):
  return jnp.maximum(0, x)

def predict(params, image):
  # per-example predictions
  activations = image
  for w, b in params[:-1]:
    outputs = jnp.dot(w, activations) + b
    activations = relu(outputs)

  final_w, final_b = params[-1]
  logits = jnp.dot(final_w, activations) + final_b
  return logits - logsumexp(logits)

I don't understand why they would subtract a constant value from all the final predictions, given that the only thing that matters is their relative values.
For clarity, here's the loss function:
def loss(params, images, targets):
  preds = batched_predict(params, images)
  return -jnp.mean(preds * targets)

Can you please provide more context? What task are they trying to solve? Which loss function? Which dataset? Ideally, questions here should not require code. Code can be provided as an additional source of info, but note that programming issues are off-topic. I don't think your question is a programming issue, but, like I just said, it lacks details.

Though I may be answering this question quite late, I would like to address your inquiry (because it's possible that other individuals may also have similar queries. :D)
You can think of the last 3 lines of the function predict as implementing a softmax layer.
In this case, the function returns the log probability rather than the probability.

To be specific, logits stores $z_i=\ln{e^{z_i}}$ and logsumexp(...) computes the $\ln\sum_{j=1}^{K}{e^{z_j}}$. The returned value is thus $\ln{z_i} - \ln\sum_{j=1}^{K}{e^{z_j}} = \ln{ \frac{e^{z_i}}{\sum_{j=1}^{K}{e^{z_j}}}}$.

It's apparently for numerical stability. From the Wikipedia page for LogSumExp:

A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems when very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precision floating point numbers.

And this answer from stats.stackexchange.com:

This is a simple trick to improve the numerical stability. As you probably know, exponential function grows very fast, and so does the magnitude of any numerical errors. This trick is based on [...]

That said, I'm not sure why it's required in your example because, unless I'm missing something, there is no exponential function which could introduce the aforementioned numerical instability.
I guess it's possible that it's a mistake in the JAX docs, but it seems more likely that I just don't understand it. Hopefully someone else will be able to comment/answer to provide an explanation. I figured I'd post this half-answer since there are no others answers here.


text:
Why do we do need compression in Semantic Segmentation?
When doing semantic segmentation, we often make use of FCN, which can be thought of in two parts: an encoder and decoder. As I understand, the encoder compresses the image into a spatially small, but high number of channels. The decoder then uses this high channel activation map and upsamples it into a representation detailing the class for each pixel in the image.
My question is, why do we do this spatial compression at all?
For example, why would the architecture shown below be a bad choice:

There is never a 100% accurate theory, however it's been observed to be beneficial, however I would argue that is due to the following:

you want to have a latent dimension, to learn the manifold compressed version, so that the decoder has a harder life to overfit (kinda speculative)
when doing convolution, you can keep the height and width dimensions constant, by introducing artifacts (which obviously you don't really want), so you are forced to decrease the dimensionality in H and W... so as a trade off, you just compensate with the number of filters/channels

in other words, there is no correct answer, just empirical evidence that this "latent compressed version" works better in some cases


text:
Can I use Alpha-Beta Pruning for real life game simulation?
I had an idea to use the this algorithm to simulate real life game scenarios where a player could train and retrain their mistake and be ready for when that situation happens again in real life.
This simulation would calculate every possible scenario and give you the best one.
This would work almost the same way as it is used on Stockfish for chess. It calculates every possible scenario and always makes the right move.
Now my question is, would that be possible? Is there another algorithm/AI/Software that could make this simulation work?

It also depends on the type of game.
The problem with Go is that a board that looks really good can turn into a disaster on the next move.
Games that have easy evaluations and experience only small incremental changes with each move are much better suited for alpha-beta pruning.
For instance, Checkers could be trivially evaluated as the difference in number of pieces each player has.
Most moves will change the value by 0 or 1 points, with double jumps being much less common, and higher numbers being rare.
Not going just one level deeper is usually not likely to be a significant problem.

Checkers could be trivially evaluated as the difference in number of pieces each player has that's a pretty greedy policy... you probably need something more than that

@Alberto, I said it "could" be, not because it would be the best way, but because calculating the value would be very efficient. For Go, there is no evaluation even remotely that efficient, which illustrates the fundamental difference that makes alpha-beta appropriate for one but not for the other.

You can do that, however you're probably going to hit the wall that alpha-beta pruning got with Go and large scale games... it does not scale too well
Depending on what's your definition of when that situation happens again in real life you might need very large and deep trees to get a good estimate, let alone that to work well you have to have a good bounding algorithm, which is not trivial to obtain
As Neil, you probably also need to develop a very sophisticated and complex model to simulate your scenario

For "real life", the OP would also need a rather sophisticated predictive model for exploring "every possible scenario".


text:
Why is it called multi-headed attention?
Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the calculation in one go and then the result is multiplied by another matrix Wo to get the dimensions required?
My understanding of self-attention calculation is based on the following blog: https://jalammar.github.io/illustrated-transformer/

The original paper "Attention is all you need" mentions the following.
"
Instead of performing a single attention function with $ d_{model} $-dimensional keys, values and queries,
we found it beneficial to linearly project the queries, keys and values $h$ times with different, learned
linear projections to $d_k$, $d_k$ and $d_v$ dimensions, respectively. "
What is single-headed attention?
The input matrix $ X $ of dimension $ T \times d_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d_{model} \times d_{model} $ to give output of dimension $ T \times d_{model} $ Now, while calculating attention  $ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $yields one attention matrix of dimension  $ T \times T $.
What is multi-headed attention?
The input matrix $ X $ of dimension $ T \times d_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d_{model} \times d_{k} $ to give output of dimension $ T \times d_{k} $. Note that $d_{model} = h \times d_{k} $ where $ h $ is the number of heads. Now, while calculating attention  $ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $yields $h$ attention matrices of dimension  $ T \times T $.
It is called multi-headed attention because each head produces one different attention matrix.
(I feel your confusion may have come from the knowledge that matrix multiplication happens in blocks. So what single-headed attention does in  $ d_{model} \times d_{model} $ matrix, multi-headed attention does in $ h $ numbers of $ d_{model} \times d_{k} $ matrices. Why not make one single $ d_{model} \times d_{model} $  matrix from these $ h $ numbers of $ d_{model} \times d_{k} $ matrices? and proceed as in single-headed attention. The only problem with that argument is we end up with one attention matrix instead  $h$ attention matrices. )
These $ h $ attention matrices help in modeling different relationships between words at different positions. The original paper mentions
"Multi-head attention allows the model to jointly attend to information from different representation
subspaces at different positions. With a single attention head, averaging inhibits this."

I meant that single headed attention is in concept only and in actual implementation the Q, K, V blocks for a single layer is just a single matrix. For example if single Q matrix is 10x8 and there are 8 heads then the actual Q matrix is just 10x64. Same for K and V and the forward operations are applied. Then in the end we have seq_lenx64 which is multiplied by 64x8 to get the final dimensions correct. So why isn't it just considered a single weight instead of multi-headed?

Work out the dimensions of your attention matrix. Imagine you have 10x8 Q,K, V matrices. Then dimension of your attention matrix is arrived as ($ KQ^T $ ) (10 x 8) x (8x10) = 10 x 10. That is the size of your attention matrix for a single head. Do it for 8 heads and you get 8 numbers of 10 x 10 attention matrices. Now , because these 10 x8 matrices can be treated as blocks of a single matrix, you have one 10 x 64 matrix for K. Same applies for Q and V. Now work out the dimensions of your attention matrix (10 x64) x (10 x 64) = 10 x 10. You get only one attention matrix of dimension 10 x10.

Each attention matrix is used to generate a part of the output in multi-headed attention. All such outputs are concatenated to produce the final output. This is not the same output as one full sized output generated using one attention matrix in single headed case. So both in actual implementation and concept, multi-headed attention is different from single-headed attention.


text:
What is the matrix representation of a dilated convolution?
I am delving into the matrix representation of dilated convolutions, especially after understanding standard 1D convolutions as Toeplitz matrix-vector multiplications.
My specific focus is on Dilated Convolution. Based on my initial understanding, a dilated convolution involves introducing "holes" or zeros between elements in the convolution process.
Therefore, would a matrix representation of a dilated convolution simply be akin to the standard convolution matrix but with zeros inserted between the non-zero elements in each row?
Any insights or mathematical references regarding this would be highly valuable to me.

I think they use sparse methods for multiplication, so instead of having actual zeros, they just change which elements they multiply together.  They claim it is same compute cost, and the only way for that is not to do zero-multiplications.  I think it works a little like striding.

Well, I don't know what insights you are looking for exactly, your understanding sounds very solid and you are correct that there will be zeros between the kernel elements in the matrix. Maybe its interesting for you how to construct such a matrix: We start from what we have and what we want to achieve:
We have some 1D data $\mathbf{a}$ and a kernel $\mathbf{k}$:
$$
\mathbf{a} = [a_0, ..., a_i, ...a_T ], \mathbf{k} = [k_0, k_1]
$$
We want the following result for dilation factor 1 (no dilation):
$$
\mathbf{a} * \mathbf{k} = \begin{bmatrix}
a_0 k_0 + a_1 k_1 \\
a_1 k_0 + a_2 k_1 \\
a_2 k_0 + a_3 k_1 \\
a_3 k_0 + a_4 k_1 \\
a_4 k_0 + a_5 k_1 \\
...
\end{bmatrix}
$$
for dilation factor 2, we would reduce the sampling rate of the model by skipping elements. The end result should therefore look like this (with a stride of 1):
$$
\mathbf{a} * \mathbf{k} = \begin{bmatrix}
a_0 k_0 + a_2 k_1 \\
a_1 k_0 + a_3 k_1 \\
a_2 k_0 + a_4 k_1 \\
a_3 k_0 + a_5 k_1 \\
a_4 k_0 + a_6 k_1 \\
...
\end{bmatrix}
$$
It's straight forward what the matrix has to look like when you start from the desired result. By first writing $\mathbf{a}$ as a column vector and then designing the matrix row by row, the computation written out would then be:
$$
\mathbf{a} * \mathbf{k} = \begin{bmatrix}
k_0 & 0 & k_1 &  & \dots & & 0\\
  & k_0 & 0 & k_1 &   &   &   \\
\vdots  &  & k_0 & 0 & k_1 & &  \vdots \\
  &  &  & k_0 & 0 & k_1 &   \\
0 &  & \dots &  & k_0 & 0 & k_1 & \\
& & & & & & ...
\end{bmatrix}
\begin{bmatrix}
a_0 \\
a_1 \\
a_2 \\
a_3 \\
a_4 \\
...\end{bmatrix} = \begin{bmatrix}
a_0 k_0 + a_2 k_1 \\
a_1 k_0 + a_3 k_1 \\
a_2 k_0 + a_4 k_1 \\
a_3 k_0 + a_5 k_1 \\
a_4 k_0 + a_6 k_1 \\
...
\end{bmatrix}
$$
Clearly you are right, that there are zeros in between the kernel elements $k_i$, i.e. we introduce a zero diagonal into the matrix. The number of such diagonals corresponds exactly to the dilation rate. To make the stride of the convolution other than 1, lets call it $s$, simply shift the $i^{th}$ row by $(s-1) \cdot i$ zeros to the right.


text:
Do full-text translators such as DeepL or Google Translate fall under the term "Generative AI"?
My question relates to full-text translators that are not specifically based on LLMs. My current understanding is that the term Generative AI goes beyond LLMs and that the full-text translators (especially those which are based on artificial neural networks) also fall into this category.
In the Wikipedia article about Generative AI, I could only find the statement that generative AI systems such as ChatGPT are also used for translations. That is quite obvious, but this does not answer the question of whether other full text translators are also commonly referred to as Generative AI.
IBM research defines Generative AI as

Generative AI refers to deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on.

AFAIK DeepL, Google Translate, Bing Translate but also a few not so well-known systems falls clearly into that definition.
My question was triggered by an answer on Meta Stack Overflow that contained the following sentence:

Besides, the banner clearly states: "Answers generated by artificial intelligence tools". Translations aren't generated answers. They're translations.

with the implicit conclusion that it is clear to everyone that pure translators do not count as generative AI, hence it does not need any further explanation. Other participants in that thread seem to agree to that point of view. However, I think their use of terminology is not the typical use in the field of AI, and I would like to hear what the experts say.

I found a duplicate on GenAI SE that answers this.

I found this interesting link about the history of Generative AI. Translators apart from LLMs are not explicitly mentioned. Still it gives a good outline about the >70 years old history of the field, the  breakthroughs since 2010, and the wide range of tools for which the term GenAI is used.

Generative AI, as defined by IBM research, refers to deep-learning models capable of creating new content, be it text, images, or other media, based on their training data. This definition indeed encompasses models like GPT-3 or GPT-4, which can generate text in various styles and formats, including translations.
However, when it comes to full-text translators like Google Translate, DeepL, or Bing Translate, there's a nuanced difference. These systems are typically based on neural machine translation (NMT) models, a specific application of deep learning tailored for the task of translating text from one language to another. While these NMT systems are indeed 'generative' in the sense that they produce new text in a target language, their primary function is not to create original content but to convert existing content from one language to another as accurately as possible.

I would also say that typical models that people call "Generative AI" like ChatGPT and Dall-E, are not meant to create original content either, so that would not be a good criteria to define generative models.

Wait... What are they meant to do then, if not "create original content"?

So is my understanding correct that in the field of AI, "Generative AI" is a term which can include more than just LLMs? Or would you say that this interpretation is quite unusual and too far-fetched?

Depending on the model, they are trained to predict next word given some context (LLMs), or to imitate a training set of natural images (Stable Difussion and Dall-E) , creating new content is a strong claim that is generally unproven. For example see the NYT lawsuit against OpenAI.

@DocBrown Correct, Generative AI existed long before LLMs.

"see the NYT lawsuit against OpenAI." is not very helpful. If you ask a LLM like ChatGPT to generate a story about imaginary characters you made up living on the moon, it will. You can't tell me that that isn't "new content"...

@Cerbrus It will plagiarize an existing story with your characters, this has been shown over and over, the NYT lawsuit just shows that it can produce word by word content from the NYT. It applies to almost everything that is generated, another example about jokes: https://arstechnica.com/information-technology/2023/06/researchers-discover-that-chatgpt-prefers-repeating-25-jokes-over-and-over/

@Dr.Snoopy "It will" is not the same as "it can".  The lawsuit claims to have shown you can get it to reproduce specific content.  To show that this is the only thing it does, you have to show all content it produces is plagiarism, which specific examples fail to prove.

@Yakk No, it is a machine learning model, it only works on its training set boundaries, to claim novelty in generating content, you need specific evidence that has not been produced at all, while there is plenty of research showing the contrary, there is even a recent paper proving that Transformers do not generalize outside of their training set: https://arxiv.org/abs/2311.00871

@Dr.Snoopy If I remember correctly, ChatGPT had also closely replicated a news article written after its cutoff, so it couldn't have been plagiarizing

@Esther may be the news article itself was plagiarized from something that existed before chatgpt's cutoff

@Dr.Snoopy I mean, a pseudo-RNG produces novel output in terms of pixel static.  Such a step is part of art generation by techniques like stable diffusion.  It might not be interesting novel content, but one of the algorithms inputs are samples from a larger space of novel "noise" than the total number of images ever uploaded to the internet.  Unless there is convergence in the output image from the random source, it cannot only be duplicating images, there aren't enough holes to put the pigeons in.  If you want interestingly novel content, no guarantee, but it is novel.

@Yakk Sorry but your explanation did not make any sense to me, Diffusion models just estimate probability distributions, there is no magic there.

The distinction between "generative" and "non-generative" AI isn't an especially useful one. A language model (to first approximation) takes a sentence as input and tells you how probable it is that a native speaker would phrase it that way. Any remotely decent translation system is going to contain a language model, either as a separate component or integrated into the translation model proper.  This helps ensure that the text it outputs is grammatically correct and scans well. Without a language model, you get a word-for-word translation that won't sound natural at all.
But if you have a probability distribution over the next word given the previous words, not only can you answer how probable an existing sentence is, but you can take a series of words and sample the next word in proportion to its probability under that model. You then put that word in the context and generate a new one, and so on. Any model that can evaluate the probability of a sentence can be trivially modified to generate a sentence. Similarly, you can take any generative model and output the distribution over the next word instead of just selecting a single one; the product of these successive distributions will be the joint distribution over sentences of that length.  The models are exactly the same in either case; only the user interface differs.
(Of course, when setting policies, those user interface details might actually matter, since we might want to encourage users to fix their grammar but discourage them from posting answers when they have no idea if they're correct. But those details are entirely uninteresting mathematically.)
Historically, the use of language models in things like machine translation and voice recognition are the serious uses, and using them to generate text is a neat side task that can be used to sanity check the sort of sentences it thinks are representative of the model. (Text generators can also be fun toys, of course, which is how they're being used in things like ChatGPT.)

Maybe the distinction is not clear cut, but obviously there are AI systems which clearly do not count as GenAI, and others which clearly count as such for the majority of people. In the middle, there is a grey area. Do you think systems like Google Translate belong more clearly into the "non-GenAI" catgegory, the "GenAI" category or somewhere into that grey area?

@DocBrown The problem is that the classification "Generative AI" is not based on strict mathematical or engineering properties. Likewise, banning use of "generative AI" cannot be relaibly based on a model containing some mathematical or enginering element. Translation systems are, to me, generative systems. They sample from a probability distribution of outputs, and to create a translated sequence, usually reinsert what they just sampled as "written so far", just like LLMs performing text completion. The models are very similar. What is different are the task-based constraints on ground truth.

@NeilSlater: yes, thanks. This confirms to me that Generative AI is a vague term which is commonly used for a lot more systems than the current AI ban on Stack Overflow seems to be intended for. Hence I think it needs to be clarified whenever it is used in a certain context. I miss exactly that clarification in SO's current AI policy text.

@DocBrown I think focusing on a strict definition for the ban is looking at the wrong thing. What the authors of the ban want to prevent is use of low-effort machine assisted answers that have a high chance of looking authoratitve, but that contain mistakes. The ban is a rough heuristic towards that goal. Looking for loopholes or theoretical overreach, either as a poster or as a mod, is missing the point.

@NeilSlater: yes, absolutely, but I am not looking for a strict definition for the ban, I am looking for a better heuristics which is not so single-edged as the current phrasing is. My post here, however, is an attempt to question my POV that a better wording is needed, since the term GenAI has IMHO a broader meaning than some of the MSO protagonists are pretending.

@DocBrown It's not that the meaning is broad so much as that the meaning is based on "how it's used" rather than "what it is". Given that, it would probably make sense to phrase the policies as something like "Don't use language models to create semantically significant parts of your answer (as opposed to modifying an answer you wrote, without changing the semantics (e.g., via translation or grammar improvement)." Say what you don't want the users doing instead of trying to exhaustively list all the tools they might use to do it.

@Ray:  That sounds really good - that is something I would greatly support. I understand that the policy lists examples of well-known tools which are banned, to make it unmistakable clear clear whats forbidden. An additional paragraph about responsible use of AI could state exactly what you wrote - other AI tools might be used for improvements of an answer as long they don't add or change the semantics of any input a user of such a tool provides.

Actually it depends, generative models are specific kind of machine learning models. Generative often means a model that models the probability distribution of data $p(x)$, you cannot do translation using this kind of model.
But also you can build conditional generative models, where the probability of data points $x$ conditioned to some input $y$ is modeled $p(x \, | \, y)$, and in this case it is possible to perform a translation task using a generative model, where $x$ is the translated text, and $y$ is the original text.
So it all depends on the model you use for the task, if you use a (conditional ) generative model to perform translation, assuming that the model is trained for that task (like ChatGPT), then yes, translation can be done using generative models.
The concept of translation in AI/ML is more general, to just produce an output given an input, for example, CycleGAN is a generative model that translates between two sets of unpaired images, from horse to zebra and viceversa. And GANs are generative models.
About the specific question on DeepL and Google Translate, it depends on what underlying model they use, if at some point they use generative models to perform translation, then they should also be flagged as generative AI.
In general the concept of Generative AI has... deformed over time... (due to hype). Generative AI has existed long before LLMs and Difussion models like Dall-E and Stable Difussion, just look at Generative Adversarial Networks (GANs),

We're not asking if you can. We're asking if these translation tools are generative. Your answer is basically "we don't know"... Considering the accuracy of the translations, and the tendency for generative AI to generate inconsistent output, I'm fairly sure generative AI doesn't play a role in them.

@Cerbrus The OP is asking more generally, if language translation can fall into the category of generative AI, and the answer is definitely Yes.

@Dr.Snoopy: you understood me right. Still I agree with Cerbrus on one point: your answer did not directly answer my question about common usage of the term Generative AI. However, your former comment does, thanks for clarifying.

@DocBrown Good, I integrated that comment into my answer now.

The original question has been updated so my answer is being updated to reflect this.
The question being asked can be boiled down to just "What is it that makes something a generative AI?" I am quite familiar in my field of computer vision with Generative Adversarial Networks, but I believe after looking at Large Language Models that they operate on a similar principle. I will go over both of these.
Generative Adversarial Networks
First I will go over the GAN which I know well, though I have not implemented one in almost 8 years. In a GAN, you have 2 machine learning models, one is generative, and one is not. There is a non-generative model called the discriminator who's job is to determine if an image belongs in the dataset or not. Then you have the generative model who is considered an attacker or adversary, who is there to try to fool the discriminator, by producing an image that the discriminator thinks belongs to the dataset, but it does not.
During training we cycle between training the discriminator on the dataset until it cannot be fooled. Then we train the adversary until it can fool the discriminator. Then we add the adversary's images to the dataset and go back to the start again. We continue this process until the discriminator can no longer be trained to tell the difference any more, and the adversary wins. Now the GAN is complete and we discard the discriminator.
The GAN can be altered to have an input parameter where you can select a class, and it will attempt to create an output of that class to fool the discriminator with that class. This is an optional feature, but something many GANs try to implement.
LLM vs GAN
LLMs on the other hand looks to only have one model. If you dig into it though, you will find out there really is 2!.
Creating LLM Vector Stores
The Vector store is a model too! It is a generative model that provides an output to the final model that interprets the output. So you have a model that is trained on a large language database, and a vector store that is trained to output segments of words similar to the input it is given. Both the input you give and the input the vector store outputs is given to the regular model, and it takes that and provides an output to you.
So mechanically, what ties these two types of generative AI together? How can we create a mechanical definition? You have a normal network, either a discriminator, or interpreter, and then you apply to it a generative network, either a adversary or tokenizer(?) respectively during training. When you are done, either you keep them together in the latter case, or you discard the discriminator and publish the adversary in the former case, and you have a generative AI.
For older translation models, they likely did not have the processing power, or technology available to perform these kinds of tasks. This is a relatively new field. They would not have been able to be generative AI. They at most would be comprised of one model during both training and publishing.
Old answer continues below:
> [arXiv:2307.15208][4] states "Generative AI refers to a set of
artificial intelligence techniques and models designed to learn the underlying
patterns and structure of a dataset and generate new data points that plausibly
could be part of the original dataset."

Considering this, what we see is that a generative AI differs from a traditional AI in that it creates new information where none was before. A traditional AI might detect which paintings belong to Raphael, while a generative AI might try to create a new painting in Raphael's style.

I think your misconception here is that all neural network AIs produce information of some kind, but they do not produce new information. The information was already there in the input in some form in a traditional AI.

 - The location of the dog was in the picture already, the neural network just pointed to it and said, "Here it is."
 - The tune matched a Metallica song in a database already, the neural network just correctly identified it.
 - The information was already encoded in a foreign language, the neural network just decoded the information and then reencoded it into a new language of your choice.

Now if the model had instead read the foreign language, decoded the information, and used it as a prompt to write a novel in the language of your choice, the contents of which included new information never before seen in that input, now we are talking about generative AI. This is in fact what they do.

A lot of the media talks about how chat GPT can hallucinate it's answers when you ask it straight forward questions, but the truth is, it is generative AI. It is giving you answers that could have conceivably been in the answer set. They just aren't. This is what some experts mean by it hallucinating up new answers. It's pulling them out of the void like a generative AI is supposed to do. It just isn't what the average person expects this type of program to do.

In short, what the difference between a traditional AI and a generative AI is that a traditional AI learns a dataset, and tries to tell you something about the input with respect to the dataset. The generative AI takes the input and tries to produce something similar that it thinks would be from the same grouping in what it believes the dataset should be.

Strange. It did not appear to be behind a paywall for me, but you are correct in that it is a medical paper, but it is also the correct definition as well. I don't see what the issue is with the definition. https://arxiv.org/pdf/2307.15208.pdf

The definition in that paper (about medical imaging) looks like it was written by authors who picked a phrasing which suits their needs in context of that paper. This is nothing I would consider to be a valid statement about the general, common use of the term GenAI in the field of AI, more one possible use of the term in the field. In that context, that is ok and surely correct, but I would not draw any conclusions about the terminolgy for translator systems from it.

... note also I never said I believed all artificial neural network systems count as GenAI, that would have been clearly nonsense. I was talking specifically about translators, where you feed one text in as input, and get a - translated - text as output. My understanding of this process is that is is not uncommon to call this "generation" (of a text in a diffferent language).

As I noted in my answer, this is a misconception on your part. Generation in this case has more than one meaning. Yes, it generates text. No it does not generate new information that did not exist in the original message, at least not intentionally. Of course mistranslations do happen, and when they do, new information can be accidentally injected, but that is considered a bug, and not a feature.

So you deny that the common usage of the term GenAI in the field can include translator systems (as long as they are based on what you call "traditional AI")? Note I mean the common usage of the term over the last decade and maybe earlier, not the "popular usage of the last year".

I deny the fact that the definition used by me or by others here is wrong simply because it is not used in a more general case. The people who create this definition are the ones pioneering the systems, and they do so by writing papers on what they do in specific cases. You can learn what they mean by when they define their terminology in their papers. When all of the papers agree on a definition, you know what it means in the general case. There isn't going to be a paper that defines it for you in the general case. Other than that, the definition is set arbitrarily by those that do this work.

As such, the definition says that a generative AI will produce a result out of nothing that will match what it believes should be in the dataset. A translation AI does not produce a result out of nothing. It produces it from it's input. By definition, a translation AI is not generative.

I did not say the definition in that paper is wrong, quite the opposite. I only said it is most probably not telling us what the common usage in the field is, because it is just one paper with a specific context.

Ok, think that new answer is a lot better that your first since it offers a much deeper insight in what counts as generative AI in the field, thanks for that. Still it leaves me puzzled what this precisely means for my title question.

I'm glad I could help.  It took a while to figure out where our disconnects were. Sometimes when you learn a field you forget how to talk to people who haven't learned it yet, you know?

No, they are not "generative".
As clearly explained by Wicket on the GenAI Beta site, sourced from SirBenet's answer on the same site, Google Translate isn't "generative":

output could be considered too tightly defined by the input text

These translation tools don't generate new text. They translate.
Sure, they use AI to infer meaning to result in better translations, but you can't tell them to "Translate Macbeth to Swahili"...
You'll never get more out of the tools than you put into them.

Sirbenet's answer was refering to What is generative AI (GenAI) according to this site?, where "this site" means "Gen AI meta". I am asking about the common use of the term in the field of AI.

In this question here, I am asking about the general use of the term in the field of AI, not just at SE or SO.

You're free to ignore my answer.


text:
If the output is 0.09, does this mean that the prediction is class 1 or 0?
I use a Keras EfficientNetB7 and transfer learning to solve a binary classification problem. I use tf.keras.layers.Dense(1, activation="sigmoid")(x) for my final layer.
My labels are encoded as the following for the model.fit():
[[1.]
 [1.]
 [0.]
 [1.]
  ...
 [1.]
 [1.]
 [0.]]

My question is about the output of the model.predict(). For example, if the output is [[0.09122807]], does this mean that the prediction is class 1. or 0.?
Initially, I assumed it would have been class 0. but my model predicts the opposite of this assumption. In some stackoverflow posts, I saw that the output should be used 1 - p where p is the probability of class 1.
There is conflicting information on the net; hence, wanted to ask your guidance.

It is nearer to 0 than 1 which means the result label is 0.

If you are using binary cross-entropy as loss function, the output of Sigmoid functions represent the probability of $\mathbb{P}[y=1]$. Hence, it means that $\mathbb{P}[y=0] = 1 - \mathbb{P}[y=1]$.
Therefore, when the output is $0.09$, it means that the prediction is $y=0$ as it is more probable ($1-0.09 = 0.91$).


text:
Are the Dot Product and Tensor Product the same thing in Machine Learning?
I'm currently reading "Deep Learning with Python, Second Edition" by François Chollet, and I need help understanding one thing.
Below paragraph was copied from the page 41

2.2.3 Tensor product
The tensor product, or dot product (not to be confused with an element-wise product, the
* operator), is one of the most common, most useful tensor operations.
In NumPy, a tensor product is done using the np.dot function (because the mathematical notation for tensor product is usually a dot)

What has been described later in that section looks like a dot product to me (I don't have mathematical background though, I've just recently taken an interest in the subject), but this term also seems to be used interchangeably with "Tensor product".
However, according to every other source I've came across - dot product and tensor product are two different things (example: wikipedia article on dot product and wikipedia article on tensor product).
Is this an error in the book or am I missing something obvious here?

In machine learning, a tensor is a multidimensional array with some operations. In mathematics, the definition of a tensor is slightly different (see the Wikipedia article). However, the definitions are related because mathematical tensors can be represented as multidimensional arrays. See e.g. this post.
In this book, the tensor product is defined as the numpy operation np.dot, which can perform the mathematical dot product (aka inner product), matrix multiplication, etc., depending on the dimensionality of the inputs.
So, in conclusion, terms are overloaded and can vary across disciplines and you should always take the context into account.


text:
Why is the sinusoidal model classified as absolute positional encoding in some literature?
I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative.
My question:
Why is the sinusoidal model classified as absolute positional encoding in some literature, given that in Vaswani's original paper it was said that it captures relative relationships between words, and this has been proven here.
However, while I was reading a research, it was mentioned that projections that occur in the attention layer destroy this:

Indeed, sinusoidal position embeddings exhibit useful properties in theory. Yan et al. (2019) investigate the dot product of sinusoidal position embeddings and prove important properties:
(1) The dot product of two sinusoidal position embeddings depends only on their relative distance. That is,  is independent of .
(2) , which means that sinusoidal position embeddings are unaware of direction. However, in practice the sinusoidal embeddings are projected with two different projection matrices, which destroys these properties.

Is this the reason?

Absolute position embeddings capture the absolute location of a token. Absolute location would refer to e.g., the 1st, 2nd, 3rd token etc. The sinusoidal embeddings in Vaswani's paper capture this absolute position information. But, if you have absolute position encodings, then you can also always derive relative positions, and sinusoidal embeddings make that really easy, but because absolute position is encoded, it would be considered an absolute position encoding.
Contrast that with relative position encoding, where only the relative position between two tokens is used. e.g., in the paper, the embeddings are only used during the attention operation, and they only capture information about the distance between two tokens.


text:
Overcoming the quadratic scaling in transformer architecture
Do you know any papers that try to overcome quadratic scaling problems by attending lower dimensional representations in the dimension of tokens?
For example, let's say that the input to the transformer is (batch_size, token_number, hidden_dimension).
Do you think it would make some sense to do let's say one Transformer Block (TB) and then map (by some weight matrix) token_number to let's say token_number/2 and do TB again?
The graph might look like this:
input_tokens -> TB -> hidden_tokens -> TB -> ... -> output_tokens (might be equal to input_tokens in dimension)
So, our shapes would be:
(B, T, H) -> (B, T/2, H) -> (B, T/4, H) -> ... -> (B, T/4, H) -> (B, T/2, H) -> (B, T, H)
where B - batch_size, T - token_number, H - hidden_dimension

Yes. What you're describing seems to be similar to the Hourglass Transformer, the Funnel Transformer, among others.
Looking at the first paper, they consider two ways of downsampling tokens (upsampling methods are similar).

Linear pooling: reshape (n, d) -> (n/k, k * d), with k being the downsampling factor. Then, use a linear map to project k * d to your desired output dimension.
Attention pooling: First do simple 1D max/mean pooling to downsample across n: (n, d) -> (n', d); n' < n. Then, pass that into an attention layer (with residual connects), where the queries are the downsampled embeddings, and the keys & values are the original embeddings. For example, you may start off with x.shape = (10, 16), then do mean pooling to get x_pooled.shape = (5, 16). Now, each vector in x_pooled gets converted to a weighted sum of vectors in x (attention operation) and added to the original vector in x_pooled (residual). This is described in more detail in the second paper.

There's also approaches that instead do a discrete selection over tokens (e.g., Pyramid-BERT).

This is exactly what I am looking for. Thank you for your answer!

@AndyYermakov No problem! Also, please accept the answer if it sufficiently answers your question.


text:
Do I understand the technology of AI upscaling of films and cartoons correctly?
With the help of artificial intelligence, it is possible to increase the resolution of images that are initially low resolution, bringing it to ultra-high resolution. Also, initially static images are turned into short animated videos. But I'm interested in AI upscaling of video materials such as old movies and perhaps cartoons.
In my opinion, the technology is like this:

video file is dismantled, i.e. is decomposed into separate pictures-frames.
using GAN, the resolution of each frame is brought to the required value.
new frames are re-edited, i.e. new pictures-frames are combined into a new video file.
as initial data for training the neural network, the original film or cartoon, images with similar characteristics, other films or cartoons, and a large group of high-resolution images similar to frames are used.

Do I understand the technology of AI upscaling of films and cartoons correctly, or are there some other nuances and important details that I forgot to mention? I would like to get a clearer.

The approach you outline in the steps 1 to 4 will work to a degree, and may be used in some very basic video upscaling systems. However, the results may not be satisfcatory. A very typical problem will be flickering due to inconsistent upscaling choices made on consecutive frames. In addition, upscaling and interpolating the spatial dimensions of a video sequence will increase the pixel "jumps" between frames (more pixels will have large change between each frame), so to keep the output visually smooth it is often desirable to increase time resolution, i.e. increase the frame rate, interpolating smartly between frames.
The first problem, differences in details when processing frames separately, is called temporal coherence. There are a variety of ways to increase temporal coherence when upscaling. All require some comparison between consecutive frame images. Some are specified as soft constraints, or a loss metric applied to the processing during upscaling. Many models are trained specifically for video upscaling, taking multiple input frames, so that such losses can be used to improve the core model.
Sometimes, temporal coherence can be implemented as a modifier when recombining the upscaled frames - i.e. as part of your suggested step 3. This will be ok for simple interpolation upscaling that doesn't attempt to add missing details. It will tend towards smooth blurring for the upscales, similar to the simplest upscale options in image editors.
This is still an ongoing area of research, with papers routinely published. Here are two that I found on a quick search. I have not vetted these papers for whether they might be useful to you, they are just examples:

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution (Decembver 2023)
Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement (May 2024 preview)

The second problem, frame rate increase, is solved by having models specifically built to solve that problem. Again this is usually achieved by having inputs of consecutive frames that need interpolating, but this time output will be an interstitial frame that is not an upscale of either input. Training data for these models is typically high frame rate videos where the input has had frames removed, and the output is the relevant missing frames.
Some video generation models perform both tasks - image and frame rate upscaling - starting with an initial small low frame rate generation, and alternating layers between frame rate increase and image size increase.

Thank you! The “physics of the process” and related problems are clear to me. But the block diagrams and mathematics underlying the methods for increasing temporal coherence and increasing frame rates need to be studied separately. Because, as I understand it, GAN in its pure form is not used for this. Therefore, surely those models that solve both problems have a specific structure?

Well, I think that using the keywords from your answer I will find enough information to study.

@dtn: There are quite a few different approaches, and different kinds of models used to solve these problems. So there isn't a specific structure, it's more about how deep you want to look into the issue. Similar is true for image upscaling in the first place. A simple model is to linearly interpolate to get values for missing pixels. Slightly more sophisticated is to use bicubic interpolation. Wavelet decomposition. The modern AI-based upscalers don't do any of that, and even looking at the generative model upscalers, there are GAN-based, Diffusion-based etc. No single model or structure

In general, I would like to study the issue at such a level that I could experiment with the upscale of some old films, which are now perceived completely differently than before... Well, and get some results and practical knowledge.

@dtn I suggest you start with a naive system based around your initial idea. Building it and experiencing the issue with temporal coherence and frame rate first hand will be valuable experience, and far more rewarding than trying to build the most sophisticated version you read about as the first attempt. Once you have a basic version, then you can refer to ideas in this answer to upgrade it

I think you're right. I can start with short films. To gain basic skills. And the rest of the base will be added in the process. Thank you for your response and for your time!


text:
Why does skip-gram uses linear maps for embedding?
This question is inspired from Lilian Weng's blog here about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ takes in one hot encoded embedding into a 'better' embedding space.
My question is, why should we require a linear map for this task? As in, why should some linear map from one-hot encoded space be useful for embedding words densely, while preserving semantic information as much as possible (this is the definition of word 'better' as used above)?
For an overview of the model, see here:

Whether linear maps are "required" is an extremely difficult question -- I'm sure there's a number of ways to implement the general idea for the skip gram architecture. But this is more of a research question.
This might actually be the intention of your question, but there are definitely clear reasons why linear maps are an intuitive choice.
As discussed in the post, the goal is the convert the sparse one-hot representations into dense vectors. One intuition is that the meaning of words is determined by it's neighbors -- so skip-gram learns a model that can project one-hot representations into a lower dimensional representation such that that representation can be used to predict the held out word. This may remind you of autoencoders. Finally, linear maps are an extremely easy way to project vectors into different dimensions (it's just a matrix multiply!), making it an intuitive choice here.
In fact, reading the original paper, the mention:

The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.

i.e., linear maps are appealing because they're simple.