text:
PPO agent of path planning does not learn
I built an agent based on PPO with actor-critic structure for a project of path planning, in which the objective of agent is to find a path toward the goal bypass the obstacle in a 16*16 grid map. This is the main information of my model in details:
Both actor and critic network has the structure of two hidden layers with 128 neuron nodes for each layer and a 40-dim input layer. The 40-dim input is the state observation including two parts of information: the first 32 one-hot generalized elements represent the 2-dim coordinate [x, y], and the last 8 elements represent the obstacle condition of current position (with 1 meaning obstacle and 0 meaning free). the output of actor is an 8-dim vector as the probability of eight actions, whilst that of critic is the state value v(s).
The main hyperparameters has been set as below:

Learning rate: actor: 2e-04, cirtic: 4e-04
Max grad for both networks: 1.5
Policy clip: epsilon = 0.2
Discount: gamma = 0.99
GAE parameter: lambda = 0.95
Entropy regularization parameter: 0.005
Batchsize = 128

The reward function is set like this rules:

the positive reward (+1.0) is obtained when the manhattan distance from current node to goal is smaller otherwise the negative (-1.0) for larger.
While the next grid will be the obstacle, the agent will stay with reward 0.0.
The agent will obtain +5.0 if it leaves from the nodes adjacent the obstacles.
Reward reshaping: If the agent continuously moves toward the correct direction, there will be an extra adding for rewards.

Experiment results: In the light of my tracing result for the loss value of both network, I can be quite certain that the model is converging during iteration. However, the convergence does not perform as well as I expected. the MSE loss of critic descent from more than 30.00 at the very beginning but finally stuck around more or less than 10.00 then start to fluctuate. On the other hand, the actor loss is constantly holden around 2.00 and it never be nagative nor has any large changes.
Moreover, I calculated the average return of the last 50 episodes as the the measurement of convergence. Just as the image illustrating, the return performs fluctuating during the entire training process, meaning it still has no significant improvement after 2,500 training episodes (with 200 maximum step of each epoch). In a quite large probability, the agent still selects the actions to far away the goal or moving to the obstacles.
Here, I really hope anybody to help me to solve this problem. I will be very grateful for your help and advices. Many thanks!

Is the position of the obstacle fixed for this environment, or is it moved in each episode?

It may help to watch a learned policy.  There’s a lot of details in the reward structure, plenty of room for misalignment.

@Neil Slater
Sorry for missing for this. Actually, I created several obstacle areas in the map. All of them are static.

@foreverska
Thank you for advice. However, could you please explain the 'misalignment' more clearly? Is that meaning the reward function is too complex?

Misalignment is when AI learns something the designer didn’t intend.  And yes it may be a bit too complex.  Although I’m not well versed in PPO so that’s just a guess.  Like you said, it heads towards obstacles.  This could be because the policy wants to capture that +5 for walking away from it.


text:
What is the relation between non-stationarity and "a moving target problem" in multi-agent reinforcement learning?
If single-agent RL algorithms such as Q-learning is applied to multi-agent systems (e.g. Markov games), the environment from the perspective of the agent is non-stationary, and the agent is faced with a moving target problem, that is, the optimal policy changes as other agents' policies changes. I understand this as the optimal Q-function, $Q^*(s,a)$, being the moving target, since the optimal Q-function depends on other agents' policies.
If the agent learns in the space of the joint action space, then according to several references, the environment is stationary from the perspective of any agent, even though the agents' policies may change over time. Suppose the goal of the agent is to learn a joint optimal policy defined as a Nash equilibrium, from which it bases its actions on. Then, the agent tries to find an optimal Q-function defined as $Q_{{\pi^*}}(s,\textbf{a})$, where a denotes the joint action and $\pi^*$ denotes the joint optimal policy. I would then conclude that $Q_{{\pi^*}}(s,\textbf{a})$ is no longer a moving target, since we are in the space of joint policies, and thereby account for other agents' policies explicitly.
So as I understand it, non-stationarity and the moving target problem are two sides of the same coin. Is this correctly understood? Or can the environment be stationary while the problem is still a moving-target?

it seems as though you're defining the moving target problem and the non-stationarity problem to be the same thing. personally, when I heard the moving target problem, I would think immediately of the fact that the function approximates in DRL is being trained towards a "moving target", in the sense that it bootstraps from itself (i.e. when you update your network, then you're also updating the target, since the target comes in part from the network).

The definition of non-stationarity in multi-agent learning is indeed correct.
The 'moving target problem' you are referring to is investigated here: Transient Non-stationarity and Generalisation in Deep Reinforcement Learning. Both of them together have been discussed in section 5.10.1 in this book: Multi-Agent Reinforcement Learning: Foundations and Modern Approaches

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?
This is a rather soft question. Some people believe that GPT-style models can eventually solve very complex math problems if the models are large enough, but I'm skeptical about this. Suppose the GPT model is indeed large enough and can make very accurate predictions about the probabilities of the next token. Now we let it solve an extremely hard math problem, e.g. proving the Riemann Hypothesis. Proof length is not an obstacle, since the model can generate lots of tokens and therefore gains a lot of computational resources.
I think the most serious obstacle is how to generate the most likely proof, instead of the most likely next token. The bad thing is, that a proof might seem very plausible in the beginning, but may look more and more ridiculous as more tokens are generated. So, greedy decoding does not necessarily work; that's why people came up with beam search. But, beam search is also kind of greedy, and it also does not guarantee to produce the global optimal proof.
There may be extreme cases in which the optimal proof is among the top k+1 candidates but our beam size is k so it's not adopted. Therefore, the problem degenerates to a classic computational complexity problem: how on earth can we find the optimal generated sequence? This seems to be rather irrelevant to LLMs.
This problem is clearly in NP since its corresponding decision problem is: given a scheme to generate the next token probabilities, can we generate a sequence with length as most l and log-likelihood at least p? This decision problem can be verified in polynomial time by providing a satisfying sequence. I suspect this problem is NP-complete but I don't know how to prove it yet.
So, I conjecture that, unless P=NP, we can't use GPT-style LLMs to solve really hard math problems. Do you think so?

I think that recently it has been a very interesting topic of conversation using LLMs for planning in the context of reasoning or for the case you mentioned for mathematical proving. Specially with the news of the supposed "Q*" that all the sensational news are very hyped about. Overall this is an opinion based on my experience and not a very factual thing based on the idea that these are currently open-ended research problems in the field and there could be a research paper in the future that proves me wrong.
I do not think that Natural Language or at least the methods we currently have are the best solution for solving sophisticated math reasoning or any type of active research problems. I do think that the models are very good intelligent algorithms able to solve math problems. I think that the field of mechanistic interpretability has reached very interesting results that contribute to the idea that LLMs are not just transformers that learn from the biases of the data and predict the next probable token based on previous data. They are able to run a model of the world and extract patterns they can generalize. I recommend the work of Neel Nanda on the topic. The question is then, if these models can be expanded more and more, can we upgrade the models in scaling, better data quality and diversity, to reach the so called "AGI"?
I think that better improvements in reasoning, especially in math, would depend less and less on prompting techniques and more on better fine-tuning of math problems or more broadly on better pretraining techniques. I think that ICL (In Context Learning) can only set you so far, and better models are going to come with better reasoning skills. It has been seen that models, even with very good few-shot learning, aren't capable of solving certain tasks. The concept of using planning like Tree of Thoughts and trying to optimize an answer by generating variants and selecting the best outcome is definitely a not very efficient way of handling the problem becuase of the computational implications and the intrincic limitations that these Models have in understanding.
In the case of LLMs proving a mathematical statement, it is very different than using Natural Language. First, because the language of mathematics must be completely logical, something that lacks in Natural Language. Inside each language, there are a lot of logical incongruencies that are innate in the way we speak. This is often called "sui generis", which means in Latin "in a class by itself", meaning that each language has its own set of rules that makes sense inside them. A mathematical analogy would be that each language has its own set of axioms that are different between them. Mathematics is just another language with its own rules. That's why only relying on the Language Model to analyze and try to create a proof can lead to various logical errors.
In fact, automatic theorem proving programs, like Lean, have already been explored and there is a lot of literature on the topic. There has been a lot of research in combining both Language Models with these programs using Retrieved Augmented Generation (RAG). Like Lean + GPT-4 + Embeddings.
In the context of Theoretical Complexity, I am not very familiar with the field, but am sure the problem is definitely a hard problem because we are not just considering proving an algorithm. We are considering if there is an algorithm that can solve an arbitrary number of proofs. It certainly sounds like an NP-Complete problem because of the complexity, but I also do not know how to prove that. The search problem is one thing, but using a Language Model makes it even a lot harder to determine complexity. I am very skeptical of Language Models for  creating new knowledge because it lacks important features in the way humans create knowledge. For example, human interactions are very important. The way that humans can overspecialize in a certain topic for years and encapsulate knowledge for other humans to use for other purposes is a very interesting phenomenon that I think because of the problem of the scaling of language models, it is something we cannot do in a feasible way. We have a single, best performant Language model that is way better in almost all tasks than any of us, but not the best at anything. And there are also a lot of different discussions on whether or not these technologies could ever reach that type of performance in realizing these types of intelligent tasks.
In my opinion, the evolution of AI for scientific research is going to be more focused as helping tools for researchers rather than autonomous agents that can solve very complicated problems. Tools like AlphaFold, AlphaTensor, AlphaDev, GNome, and basically all the other research for scientific AI of DeepMind has more future in my understanding. We still do not have good models for imitating the scientific development that humans can do. Either way, I do not think we should not explore this type of idea because it is a very interesting problem with a lot of important implications for humanity.

Thanks a lot for writing such a long answer. I do agree that the important thing to do now is to make LLMs a helper rather than a mathematician, but there are people working on using LLMs to do automatic reasoning, like Yuhuai Wu (a co-founder of xAI). I'm trying to say that beam search does not allow LLMs to undo what they've generated, but humans can backtrack when they've found the current way of solving problems is seemingly ineffective. My question focuses on the defect of beam search and impossibility to search over a complex space with greedy method

@Soha - your problem is deper than that, Beam search ranking criteria has nothing to do with assessing correctness, usefulness or matching to reality of the results. It refines only the language. So the whole "what if the best result is in k+1" thing is a red herring. A good/correct answer could be in process of being formed in the current beam and then rejected because there is no mechanism to promote it, no part of the system recognises it, and if it were novel in some language-affecting way it could well score badly.

@Soha - The specific problem with automatic theorem proving using language models is that it doesn't guarantee 100% accuracy in predicting the steps of the proof (it can perform infinite illegal reasoning steps). However, this is not the case for systems like Lean, which use other ML methods to search in the possible action spaces. Your question would be more appropriate for these types of systems, not LLMs. Besides, it's probably almost impossible to answer that question. How could we optimize for solving the Riemann Hypothesis? It's even more challenging to establish its computational cost.

@CesarRuiz I know that formal proof assistants like Lean or Coq are more suitable for searching for proofs. But I was asking about GPTs, because I just want to emphasize that LLMs are typically not good mathematicians and refute the rumors like "LLMs will replace mathematicians" etc.

@NeilSlater I don't think beam search "merely" refines the language. If the token-generating model is big enough, the token that makes the "most sense" will tend to be given a high probability; the model has an attention mechanism and can look back at what it has already generated, so the most probable token should be also "mathematically correct". Ideally, if the model can tell the correct proof in the first place, then greedy decoding is enough. My argument focuses on the fact that a model can't "search" for a proof since beam search doesn't allow it to backtrack.

@Soha Even backtracking will not help because LLM perplexity scores will not rank correctness well enough, and there is no scoring mechanism for correctness.

@NeilSlater It might be true for today's "small" models, but I think the output probabilities will reflect the correctness of the proof if the model is large enough.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What role does data quality plays in the LLM scaling laws?
DeepMind released the Training Compute-Optimal Large Language Models paper in 2022 which describe some scaling laws for LLMs. As far as I understand this is the most accredited reference to estimate the optimal relation between dataset size, compute power and model size.
Recently a number of models have been developed using far less data, parameters and compute than the bigger LLMs. Yet these models achieved great results thanks to much better data quality. For instance models like WizardLM, TinyStories and phi-1. The recent article published on nature Human-like systematic generalization through a meta-learning neural network seems to imply that data quality is the key.
I'm curious about what role the data quality plays in the training of LLMs. I'm wondering things like: is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too? Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?

is the set of values estimated by the Chinchilla scaling laws optimal for these smaller models with optimized data too?

This is an open research question, but some research such as Beyond neural scaling laws: beating power law scaling via data pruning and Tiny Data seem to imply that higher quality data gives better scaling laws than Chinchilla estimates.

Do we have any model to estimate the quality of some datasets and some scaling laws that take it into account?

Not yet!


text:
How feasible is it to automate Theorem Proving via Reinforcement Learning?
Theorem proving is basically a turn-based game with perfect information: you start from a given gamestate (a proposition) and make moves that lead to other valid gamestates. A piece of software can check whether the moves you make are legal and whether you reached a valid solution completing the game.
That doesn't sound very different from Go, although of course the search space is much larger.
A similar game seems perfect for RL; for an algorithm like AlphaZero. We could generate an unlimited amount of theorems by combining existing ones (possibly aided by a model trained to generate interesting or difficult theorems), and train a model to solve them.
In the past years I've followed this field, and I've seen the birth of a bunch of tools (my favorite is LeanDojo), as well as a lot of research (using Lean, Metamath, Agda, Coq, as well as human-generated non-formal proofs). Nevertheless I haven't seen any noteworthy attempt at using Reinforcement Learning in combination with a theorem prover to create a powerful proving model.
I'm thus wondering whether it's feasible with today's technology to automate Theorem Proving (up to human-level) via Reinforcement Learning?
If feasible, I'd love to see some research on the problem by respectable organizations. While if it's not, I'd like to understand what the issues are: is it just because the search space is so large, or are there other challenges I can't see?

I've also wondered about this. Not a mathemetician, but math researchers probably care more about proving or disproving particular conjecture than obtaining a huge number of true statements. With that in mind, I expect it will be much harder to evaluate whether a particular state of your theorem prover is close or far to your target theorem than it will be to determine if a particular state of Go is winning or losing. Especially because those regions of state space will not have been explored in existing data (if they were, we might have proved/disproved the theorem already).

It is feasible, even though we are at the level of high-school mathematics at the moment. But there is room for scaling. Examples are:
HyperTree Proof Search for Neural Theorem Proving (2022)
DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback
for Reinforcement Learning and Monte-Carlo Tree Search (2024)
The first paper use a generalization of AlphaZero, while the second build a much more complex pipeline including supervised training, MCTS and reinforcement learning.


text:
What is actually being saved in the file when you save a model? For example a Tensorflow SavedModel file
I'm building a feature for my application that requires reading the properties of a saved ML model file (after it's trained). However, as I am pretty new to this field, I don't really understand the foundational knowledge about developing AI, only knowing that parameters are trained and saved. I have some experience with the Tensorflow framework, and I know that a model can be saved in either a Savedmodel file or a h5 file.
My questions are:

What is actually being saved in those files typically? And what does it look like?
Is it a big difference when you save your model differently? for example h5 vs savedmodel or tensorflow vs pytorch.
Is it possible for me to read meaningful information from any saved files regardless of the format? So it doesn't matter if someone else used Pytorch, Tensorflow, or C to do ML

Your questions seem to be off-topic here. They are mostly programming questions and specific to frameworks. We focus on the theoretical/non-programming aspects of AI.

Answer to Question 1
TensorFlow's documentation provides the following information on what is saved:

The model config, weights, and optimizer are included in the SavedModel. Additionally, for every Keras layer attached to the model, the SavedModel stores:

The config and metadata -- e.g. name, dtype, trainable status
Traced call and loss functions, which are stored as TensorFlow subgraphs.

You can use TensorBoard's Graphs dashboard to visualize your model.
For a quick dive watch Tensorboard with Tensorflow 2.0 under 10 mins.
Answer to Question 2
See jdehesa's answer to a similar question regarding TF and H5 files: What is the difference between .pb SavedModel and .tf
Answer to Question 3
Yes. Both formats, .pb and .tf, are used to save meaningful data and are used for reading in the models.

so based on your answer, is it true that most ML models are saved in a similar architecture that contains model config, weights, and optimizer? Or I think what I really want to ask is, how do I read information from ML model files of any format, regardless if it was trained with Tensorflow or Pytorch.


text:
Is it possible to write/build an AI model without using Frameworks?
I'm a relatively newbie in this world of Artificial Intelligence, although I am able to use frameworks such as Tensorflow and also understand the general concepts behind training weights and parameters, I wonder about a lot of stuff.

So is it necessary that people use frameworks like Tensorflow and PyTorch to do ML
And what do people do before frameworks were introduced?

In short : YES, you can build AI models without a framework

Now, onto the details:
Before discussing "why", let's look at what is an ML framework:
Simply put, an ML framework is a collection of abstractions that allow you to build ML models. If you are from a software engineering background, you may think of them as libraries with built - in modules to accomplish common ML related tasks, including but not limited to:

Manipulating Tensors
Loading and Processing Data
Training a model
Deploying your model

For example, Keras is a popular Deep learning library with many 'ready to use' components.
The 'Why' Part
To understand why you need a framework, you should have a basic understanding of what building an ML solution entails:

Loading the data into your computer's memory
Preprocessing data to some format that is suitable (for example you may have to preprocess images to have some fixed dimensions before feeding it to your model)
Defining a basic model architecture. This architecture would vary depending on your task
Training Your model
Deploying it further

Let's see how a framework helps at each stage:
Stages 1, 2 : Major Deeplearning frameworks provide you with very easy to use interfaces for loading and processing data. These are generally as simple as making a few lines of core API calls.
For example, in PyTorch, loading the famous MNIST database is as simple as

training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

Without such a simple API, you would be writing a ton of code to manually fetch, download, extract, copy / load your dataset.
Stage 3: These DL frameworks also provide you easy interfaces to manipulate underlying Tensors in memory. (Essentially mostDL Models can be decomposed into Matrices/Tensors). This is by far the single largest reason, i.e. Deeplearning Frameworks are capable of moving and manipulating these Tensors in GPUs. And you my be well aware about how much Compute Power ML Models need. GPUs are especially good at handling parallel processing tasks. (For example, you may take say 10,000 dimension Tensor and add a single number to it. Conventional programming would involve running a for loop to iterate over each individual element, whereas GPUs can process the entire Tensor parallely)
Stage 4: Finally, most DL frameworks also provide you means to deploy your trained model into production. (Although, this is a debatable topic as many people argue that it's better to re - write the model in performant languages like C++ before deploying to production)
Addressing your concern over what people used before these frameworks came into existence:
Well, these popular DL Frameworks per se did not exist a decade ago, but we have had libraries and packages that attempt to perform some of the above tasks.
For example, the popular python package Numpy provided the capability of operating over nd arrays long before Tensorflow came into picture.
As AI Expert Andrej Karpathy pointed out, during his academic journey, before GPU accelerated frameworks, he relied on tools like MATLAB to perform his experiments.
Should you or should you not use a framework?
For most cases, it's better to use a framework as it will save you a great deal of pain.
If, however you want to learn the fundamentals of machine learning, down to the individual tensor and how it is manipulated, it might be useful to implement a Neural Network from scratch without any libraries (you might still need to use stuff like Numpy, or go full on hardcore C/C++ )


text:
What AI do I need for pizza delivery system?
For example I want to have a chat bot for pizza ordering.
So far I assume that I need (I'm not sure):

Context Question Answering (questions about menu, pizzas)
Entity Recognition (ordering some pizza)

Questions:

What should development process look like for this kind of neural network?
Should it be custom model from a scratch or existing model?
What about language translations of existing model - is it possible to translate text model to another language? We are talking about languages with similar grammar, so translation of words should be enough, but how to do it?


text:
How do you define an action space for a card game with an unlimited and variable hand size?
I'm new to the world of AI and have been primarily reading through the documentation for OpenAI's Gym/Gymnasium in hopes of training an AI to play a board game. One piece of information I haven't been able to find is the best way to define an action space when the number of possible actions is countably infinite.
For instance, I've seen several examples of simple games where the player must choose a card to play from their hand. In all of these, the hand has a maximum size, and the action space is discrete with its cardinality being the largest possible hand size. But what if there is no theoretical limit to the size of the player's hand? Let's say a player starts with a hand of seven cards, but the number of cards in their hand could grow (or shrink) to any natural number throughout the course of the game. How would one go about defining the action space in this scenario?
I've thought of four potential ways to handle this:

Define one action for every possible card and mask the action space so that only cards in the player's hand are legal actions. This would mean 52 actions for a standard deck of playing cards, which doesn't seem so bad. But for certain games, there could be many more kinds of unique cards (potentially thousands in the case of trading card games), and I could see the action space becoming unwieldy in those situations.
Define a continuous action space on the interval [0,1] and partition it equally into however many cards are in the player's hand. For example, if there are 10 cards in the hand, an action in [0,0.1) would play the first card, one in [0.1,0.2) would play the second, and so on. I'm unsure if this would be a wise setup since "nearby" actions like 0.09 and 0.11 wouldn't necessarily correspond to similar results because cards 1 and 2 aren't ordered in a meaningful way.
Define two actions: one for playing a card and the other for passing. Loop through each card in the hand and apply the action to each one until the "play" action is used. In this case, I'm not sure if the AI would be able to "look ahead" to know that a better card is coming up so therefore it should pass on the current card.
Pick an arbitrarily large hand limit that a player would never reach in practice and define a discrete number of actions up to that limit.

Would any of these approaches work? Are any favored over the others? Are there other ways of approaching this that I haven't thought of?

is this a question in general or is there a specific task you are trying to get an agent to perform? Without further information it may not be possible to give a satisfactory answer because this is a very open ended question. For example, what are the rules of this game? Do we start with a limited number of cards in our hand but this value could grow? Are we "dealt" a countably infinite hand?

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

@respectful I meant this as a general question, but I've added a couple of details to make it a little more specific--namely, starting with a fixed hand size that could grow or shrink throughout the game.

A policy that handles an arbitrary number of actions can be specified as a function (think: neural network) that takes one specific action as input and outputs some kind of score.
That score may be the estimated action-value, but the point is that you can simply pick the action with maximum score now. Or feed the variable number of scores into a softmax distribution, then sample from it to pick a plausible random action.
I'm not sure if the Gym/Gymnasium API is any help in modelling this kind of action-space. For board games specifically you may find some recent papers[1] that explain their approach, which usually involves tree-search.
Also think about how your agent will see the world. If it gets to select one of 500 actions, it will have a really hard time (need a lot of data) to learn that action number 311 means something completely different in one state compared to another. So depending on the game, it may be useful to add local features. (Like state of the evaluated action's neighbours, if applicable.) If the game state is just one huge opaque bitmask, the agent will have to discover via trial and error that there are local neighbourhoods, symmetry, etc.
Again specifically for board-games, if you can exactly predict the next state based on an action, it may be sufficient learn the value of a state.
[1]: Schrittwieser et al., 2020: Mastering Atari, Go, Chess and Shogi by Planning with a
Learned Model


text:
Is it possible training accuracy never changed while training?
Question summary

What informations can get from this epoch_accuracy graph?
Is it possible training accuracy never changed like after 10 epoch in graph while training?

Body
I do some experiments with neural networks, just for hobby.
but, I get some strange epoch-accuracy graphs.

This graph result came from my models, and I have some questions about this graph.
In epochs 10, and training accuracy never changed. I see text result when training with models.fit() in tensorflow.keras. Before I learned, Training(not validation) accuracy should be changed while training. Because Model updating parameters, and parameters continuous changed while training. so, This is my first time seeing a graph like this.
I use flatten layer for input layer, and only use MLP for hidden + output layer. so, I really wondering Is it possible training accuracy never changed while training?
Edited(update loss, accuracy log)
Sorry,I try to update loss graph, but I deleted my model.(because It's just hobby, and I found some stranged status.)
So, I make new model to represent that situations.

After 20 epochs, then accuracy never changed. I show the train log. Please note that loss and val_loss are changed very little, accuracy and val_accuracy never changed after 20 epochs.
Epoch 18/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5232 - accuracy: 0.4410
Epoch 18: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.5273 - accuracy: 0.4410 - val_loss: 1.5631 - val_accuracy: 0.4818
Epoch 19/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6061 - accuracy: 0.4176
Epoch 19: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6053 - accuracy: 0.4186 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 20/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6062 - accuracy: 0.4109
Epoch 20: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5734 - val_accuracy: 0.4320
Epoch 21/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6069 - accuracy: 0.4097
Epoch 21: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5757 - val_accuracy: 0.4320
Epoch 22/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6040 - accuracy: 0.4125
Epoch 22: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6044 - accuracy: 0.4115 - val_loss: 1.5735 - val_accuracy: 0.4320
Epoch 23/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6041 - accuracy: 0.4121
Epoch 23: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5747 - val_accuracy: 0.4320
Epoch 24/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6065 - accuracy: 0.4105
Epoch 24: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5779 - val_accuracy: 0.4320
Epoch 25/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5998 - accuracy: 0.4168
Epoch 25: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5769 - val_accuracy: 0.4320
Epoch 26/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6081 - accuracy: 0.4086
Epoch 26: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6057 - accuracy: 0.4115 - val_loss: 1.5739 - val_accuracy: 0.4320
Epoch 27/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6015 - accuracy: 0.4146
Epoch 27: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5723 - val_accuracy: 0.4320
Epoch 28/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6021 - accuracy: 0.4125
Epoch 28: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6053 - accuracy: 0.4115 - val_loss: 1.5726 - val_accuracy: 0.4320
Epoch 29/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6051 - accuracy: 0.4110
Epoch 29: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5741 - val_accuracy: 0.4320
Epoch 30/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6050 - accuracy: 0.4116
Epoch 30: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5764 - val_accuracy: 0.4320
Epoch 31/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6045 - accuracy: 0.4113
Epoch 31: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5747 - val_accuracy: 0.4320
Epoch 32/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6057 - accuracy: 0.4082
Epoch 32: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5730 - val_accuracy: 0.4320
Epoch 33/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5995 - accuracy: 0.4148
Epoch 33: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 34/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6082 - accuracy: 0.4094
Epoch 34: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5736 - val_accuracy: 0.4320
Epoch 35/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6038 - accuracy: 0.4105
Epoch 35: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5740 - val_accuracy: 0.4320
Epoch 36/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6071 - accuracy: 0.4090
Epoch 36: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5733 - val_accuracy: 0.4320
Epoch 37/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6022 - accuracy: 0.4131
Epoch 37: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 38/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6033 - accuracy: 0.4121
Epoch 38: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5755 - val_accuracy: 0.4320
Epoch 39/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6027 - accuracy: 0.4152
Epoch 39: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 40/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6124 - accuracy: 0.4039
Epoch 40: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5741 - val_accuracy: 0.4320
Epoch 41/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6011 - accuracy: 0.4145
Epoch 41: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5734 - val_accuracy: 0.4320
Epoch 42/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6060 - accuracy: 0.4117
Epoch 42: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5746 - val_accuracy: 0.4320
Epoch 43/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6046 - accuracy: 0.4105
Epoch 43: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5743 - val_accuracy: 0.4320
Epoch 44/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6020 - accuracy: 0.4136
Epoch 44: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6044 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 45/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6118 - accuracy: 0.4062
Epoch 45: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5757 - val_accuracy: 0.4320
Epoch 46/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6063 - accuracy: 0.4090
Epoch 46: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 47/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6035 - accuracy: 0.4121
Epoch 47: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5760 - val_accuracy: 0.4320
Epoch 48/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6094 - accuracy: 0.4098
Epoch 48: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5738 - val_accuracy: 0.4320
Epoch 49/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6120 - accuracy: 0.4051
Epoch 49: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6049 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 50/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6088 - accuracy: 0.4094
Epoch 50: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5761 - val_accuracy: 0.4320
Epoch 51/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6042 - accuracy: 0.4126
Epoch 51: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5762 - val_accuracy: 0.4320
Epoch 52/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6049 - accuracy: 0.4114
Epoch 52: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 53/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.5987 - accuracy: 0.4157
Epoch 53: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5753 - val_accuracy: 0.4320
Epoch 54/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6033 - accuracy: 0.4133
Epoch 54: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 55/10000
84/88 [===========================>..] - ETA: 0s - loss: 1.5977 - accuracy: 0.4185
Epoch 55: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6043 - accuracy: 0.4115 - val_loss: 1.5780 - val_accuracy: 0.4320
Epoch 56/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6007 - accuracy: 0.4160
Epoch 56: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6059 - accuracy: 0.4115 - val_loss: 1.5768 - val_accuracy: 0.4320
Epoch 57/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6001 - accuracy: 0.4153
Epoch 57: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5758 - val_accuracy: 0.4320
Epoch 58/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6060 - accuracy: 0.4113
Epoch 58: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 59/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6009 - accuracy: 0.4148
Epoch 59: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 60/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6017 - accuracy: 0.4153
Epoch 60: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 61/10000
83/88 [===========================>..] - ETA: 0s - loss: 1.6038 - accuracy: 0.4134
Epoch 61: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5766 - val_accuracy: 0.4320
Epoch 62/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6066 - accuracy: 0.4093
Epoch 62: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5738 - val_accuracy: 0.4320
Epoch 62: early stopping

It's not possible to answer this question without also considering loss curves

@Karl Sorry, I added the train log. I knew this model can't learning well. But why this model's accuracy and val_accuracy looks like fixed value? Thank you.

What happened with your model is that it suffered from a Neural Network collapse. This means that your network didn't learn to generalize with the data or that the local minimum found in the gradient descent was to output an output that almost doesn't change. This often happens when the neural network didn't learn the features of the data and the best pattern it found was to give an almost constant mean value that maximizes the accuracy. Let's say that your distribution on the training data is 60% for class 1 and 40% for class 2, then it will try to predict all data as class 1, in order to maximize the accuracy. Things that you may try for solving this issue could be, increasing the layers of your model, adding more inductive biases that align with your data, or changing hyperparameters. You may want to see the output probabilities of your model to understand my point. Another thing about your data is that your loss doesn't seem to decrease, but changes randomly. I plotted your log prints and it looks like this, which for a Neural Network, is not a good loss curve.


text:
what's color dropping meaning in data augmentation?
I'm reading about contrastive learning paper, they use data augmentation as method.
By the way, I have some questions about data augmentation. What is color dropping?
color jittering is changing HSL(Hue, saturation, lightness) on image. But I don't know color dropping.
I search on google, but it doesn't help. I assumed color dropping as ...

bluring?
remove color on image? then show as grayscale or binary image?
add some colored water drops on image? like raining windows which rain  contained color water.
change the color some part on image?
change a image as pointillism art?

I really don't know color dropping on data augmentation. could you help me?
Thank you.

Could you link the paper related to your question?

@LucaAnzalone Sure. I give a paper's title by ISO 690 format, I recommand to search google scholar.

CHEN, Ting, et al. A simple framework for contrastive learning of visual representations. In: International conference on machine learning. PMLR, 2020. p. 1597-1607.

they reference second paper's title, but That couldn't search with "color drop"

HOWARD, Andrew G. Some improvements on deep convolutional neural network based image classification. arXiv preprint arXiv:1312.5402, 2013.

It cannot find about "color drop"

@LucaAnzalone 3. HENAFF, Olivier. Data-efficient image recognition with contrastive predictive coding. In: International conference on machine learning. PMLR, 2020. p. 4182-4192.
They reference fourth paper's title, but I couldn't understand fourth paper's key sentence yet.

@LucaAnzalone 4. DOERSCH, Carl; GUPTA, Abhinav; EFROS, Alexei A. Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision. 2015. p. 1422-1430.
They said "An alternative approach is to randomly drop 2 of the 3 color channels from each patch('color dropping'), replacing the dropped colors with gaussian noise" on 4th page in pdf, or 1425 page in paper. I think this is key sentense, but still confused what's the meaning of the drop in this sentense. there are any existed example pictures, so I am confused.

By dropping color channels, they mean replacing the color channel with noise.
For example:
import numpy as np
import cv2
import matplotlib.pyplot as plt

test_image_path = '4.2.07.tiff' # peppers from https://sipi.usc.edu/database/database.php

# read image
img = cv2.imread(test_image_path, cv2.IMREAD_COLOR)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img.astype(float)
img = (img / 255.0) - 0.5
plt.imshow(img + 0.5)

Then dropping channels (the output will vary based on how the parameters of the gaussian noise):
# keep the first channel, drop the other two
img[:, :, 1] = np.random.normal(0, 0.1, (img.shape[:2]))
img[:, :, 2] = np.random.normal(0, 0.1, (img.shape[:2]))
plt.imshow(img + 0.5)

Thank you so much! I understand with examples.


text:
Does data preprocessing is necessary and important in deep learning?
I really wonder about data preprocessing is really necessary and important in deep learning.
It's really hard to say clearly about difference between Machine learning and deep learning. In definetion, The difference is whether or not humans intervene in the process of finding features. Machine learning need humans intervene to do feature extraction, so must need data preprocessing. So we do PCA, denoising, data balancing, and so on.
Machine learning is important to pre-processing. But, In deep learning, Is really important to do preprocessing at data? I'm not asking about such as embedding in NLP. text embedding is necessary for transform text to computer recognizable.
I'm really confused while studying about deep-learning. Someone says that it's important to do pre-processing, and someone says that isn't important to do pre-processing.
For example, I'll show some situation about computer vision. Let's think about Denoising, erase such as pepper and salt noise in image. Is it important to do denoise for input image?  someone who agree about to do preprocessing says that noise is unexpected pixel value, so if we denoised noise, than we can get more clear image, so It brings more good performance. someone who disagree about to do preprocessing says 2 reasons.
First, If we can get 100% clear original image without any noise, then compair denoised image and clear original image. It's not totally same image. Also, we can't get without any noise image because of physical reasons.
Second, Deep learning shows better performance when the amount of input data is large. If human intervene to feature extraction, It's not efficient to do with human power. Deep learning can do feature extraction itself, don't need human to feature extraction.
I apolized that I think it's not good example about my question. so, I give another example.
Think about Iris dataset. In Machine learning, We do PCA(or else) and select efficient data(ex: sepal and petal width and length data) and input data to model. But, deep learning don't need select efficient data. If we have nice deep learning model, then we can input whole data.
last example, It is estimated that GPT-3 has been learned with more than 300 billion sentences. I don't think training dataset did pre-processing. Because, sentence might have typo, wrong grammer, slang, abbreviation, etc because human writing. If do preprocessing about typo, wrong grammer, etc then It would have consumed a lot of human labor and costs.
So... Is deep learning really need pre-processing?

About your salt & pepper noise, actually in most situation (like classification) it might be good to add it/keep it since it allows the model to be more robust. It's a particular case of data augmentation... Most of the time, preprocessing means making the optimization process easier by e.g scaling the numerics. Do not confuse it with data augmentation (e.g adding rotation, translation, chaning brightness, contrast, adding noise etc).

@Lelouch I agree salt and pepper example is not a good example. But, examples of Iris dataset and gpt isn't correct in preprocessing part?

Building off of @Lelouch's comment -- adding augmentations like noise and performing preprocessing can help make a model robust, especially if your dataset is limited. However, if your dataset is huge to begin with (e.g., containing 300 billion sentences), then that's not really necessary as there's enough edge cases in your training data.
For your comparison to PCA/feature selection, deep models do that automatically to some extent. Early layers can be thought of as "feature-extractors" for later layers. This means that your model can be more prone to overfitting, but that can be fixed with data augmentation (or just by adding more data).

Thanks for answering. I know that deep learnings are doing feature-extractors(In transformer, encoders do that). So I think that deep learning isn't need pre-processing. But, data-miners(also do deep learning) ask me why I don't using preprocessing dealing with data. I can't answering about that. Data-miners who I met belives that preprocessing bring more better result. If It is true, but why many researchers are not included preprocessing tasks on their paper?

Of course, many researchers doing preprocessing tasks and included preprocessing tasks on their paper. As far as the papers I read, there were more that did not have a preprocessing process. So I really want to know with reference if data preprocessing really don't need preprocessing(And vice versa). I couldn't find, and searching continuously about reference that deep learning really needed preprocessing.


text:
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an encoder and a decoder. Depending on the model, you may use only the encoder, only the decoder, or both. However, for what purpose do model designers use only encoders, only decoders, or both?
I already knew that encoders in transformers are as known as taking in a sequence of input data and generates a fixed-length representation of it. This representation can then be fed into the decoder to generate an output sequence.
In other words, the encoder can be thought of as a kind of compression that extracts features from the data. And the decoder can be thought of as playing a role in returning the compressed information in the encoder to its original state.
So I'm wondering why some models work without having both an encoder and a decoder.
Few days ago, I think use only encoders are useful to classifying classes. Because DOSOVITSKIY, Alexey, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. paper shows only encoder to classification images. Decoders are useful in generative things, because WANG, Jianfeng, et al. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. paper using encoder to encode the visual information from the input image into a representation that can be used by the text decoder to generate text. Then, to generate text, they give the 'encoder's output and the text' as the decoder's input.
But, I am sure about that my think are wrong because of BERT and GPT. BERT using encoder and does not have a decoder. GPT uses decoder and does not have a encoder. A typical user thinks that BERT and GPT equally answer the question asked by the user. So they think BERT and GPT provide the same service. However, in terms of model structure, BERT and GPT are completely different.
So, I have two questions about each functional part that makes up the transformer.

what does encoder and decoder do in transformer? The transformer referred to here can be text or image.
For what purpose do model designers use only encoders, only decoders, or both encoders and decoders?

Thank you.

I really struggled to get your point, but the whole idea of encoder-decoder, is based on the necessity of conditioning your model... you want just test, just use the  encoder to generate data... you want to condition your text generation with an image, use encoder decoder

@AlbertoSinigaglia I really apologized that you are really struggled to get my point. Now, I edit my question more clearly. If you still hard to get my point, please notice one more time to me. If you notice to me, then I read and edit more clearly and add more explain. Thank you.

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.

The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "transformer model", but it actually consists only of an encoder or only of a decoder. That's fine as long as you know what exactly is going on.
(I would say that the distinctive feature of the transformer model is the attention layer.)
For example, as you mentioned, Generative Pretrained Transformer (GPT) is a decoder-only model, and Vision Transformer (ViT) is an encoder-only model.
I feel like your question already contains the answer :)
Yes, encoders are used exactly for that - encoding (or compressing) the input. Note that the output sequence from the encoder has the same length as the input sequence. This is why encoder-only models cannot be used for tasks where the output sequence has a different length or unknown length, e.g. machine translation, text summarization. In these cases you need the decoder.
Encoder-only models like ViT and BERT will encode the input sequence and produce an output sequence of the same length. Then for classification they will use only the first element of the output sequence.

As I said, encoder-only models cannot be used for generating an output sequence of different (or unknown) length, i.e. BERT cannot be used for machine translation or generative question answering. BERT can only be used for question answering on SQuAD-type datasets, i.e. your answer is a continuous segment from the input and the model only outputs <START_IDX> and <END_IDX> to mark it.
For generation tasks you need a decoder. The decoder is actually an auto-regressive model. It will generate the elements of the output sequence one-by-one until it decides that the sequence is ready and then it generates the final <END> token. See here.
Previously people used an encoder-decoder architecture to solve these sequence-to-sequence tasks (e.g. T5). You want the encoder to encode your input sequence and the decoder to decode it and produce the output sequence. However, it turns out that you can use a decoder-only model. You simply concatenate your source and target sentences and treat the task as a language modelling task. Now there are pros and cons to this approach, but the main takeaway is that you can do it if you want to.
You can checkout these papers to read more about encoder-decoder vs decoder-only models for sequence to sequence tasks:
https://arxiv.org/pdf/2304.04052.pdf
https://aclanthology.org/2022.aacl-main.43.pdf

Thanks for helping about my question. I need more study with your answer. I still confused, because why google bard(based on bert, made with only transformer encoder) can do text summmarization on google services. I test google bard with my question, request like "please summarize about given text." and paste my question. The bard gives answer like.... (next comment)

Sure, I can help you with that. Here is the summary of the text you provided:
-Transformer models consist of two main components: an encoder and a decoder.
-The encoder takes in a sequence of input data and generates a fixed-length representation of it.
-The decoder takes in this representation and generates an output sequence.
-In some cases, only the encoder or only the decoder is used.
-The encoder is often used for tasks such as classification, while the decoder is often used for tasks such as generation.
-However, there are also cases where both the encoder and decoder are used.

And then bard give answer about my original question. As we know(according to Pi-tau give knowledge), only encoder model can't do summarization. But google Bard tried to summarize the text. why bard looks like bard can do summarization? I need more time to search more info, because I want to reask with more background. Thank for helping, and If others still don't write answer on this question, I accept your answer after 1 week. :)

Okay, so first of all Bard is not based on BERT -- It uses LaMDA (https://arxiv.org/pdf/2201.08239.pdf). You can see in the third paragraph of Sec. 3 from the paper that this is a decoder-only model, just like GPT. The difference is that it is pre-trained specifically on dialog data.

"We use a decoder-only Transformer [ 92 ] language model as the model architecture for LaMDA."

Please note also that Bard is not just a transformer decoder language model. It is a complex system consisting of many, many components, one of which is the transformer decoder. It is also connected to google's search engine, so it can search the web for information that can help it generate the correct answer. I would recommend this podcast if you want to hear more about how models are trained to use the web: https://www.talkrl.com/episodes/john-schulman.

Thank you! It really help to me, then I understand about my question. :)

So, the difference is about seqlen where encoder is fixed and decoder is dynamic. In that case, encoder also can be used for summarization and generation, just set the fixed seqlen as much as possible (though it's impractical). Where we can also use stop token for both input and target, the dataset is simply the sliding window, where padding token is mandatory here.


text:
MCTS: Units away from the action
I'm trying to implement Monte Carlo Tree Search for (a simplified version of) the boardgame Commands and Colors -- I'm setting up a scenario where the AI side has overwhelming force: 6 units vs 3 units played by the human.
I would hope that MCTS moves the 6 units in for the kill; instead what happens is that some units attack, some move sideways, and some retreat.
I suspect that the units in the front are already strong enough to make victory likely, so that the unit in the back does not see a difference in its moving closer to the action, and chooses to move away from it.  I suspect that when evaluating the value of the moves of the far away unit, the "noise" caused by the action of the units in the front, whose actions make the value of the position swing heavily, make it very difficult to evaluate the smaller contribution to the quality of the position made by moving the unit in the back.
This is sad!  A human player would move all units towards the enemy, bc if the front units get damaged, they will be moved away from the front and replaced by second-line units.  Having units move randomly away from the action makes no sense.
How do I fix it?
-- Edit:
The source code is here https://github.com/xpmatteo/auto-cca
The case that does not work as expected can be observed with:

make server
(open another terminal)
make open
click "End Phase" twice

The lone brown (Carthaginian) unit should close in against the gray (Roman) ones, but it doesnt.

Do you have one centralized logic for all the units, or is it a logic per piece? The way you describe it, each piece has its own agent?

Unit logic is centralized -- at every step I generate the set of possible moves, which are the same for all units: they can move one hex each.  Once a unit has moved, it's "spent" for the current turn and cannot move again, so the set of available moves is reduced by one

Monte Carlo Tree Search is searching for the action with the best expected reward. If the reward is determined by whether it wins or losses, then all winning moves are equal and one will be selected arbitrarily. If you want them to win quickly, then you can adjust the reward for winning quickly.
Supposing your reward distribution is [0, 1], I would keep some constant reward for winning and add some reward that decays as time to win increases. For example, your reward formula could be for a win 0.5 + 0.5 / T where T is the time taken to achieve victory.

One option is that the RL is not there yet, you may need to continue with the training, or maybe add to the win/lose reward some tiny "hints" what is a good move (0.001 when you eat an enemy, -0.001 when your unit is being eliminated, etc..).
I do assume that all the moves of your units are taken in one "action" of your army, and that you evaluate the reward for all the units as one single "army".

The avaiable moves are modelled as individual unit moves, so during a turn, the AI moves all its units one by one, then it executes combats one by one, then the turn is over.  The playout ends with an evaluation of the position which is based on how much damage the AI inflicts the opponent minus how much damage it receives from the opponent

If you have 6 units with about 4-5 available moves each, there is a lot of move combinations the AI needs to choose

"the RL is not there yet" <-- I tried increasing the number of iterations, but it does not seem to find the solution I want. After a while, a "winner" move keeps getting selected even though it's not the move that I want, and increasing iterations does not help

Can you please share code. It will be a bit clearer.  Should be interesting for me and for others.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Shared the code -- have fun!

It is very nice and I recommend others to play with your code. Really nice.
Did I get it right that the Carthaginian are the ones with extra power and you refer to them as the AI?
It seems that the Carthaginian always win, so why should they learn to win this way rather than another way? If it is important to you how fast they will win, then add some penalty on the time maybe?

It is very nice and I recommend others to play with your code. Really nice. Did I get it right that the Carthaginian are the ones with extra power and you refer to them as the AI? It seems that the Carthaginian always win, so why should they learn to win this way rather than another way? If it is important to you how fast they will win, then add some penalty on the time maybe?

Thanks Oren.  The problem is, if I start with equal forces, I always beat the AI because it does not play very intelligently


text:
In the original diffusion model paper, why do they sample the first step with the same loss?
In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a repository with code here, which gives a few more clues.
Now there is one thing I don't particularly understand, and that is that although the KL divergence is taken over $t=2..T$,

in the code, they sample $t=1..T-1$ and say in the comments,
# choose a timestep in [1, self.trajectory_length-1].
# note the reverse process is fixed for the very
# first timestep, so we skip it.

Now if I understand correctly, with the first timestep of the reverse process is just $t=T$, which is just the isotropic gaussian, but what I don't understand is, why do they sample from $t=1$ instead of $t=2$ like in the KL divergence.
Also, if you were indeed to sample from $t=2$, how do you learn $f_{\mu}(x^1,1),f_{\Sigma}(x^1,1)$ so that you can reverse the last step? I would expect that to come form the entropy $H_q(X^{(1)}|X^{(0)})$, but from the code we see that they replace that with the entropy of a Gaussian with $\sigma=\sqrt{1 - a_1} = \sqrt{b_1}$,
$$
H_q(X^{(1)}|X^{(0)}) = \frac{1}{2}(\log 2 \pi + 1) + \frac{1}{2}\log b_1
$$
In the reverse process, you also see that they don't handle the last step of the reverse process any differently.
Summarising, how can they sample $t \in 1..T-1$, and calculate the KL divergence, while the equation specifies that $t \in 2..T$?
EDIT: After analysing the code further, it looks like they are not learning the last denoising step, and are only generating $x^{(1)}$.

I didn't watch the code but let me have a guess: For the reverse process, the last step from x(1) to x(0) should not add any noise respecting to f_sigma. Because this is the last step so adding any noise is not reasonable. So, just take x(1) as the final output is good enough to end the process.

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.


text:
Recognition of Part Numbers in text
I am veteran programmer but an AI/ML newbie.
I have a large body of technical text sprinkled with part numbers. The part numbers don't follow a strict format that I can easily use a regular expression to recognize. I have a large collection of "labeled" data. Each entry is a short snippet of text "labeled" with the correct part numbers that are recognized inside it.
What is the best strategy to create such a recognition machine?
EDIT - Clarification
So sorry, I live in my business domain and I assumed everyone knows what I was talking about.
Here are some examples of part numbers:

SAK-TC233LC-24F133F
MAX1480BEPI+
LM2676SX-5.0/NOPB
PIC16F1708T-E/SS
MT47H64M16NF-25E IT:M TR
IS42S16800F-7TL-TR
BSS110 (Q62702-S489)
BSS209PW L6327

A row is a line of text (usually the subject of an email) and can be extremely unstructured. Here are some subject line examples:

回复: Re: RFQ FROM IK STONE: MT47H64M16NF-25E IT:M TR 4000PCS
Re: RE: Fwd: MKL16Z128VFT4 1300pcs RFQ
Cancel: ORDER: STM32H743IGT6 IMPROVED PRICE
PO#2923960045---PCF8577CT/3,118  1000pcs
Re: Fw: Fwd: Requirements (China) of TXS0102DCTR from Shanghai QuiQun Capital Co., LTD
回复：LM2675MX-ADJ/NOPB IMPROVED OFFER

I have over 100000 correctly labeled subject lines, i.e. a subject line with the part numbers that are detected in it (can be 0 part numbers as well, not all subject lines contain part numbers)

Could you clarify what a "part number" is? Could also provide an example of your dataset? What is a row? What are the classes here, i.e. which possible "labels" can you assign to each input? How many examples (or samples) do you have in your dataset?

I added clarifications to the question, thank you for pointing it out.

One possible way to solve this is to first remove everything that looks like an english word (along with its adjacent punctuation)


text:
In which community does using a Bayesian regression model as a reward function with exploration vs. exploitation challenges fall under?
I am trying to find research papers addressing a problem that, in my opinion, deserves significant attention. However, I am having difficulty locating relevant information.
To illustrate the problem at hand, consider a multivariate Bayesian regression model represented as $y_t = b_1 \cdot x_{1,t}^{c_1} + b_2 \cdot x_{2,t}^{c_2}$. The purpose of this model is to estimate rewards $y_t$ based on inputs ($x_{1,t}$, $x_{2,t}$) at each time step denoted by $t$. The parameters $b_1$ and $b_2$ are assumed to follow a half-normal distribution, while $c_1$ and $c_2$ follow a gamma distribution. I aim to plan inputs over a given horizon $H$ given a specified budget $\left(\sum_t x_{1,t} + \sum_{t} x_{2,t} \le \mbox{budget}\right)$, with the main objective of maximizing cumulative rewards. To help with this goal, it is important to also gain knowledge about the model parameters to facilitate improved allocation of inputs for maximizing cumulative rewards in the future.
Every day we make a decision on how to allocate a portion of the total budget and every day we receive a response ($y=\mbox{sales}$). Typically, we have access to 30-300 daily data points to establish the initial posterior, and the planning horizon is typically 30 days.
This problem exhibits a trade-off between exploration and exploitation, which is commonly encountered in communities such as reinforcement learning and dual control. However, the constraint of only being able to run experiments based on the exact functional form of the model, rather than simulations in the real environment, makes it challenging to apply standard reinforcement learning techniques. While elements of this problem can be found in other communities, such as Bayesian experimental design and active learning, these approaches primarily address the exploration aspect.
An intuitive strategy could involve incorporating certain aspects of active learning, such as maximum information gain, along with an $\epsilon$-greedy approach. In this strategy, the mean of the posterior predictive distribution of the input can be used during exploitation, while maximum information gain can guide exploration by proposing new inputs. I can of course think of more approaches, Thompson sampling being one of them where we sample a single sample of parameters from our posterior and use it as ground truth in the downstream optimization task. Another one is to take multiple samples and use the mean of the posterior predictive distribution. One could be UCB with some tweaks etc. A third one could be to take a Thompson sample and compute the downstream optimization task and repeat it iteratively to get an comprehension of the variance of possible solutions and maybe in the end take the mean or median.

Despite the belief that this problem falls within the realm of sequential decision optimization and should have received extensive research attention, it is challenging to find specific resources that tackle this type of problem.
Question: In which communities can I find more information about my problem type?
Crossposted: https://or.stackexchange.com/questions/10611/bayesian-regression-model-as-reward-function-in-exploration-vs-exploitation-sett

EDIT
After reading the reports provided in the response below and examining various model-based Bayesian reinforcement learning techniques, several concerns arose in my mind, accompanied by potential solution strategies. To address these concerns and demonstrate some techniques applicable to resolving this matter, I initiated a new discussion thread. The problem formulation in that thread bears a striking resemblance to the one above, albeit with additional contextual information that closely aligns with the real problem I am currently engaged in.
This thread can be read here:
Methods for sequential decision optimization problem with nonlinear bayesian reward function In this current question, I am solely interested in hearing about which communities have been trying to attack this problem type.

Hi @paul and welcome to AI Stack Exchange! If possible, please outline your specific question in the body of the post. Right now, I'm unsure of what question to answer. I did my PhD in some very similar topic, and I'd like to help out if possible. Thanks, and we look forward to more of your questions in the future!

hi @DeepQZero , just edited: My main question is, where can i find more information about my problem type, under which communities, has this problem-type even been tackled before?

i would love to discuss this problem with someone who actually worked on something similar @DeepQZero because everywhere i look it is extremely difficult to find relevant info on this, bits and pieces can be taken from all the sequential decision communities but it seems so strange that noone had tackled this exact problem type before

also added a couple of more strategies(policies) i been thinking off, it seems so strange however that this problem has not been tackled before extensively in the literature.

Thank you for the update. Another question - is gaining info about the model parameters rewarded somehow, or is it only helpful in maximizing the total return? I'm asking this question since I usually have seen maximizing total rewards as the sole goal in similar problems, whereas gaining info about the model parameters is only a byproduct of trying to maximize the total rewards. If you need clarification, just let me know.

@DeepQZero it is solely helpful in maximizing the total return. No extra reward added for gaining info about them.

i would say that certain bandit-problems, active learning for regression, MPC is the closest i got to finding research that resembles this problem but they all have parts that are very different from this setting

I'll try to give an answer by the end of next week. If I don't give an answer or don't give an update here, please send me another message in this comment chain to get my attention again.

@DeepQZero  Lovely, thank you very much. Are there any specific research papers or Google searches that you recommend right away?

Try this arxiv paper: Bayesian Sequential Optimal Experimental Design for
Nonlinear Models Using Policy Gradient Reinforcement Learning

@DeepQZero wow, finally i found reports tackling something similar, awesome reports, not just that one, but all the other sOED reports. Will definitely reference them in my final report about my real problem(the one presented here is just a toy one showcasing main considerations). A wide range of questions popped up in my mind as i am digesting the reports and trying to bridge it to my problem. I aim to create maybe separate threads showcasing these questions by friday approx and then link those threads in this one. Hold back until i create those threads, i think you will ace all of them.

Glad to hear it! I'm busy until the weekend (at the very earliest) anyways, so I can't look at any of this until at least then. This sOED problem is very rich, so it doesn't surprise me that it's prompting more questions in your mind.

@DeepQZero no worries, i am just extremely happy someone can actually answer some of my questions, no matter how long it takes :), i created a separate thread/question(read the EDIT part in this question), that sums up my main concerns that popped up in my head when digesting the sOED reports, i would be so happy if you could shed some light on my thoughts in that thread(it is basically a bridge between an algorithm stemming from the problem above and sOED reports): https://ai.stackexchange.com/questions/41198/is-this-deterministic-lookahead-policy-greedy-myopic-and-suboptimal-if-so-can

@DeepQZero hi, hope you are doing well, just wanted to send a little reminder in case you had forgotten, i appreciate your help immensely. kind regards, Paul

I finally got some time to look at this question again. I'll try to give an answer soon, perhaps today.

One community that has very recently been attacking problems of the type posed by your question is the Bayesian sequential optimal experimental design (Bayesian sOED) community. The Bayesian sOED setting generally assumes that there is a current belief state $x_k \in \mathcal{X}$ over an underlying system with some unknown quantity (in your case, unknown parameters) and possible design choices $d_k \in \mathcal{D}$ that may be conditioned on the current belief state. Performing experiment $k \in \{0, 1, 2, \ldots, N-1\}$ is equivalent to choosing a design $d_k$, receiving an observation $y_k \in \mathcal{Y}$ from the underlying system, and earning a utility $g(x_k, d_k, y_k) \in \mathbb{R}$ based on the current belief state $x_k$, design choice $d_k$, and observation $y_k$. The goal is to choose a sequence of designs $(d_0, d_1, d_2, \ldots, d_{N-1})$ that maximizes the expected sum of utility across all experiments. Learning more information about the underlying system (e.g. the unknown parameters) is directly helpful in achieving that goal by producing more informative belief states $x_{k+1}$ after the conclusion of each experiment.
The above experimental design setting is:

Bayesian due to incorporating a belief state over the underlying system,
sequential because experiments are performed sequentially to use all information about past experiments as opposed to performing all experiments concurrently (batch design),
optimal due to maximizing the expected sum of utility across all experiments as opposed to only performing the maximization over the next experiment (greedy design).

Since your exact question is simply asking about relevant research communities, I won't make any specific remarks in this answer regarding how to solve your illustrative problem using Bayesian sOED (feel free to make a follow-up question on that topic, if desired). Instead, I will list some references that I personally have found useful for solving problems in this field, with those most directly related to the Bayesian sOED problem listed first:

Bayesian Sequential Optimal Experimental Design for Nonlinear Models Using Policy Gradient Reinforcement Learning
Optimizing Sequential Experimental Design with Deep Reinforcement Learning
Sequential Bayesian optimal experimental design via approximate dynamic programming
Bayesian Sequential Optimal Experimental Design for Inverse Problems using Deep Reinforcement Learning
Bayesian Sequential Optimal Experimental Design for Linear Regression with Reinforcement Learning
Bayesian experimental design for the active nitridation of graphite by atomic nitrogen
Simulation-based optimal Bayesian experimental design for nonlinear systems
Bayesian Experimental Design: A Review

Marked it as solved. Thanks. Very useful links, some of which i had not seen. I will formulate a subsequent question on how to formulate my problem as an sOED formulation, i make sure to tag you here when i have the question up and running. Thanks alot. Kind regards/paul

i just gave the answer an upvote! thanks a lot. I removed the previous secondary thread i created and instead made another thread which really sums up my problem and addresses my primary concerns when it comes to "solving" this problem. Find that thread in the edit part or:  https://ai.stackexchange.com/questions/41460/methods-for-sequential-decision-optimization-problem-with-nonlinear-bayesian-rew  I would really appreciate you giving it a try, it is incredibly helpful to me to have someone who knows these stuff looking at my problem. Thanks in advance, kind regards.


text:
Methods for sequential decision optimization problem with nonlinear bayesian reward function
I am attempting to grasp if there are any other methods out there that i am not aware of that can be beneficial given my problem context.
Being inspired from optimal experimental design communities and RL-communities i have a sense there is.
To illustrate the problem at hand, consider a multivariate Bayesian regression model represented as $\hat{y}_t = b_{1, t} \cdot x_{1,t}^{c_1} + b_{2, t} \cdot x_{2,t}^{c_2}$ where as $\hat{y}_t$ denotes predicted values and $y_t$ observations. The purpose of this model is to estimate rewards $\hat{y}_t$ based on inputs ($x_{1,t}$, $x_{2,t}$) at each time step denoted by $t$. The parameters $b_{1, t}$ and $b_{2, t}$ are time-variant implying that they may vary over the horizon and are not constant. For instance, we could consider day of the week dummies where day-of-the-week parameter has a gaussian prior or e.g a Gaussian random walk prior that associates with the day of the month, allowing it to capture and generalize the day-of-the-month effect observed across multiple months (assume we have daily historical data). $c_1$ and $c_2$ follow a gamma distribution. Lets assume a normally distributed likelihood $L(y_t|b_1, b_2, c_1, c_2, x_{1, t}, x_{2, t}) \sim \mathcal{N}(\hat{y_t}, \sigma^2)$. I aim to plan inputs over a given horizon $H$ given a specified budget $\left(\sum_{t \in H} x_{1,t} + x_{2,t} \le \mbox{budget}\right)$, with the main objective of maximizing cumulative rewards. To help with this goal, it is important to also gain knowledge about the model parameters to facilitate improved allocation of inputs for maximizing cumulative rewards in the future.
Note that every day, we have the ability to adjust our planning in accordance with the newly acquired information. As a result, we can recompute our decisions daily and implement them, essentially enabling us to recompute our decisions "online". Also note that we have a lot of time to recompute our (batch)decisions every timestep.
I am aware of essentially two, maybe three different strategies of "solving" this problem.
Consider the following algorithm showcasing these different strategies:

Estimate the posterior with e.g MCMC factoring in our current dataset
Sample from the posterior

Strategy 1: Sample a single set of parameters from the posterior(thompson sampling)
Strategy 2: Sample many samples from the posterior(SAA)

Construct a deterministic optimization problem over the whole horizon with the previously sampled parameters

Strategy 1: Objective function being $\sum_{t \in H}\hat{y}_t$ with the single sample as parameters
Strategy 2: Objective function being $\sum_{t \in H}\frac{1}{M}\sum_M \hat{y}_t$ where $M$ denotes the quantity of samples

Solve the determinstic optimization problem subject to the budget constraint
Set allocation for today
Retrieve reward

A concern regarding all of these approaches is the omission of consideration for how our decisions might influence our future belief states and, consequently, subsequent decisions. The proposed exploration techniques seems to me somewhat adhoc.

Question: Considering the context, are there any other methods beyond the ones I proposed in the algorithm that could be superior and that can be employed to address this problem? If so, how would I proceed with implementing them?

that is a very long question.  very detailed.

@EngrStudent should i shorten it down?, it is a complex problem and i think my proposals of strategies and breaking down some of my research could be really beneficial also for other practitioners.

@DeepQZero thank you, let me know if i should change anything in this question. best regards

@DeepQZero no worries, im just happy that anyone actually can answer these quite complex questions, wish you the best for your publications. best regards

@DeepQZero hiya, just a reminder in case you forgotten, i am very grateful for your assistance in this quite complex matter. Hope you did well with your publications. kind regards Paul

@DeepQZero i understand, i'll i have to work on my patience haha. I am just so excited to hear your take on this. kind regards, paul

One possible alternative approach is incorporating deep reinforcement learning (DRL) techniques. These techniques are sequential since they incorporate lookahead, and they are designed to attempt to reach an optimal solution. Moreover, a Bayesian viewpoint can be adopted through incorporating a belief state over the unknown parameters. This belief state transforms the original POMDP into a belief MDP (see second paragraph of page 2 of this paper) where the observations are now belief states (see the wiki for more formal details). The belief state is generally more informative than a simple observation of a POMDP, which should aid in training speed and stability.
These DRL methods are appealing for the posed problem setting since (i) many have off-the-shelf implementations for fast use, (ii) they natively navigate the exploration/exploitation dilemma, and (iii) they can be quickly applied to any new variant of the posed problem as long as it is modeled correctly (e.g. as a belief MDP). Since the question is asking about implementation details, this answer will detail the some aspects of the approach discussed in aforementioned paper. The first major consideration is the construction of the state for use in the neural network. Most practical DRL algorithms will optimize over a fixed planning horizon which we will assume is of length $T$. The first order of business is to create a one-hot encoding vector that indicates the current experiment number. This initial vector will be $0$ everywhere except in index $k-1$ during experiment number $k$ (starting from index $0$):
$$[\underbrace{0, 0, 0, \ldots, 0, \underbrace{1}_{\mbox{index}\ k-1}, 0, \ldots, 0}_{T}].$$
This vector needs to be augmented with the designs of each experiment as follows:
$$[\underbrace{x_{1, t_0}, x_{2, t_0}, x_{1, t_1}, x_{2, t_1},\ldots, x_{1, t_{k-2}}, x_{2, t_{k-2}}, x_{1, t_{k-1}}, x_{2, t_{k-1}}, 0, 0, \ldots, 0, 0}_{2T}].$$
All designs yet to be chosen can simply be set to 0. Finally, the history of observations needs to be augmented as well, with future observations set to 0:
$$[\underbrace{y_0, y_1, y_2, \ldots, y_{k-2}, y_{k-1}, 0, \ldots, 0}_{T}].$$
This entire vector is a vector of length $T+2T+T = 4T$, which is definitely a reasonable input size for a neural network under a planning horizon of ~30 days. Note that the designs and observations are all that is necessary (along with the original prior distribution) to construct the belief state. Since the state vector includes these quantities as a subset, also known as the information set
$$I_k = \{x_{1, t_0}, x_{2, t_0}, y_0, x_{1, t_1}, x_{2, t_1}, y_1, \ldots, x_{1, t_{k-1}}, x_{2, t_{k-1}}, y_{k-1}\},$$
then the belief state is indeed encoded in nonparametric form in the state vector. By not constructing the belief state explicitly, intermediate belief states do not require any additional computation, which generally is non-negligible (see page 10 near equation 26 in the aforementioned paper for other details and discussion of a nonparametric belief state).
Following the state construction of the belief MDP, the issue of action constraints needs to be addressed since there is a design budget in the posed problem. Some off-the-shelf DRL libraries allow for parametric or constrained action spaces that are dependent on the state. In this case, augmenting the state vector with the remaining budget may aid in training speed due to a more explicit state representation (as opposed to the neural network needing to learn budget remaining from the designs).
After the above primary considerations are decided, all that is left is to code the belief MDP as a RL environment and apply some DRL algorithms. Some of the references in the linked question can help provide insight regarding the application of DRL to such environments. The posed problem seems non-trivial, open-ended, and may require substantial experimentation; therefore, this answer will be inherently limited in scope. Here are some more major considerations that may aid during the experimental process:

Reward: Some applications provide a reward solely for the information gain in the unknown parameters. If information gain should be incorporated into the reward function, check out the references in the link question for ideas.

Recomputing decisions: Since the entire analysis can be recomputed each night, the algorithm can be rerun with a shortened state vector. The new belief state can be computed and used as the prior distribution in the new algorithm. A shorter time horizon is generally easier for DRL algorithms and could either confirm the future design choices of past algorithms or even improve upon them.

Belief state: The belief state representation was nonparametric and somewhat primitive. A better belief state representation may aid in training efficiency (e.g. parameters of the posterior distribution, if they are available).

POMDP solutions: Since a belief MDP is a special case of POMDPs, techniques that solve POMDPs might be good baselines. For instance, this paper shows how memory-based architectures provide strong baselines for solving POMDPs with DRL.

Good luck, and keep us posted with any success stories!

Addendum: Other Belief State Representations
Since you were curious about other possible belief state representations, I will detail two general possibilities below.
The first possibility is to make use of a conjugate prior distribution (wiki). Given a likelihood function, a conjugate prior can sometimes be chosen such that the posterior distribution is from the same distribution family as the prior. This approach prevents costly computational Bayesian updates. For reference, this paper makes use of a conjugate prior in section 4.1.
The second possibility is to use a grid discretization over the unknown parameters. In the case of two unknown parameters, a grid of rectangles can be placed over the possible (or high-likelihood) locations of the unknown parameters. Each rectangle can be associated with a single set of parameters, often the parameters located at the center of the rectangle. In this manner, a Bayesian update can be performed by computing the product of the likelihood and prior at each rectangle and then normalizing by dividing by the sum of such products across all rectangles. This method is appealing due to the ability of viewing each posterior as a heatmap. The previously linked papers by Shen & Huan and Huan & Marzouk serve as good references for this method.


text:
LUT-Based Sigmoid and Tanh Activation-Functions in Integer Quantized Networks
I want to understand how activation functions, specifically tanh and sigmoid, are used in int8 quantized neural networks. Even more specific, I want to understand a Look-up-Table based approach.
My problem is that the relation of data width and size of the LUT, scaling factor and zero point will affect which index of the LUT I have to use for the calculation. E.g. TFLite-Micro has an implementation of such a LUT for tanh and sigmoid. They have an implementation, but I do not understand how they calculate the indices. Is there some guidance on how to design such a function available somewhere? I haven't managed to find anything more specific.

Can you please reformulate the title as your specific question?

Their lookup-table is declared as uint16_t sigmoid_table_uint16[256]. At the bottom is a plot.
Input Scaling
Their Tanh() function gets 16bit inputs. Those are presumably the accumulated (and biased) results of several 8bit multiplications. So at this point there is no zero-point to consider any more, only a scaling factor.
(My understanding is that zero-point this is only relevant during the 8bit matrix-multiplication, but I'm not 100% sure to be honest. The results of the 8bit multiplications are of course 16bit, and you don't go back to 8bit until you have to.)
According to the code, the scaling is $\frac{\text{input_multiplier}}{2^{\text{input_left_shift}}}$. But the point is really to scale the input such that $2^{8+8}$ corresponds to a floating-point input of 10.7 (the chosen end of the table). We are in 32bit integer-land by now, and the input can be larger than $2^{8+8}$.
Look-Up
The number $2^{8+8}$ contains two engineering choices. First, the size of the look-up table. They have chosen 256 entries, so the top 8 bits will be the index. Second, the precision for linear interpolation (fraction part of the index). They have chosen 8 bits. An alternative would be zero, simply returning the nearest value from the table.
Now they divide by $2^8$ (the expression abs_input_data >> 8) to get the index, and use the remainder of the division (abs_input_data & 0xFF) for linear interpolation between the actual and the next value from the look-up table.
Interpolation
result = (ua << 8) + ut * (ub - ua);

Here they interpolate between ua and ub, which are adjacent 16bit values from the table, using ut (the 8-bit remainder).
To get a float sigmoid result (between 0.5 and 1.0) you'd have to divide by $2^{16+8}$.
Scaling back
If you check the plot below, the table has the right half of a 16bit unsigned sigmoid. But they want tanh. So they add back the sign and subtract the zero-point. The zero-point is at $2^{16+8-1}$ and written as 1<<(14 + 9) in their the code.
We still have to divide by $2^8$ from the interpolation, and shift one bit right to make room for the sign that we added, but we also have to divide by two to go from sigmoid to tanh. Those cancel out, and we divide by $2^8$ with rounding.
Rounding is done by adding 128 (or 1 << (9 - 2) in the code). But you also have to be concerned about the asymmetric range of signed integers, and whether C is rounding towards zero or towards negative infinity. Which is probably why they saturate at 0xFFFF00 instead of 0xFFFFFF but I'm not certain. With that, we're knee-deep in the engineering swamp and I've probably lost all the machine learning people, so I'll leave it at that.
TL;DR: Fixed-point math is not too hard to grasp, just impossible to read and validate. That's the price you pay to avoid costly conversions to/from floats.

Thank you very much! Why is net necessary to scale the input so that 2^(8+8) corresponds to the end of the LUT? Because that expression always matches the maximum value. Also, do you understand what the shifting of the result does? I understand it's fixed point, but I don't see where the exact values are derived from.

I've made a major edit which should answer both questions, I think.

Thanks! There are a few minor things I wonder after reading the update. Do you see any indication where the scaling factor for the input would come from? Also, you mention that they subtract the zero-point, but what zero point does this refer to? Finally, why do they add 128 to round? That number seems very random. Is it just an engineering choice, or is there some deeper meaning?

The first value in the table (see plot) is used for input 0. Because tanh(0)=0 this is the zero-point. They multiply it by 256 with (ua << 8) and later divide by 256 with an arithmetic right shift (line 108, result >>= 8). This division truncates the result (rounds towards -oo). If you add 128 first, you get normal rounding.

For the scaling factor, you'd have to read the rest of their code. I've only read the part that interested me personally, which was the LUT.

Thank you so much! This clears up a lot of my confusion. For the actual inputs: Are those just regular int values that are interpreted as fixed point here for the sake of the interpolation? Also, can you recommend any resources on fixed point operations? I feel like I am lacking quite a bit here.

Please refer to following document link, we convert any Q15 integer to Q4.12 fixed point format with help of input scaling with factor of 3/4. Q4.12 {3 bits - Integer, 12 bits of fraction}. Then ua uses upper 8 bits {Q4(INT).4(precision)} while ut uses lower fixed point precision.
Link - https://github.com/tensorflow/tflite-micro/pull/1253
Regards,
Chaitanya

Thanks! But I don't understand what scaling with the factor 3/4 does, or where in the code it is applied. Could you expand on that?