text:
DRQN with LSTM, abnormal Q-values
I'm implementing Deep Recurrent Q-learning mentioned in this paper https://arxiv.org/pdf/1507.06527 with sequential update.
To verify my implementation is correct, I plotted the following:
1.test return over training episodes (1 test every 1000 training episodes)
2.Q-values of all actions at first time step (1st time step Q should converge to optimal return)
3.selected action's Q vs observed return (it should look like a y = x function)
The result look like this(labels and legends of plot might be incorrect):
return converged to around 85 which is the optimal return

first time step Q-value

selected action's Q vs observed return for a test episode(5 time steps so 5 data points)

The Q-values seem abnormal as they rise to values much higher than the maximum return, then suddenly drop to values that are close to maximum return. I also plotted the same metrics for DQN, and the Q-values were smooth and accurate.
I suspect that the LSTM hidden state input might be affecting the output, but I have no experience with LSTMs and not sure if this behavior is common.
Can anyone offer any suggestions?


text:
How to solve the issue with getting free ports in Pytorch DDP?
I am facing issues with getting a free port in the DDP setup block of PyTorch for parallelizing my deep learning training job across multiple GPUs on a Linux HPC cluster.
I am trying to submit a deep learning training job to a Linux HPC cluster using a SLURM script. The training requires parallelization of the model and data across 4 GPUs. I am using the Distributed Data Parallel (DDP) module of PyTorch for distribution of the model and data across 4 GPUs. I have defined the DDP setup block as follows:
def ddp_setup(rank, world_size):
"""
Args:
    rank: Unique identifier of each process
    world_size: Total number of processes
"""
try:
    # Find a free port dynamically
    def find_free_port():
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.bind(('', 0))
            s.listen(1)
            port = s.getsockname()[1]
        return port

    # Destroy any existing process groups
    if dist.is_initialized():
        dist.destroy_process_group()

    # Set environment variables with a dynamic port
    os.environ['MASTER_ADDR'] = 'localhost'
    master_port = find_free_port()
    os.environ['MASTER_PORT'] = str(master_port)

    # Initialize process group with additional error handling
    init_process_group(
        backend="nccl",
        rank=rank,
        world_size=world_size,
        timeout=timedelta(minutes=60),
        init_method='env://'
    )

    # Set the GPU device
    torch.cuda.set_device(rank)

    print(f"Process group initialized successfully for rank {rank}")

except RuntimeError as e:
    print(f"Error initializing process group: {e}")
    # Provide more specific error handling
    if "Address already in use" in str(e):
        print("Port is already in use. Ensure no previous processes are running.")
    raise

except Exception as e:
    print(f"Unexpected error in ddp_setup: {e}")
    raise

But the issue which I am getting each time while running the code is that, the allotted port is showing as "already in use" whatever may be its number. I have attached the error block below:
[W socket.cpp:426] [c10d] The server socket has failed to listen on [::]:12356 (errno: 98 - Address already in use).
[W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:12356 (errno: 98 - Address already in use).
[E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
Traceback (most recent call last): File "/scratch/j20240138/final_rdnet.py", line 401, in mp.spawn(main, args=(world_size, args.save_every, args.total_epochs, args.batch_size), nprocs=world_size), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes while not context.join(): ^^^^^^^^^^^^^^
File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/scratch/j20240138/final_rdnet.py", line 369, in main ddp_setup(rank, world_size) File "/scratch/j20240138/final_rdnet.py", line 195, in ddp_setup init_process_group(backend="nccl", rank=rank, world_size=world_size, timeout=timedelta(minutes=60)) File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 900, in init_process_group store, rank, world_size = next(rendezvous_iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/distributed/rendezvous.py", line 245, in _env_rendezvous_handler store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/distributed/rendezvous.py", line 176, in _create_c10d_store return TCPStore( ^^^^^^^^^ RuntimeError: The server socket has failed to listen on any local network address. The server socket has failed to listen on [::]:12356 (errno: 98 - Address already in use). The server socket has failed to bind to 0.0.0.0:12356 (errno: 98 - Address already in use). srun: error: gpu010: task 0: Exited with exit code 1 Traceback (most recent call last): File "/scratch/j20240138/final_rdnet.py", line 401, in mp.spawn(main, args=(world_size, args.save_every, args.total_epochs, args.batch_size), nprocs=world_size), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 197, in start_processes while not context.join(): ^^^^^^^^^^^^^^ File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 0 terminated with the following error: Traceback (most recent call last): File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/scratch/j20240138/final_rdnet.py", line 369, in main ddp_setup(rank, world_size) File "/scratch/j20240138/final_rdnet.py", line 195, in ddp_setup init_process_group(backend="nccl", rank=rank, world_size=world_size, timeout=timedelta(minutes=60)) File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 932, in init_process_group _store_based_barrier(rank, store, timeout) File "/home/j20240138/miniconda3/envs/projenv/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 469, in _store_based_barrier raise RuntimeError( RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=2, worker_count=3, timeout=1:00:00) srun: error: gpu010: task 1: Exited with exit code 1
I need help in resolving this issue.

Have you tried to find the ports already in use and try to kill the processes on those ports? Of course, check if you need them... For example, you may be able to use lsof -i :<the-port> to find the processes that are using your port, and then kill -9 <process-id-that-is-the-PID-that-appears-when-you-use-lsof>. From the terminal of course

Yes I used lsof -i  :. But that doesn't give any output. So as per my understanding, it means that there are actually no processes that are using the port.

source: Stack Exchange

license: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/

sort: votes


text:
How to properly train a GAN model?
I tried to train a ViT-GAN model (from this repo) on my database, where i have images as input and output. The input image is a PNG map of a path planning problem. Red channel is obstacle map, green and blue is one pixel with start/goal. The output image would be the planned path on the red channel and this is what i am trying to teach the model. The model from the repo works nicely on the included examples and that is a much more complex problem than mine, so i naturally thought this network would solve my problem without much further tuning.
My issue is that even without touching the network, just replacing the training data to mine it completely messes up the output. I tried tweaking the network in so many ways, but the result always remains the same -> either the same structured garbage output for any given input (white patch on black), or some noisy rgb patches on black background. I just cant get it to learn the expected output.

On the left is the given input and on the right is the expected output. Middle is the output of the network. I read about training GAN's on i get they are volatile and hard to find the right structure/parameter set, but i just cant figure out what to do next with it. What i already tried:

Making the network smaller by removing some transformer/convulutional layers
Lowering the number of filters hence my problem is not that complex
Adjusting the learning rates of the generator/discriminator -> together and separately
Changing the activation function of the output layer of the discriminator to sigmoid as suggested in another similar post
Removed all ReLU activation functions and replaced them with LeakyReLU to address possible vanishing gradient problems
Adding Wasserstein loss to the discriminator loss function to avoid model collapse
Changing mean diff of pixels in the generator loss function to sum diff (to punish full black output more)
Use bias to avoid vanishing gradient
Changing the random initializer of filter from (0.0, 0.02) -> (0.0, 1.0) to make the initial filter more varied
Changing the lambda, batch size, ff_dim, number of heads, patch size, embed dim, projection dim params
Train for 200 epochs and for 20 also. All for the same output structure.
Train on smaller portion of dataset (400 images), and on larger (1000 and 3000 images)

I am aware that this problem could be solved with other (maybe simplier) netwroks also, but i would like to make this GAN work and understand why it wouldn't work in the first place. I added my current state of changes to this repo. I have no idea where to look next. If someone has some suggestions please feel free to share. If you suggest to adjust some params please write a concrete value. If i forgot anything from the description let me know.
Edit1: Some explanation regarding the included repositories. The train.py and the models.py files contain the logic. In my repository i removed the code from those files, which i dont use, eg. random jitter and random crop. The train.py contains the loss calculation and the learning rates, as well as the prepocessing steps of the data. I think there are no bugs in the code, but if you notice something please let me know. At the preprocessing steps i changed the decode_jpg to decode_png. The models.py file contains the structure of the model, where i commented out some parts of the original layers and removed the duplicate functions. Also i added a LeakyReLU layer in the transformer class and removed the ReLU activation function from the dense layer. Overall i operated with 64x64 RGB images, but i tried with the original 256x256 structure as well and got the same faulty results. In my repo i included 3 sample images from my training set as well. The evaluate.py file contains the metric calculation and image plotting of the results. There i also removed the random jitter and crop funcitons and in the pre_process function i changed the upscaling way from (output + 0.5) x 255 to (output + 1) x 127.5 . These three files can be found in the src folder of the original repository.

In general, I would suggest you need lot more data/examples. I don't remember doing GAN, but I think dataset size you suggested (3000 examples) is very small. If I were you, I would add lot more examples. Just to make sure what is the dataset size in the repo vs what is your dataset size? Also, make sure you've right dataset. Many a times, there will issues with dataset that no amount of training/tuning can fix!

I see. I will give it a try. The dataset included in the original repo has 3000 training images and under 100 epochs it learns well. I tried with a 400, 1000 and 3200 sized dataset as well. For the size of 3200 images i tried at the beginning, but i will try again and update here.

all_licenses: Creative Commons - Attribution Share-Alike - https://creativecommons.org/licenses/by-sa/4.0/


text:
What can I do with model that always overfits?
I need to recognize 35 classes (9 numbers and 26 English letters). I have 700 images, example. I split it 80% - train, 20% - validation.
My model is very small:
import torch
from torch.nn import Module, Conv2d, MaxPool2d, ReLU, BatchNorm2d, BatchNorm1d, Linear, LeakyReLU, Softmax, AdaptiveMaxPool2d, Dropout

class CNNBlock(Module):
    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.conv = Conv2d(
            in_channels = in_channels,
            out_channels = out_channels,
            kernel_size = 3,
            padding = "same"
        )
        self.batchnorm = BatchNorm2d(num_features = out_channels)
        self.act = ReLU()
        self.dropout = Dropout(0.8)
        self.maxpool = MaxPool2d(
            kernel_size = 2,
            stride = 2
        )

    def forward(self, x):
        return self.maxpool(self.dropout(self.act(self.batchnorm(self.conv(x)))))

class RecognitionModel(Module):
    def __init__(self):
        super().__init__()

        # Input 1x80x180
        self.block1 = CNNBlock(1, 32) # Block1 32x40x90
        self.block2 = CNNBlock(32, 64) # Block2 64x20x45
        self.gmp = AdaptiveMaxPool2d(output_size = 1) # GlobalMaxPool 64x1x1
        self.sqz = lambda x: x.squeeze() # Squeeze 64
        self.linear1 = Linear(
            in_features = 64,
            out_features = 32
        )
        self.batchnorm = BatchNorm1d(num_features = 32)
        self.act1 = LeakyReLU()
        self.dropout = Dropout(0.8)
        self.linear2 = Linear(
            in_features = 32,
            out_features = 35
        )
        self.act2 = Softmax(dim = 1)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.gmp(x)
        x = self.sqz(x)
        x = self.dropout(self.act1(self.batchnorm(self.linear1(x))))
        y = self.act2(self.linear2(x))

        return y

    def save(self):
        torch.save(self.state_dict(), './trainedmodel.pt')

    def load(self):
        self.load_state_dict(torch.load('./trainedmodel.pt', weights_only = True))

Also I have augmentations:
[
    Resize((80, 180)),
    RandomHorizontalFlip(),
    RandomRotation(20),
    RandomAffine(degrees=15, translate=(0.1, 0.1)),
    Grayscale(),

]

I also divide image by 255.
I tried to set dropout probability to 0.5 on linear, 0.2 on conv:

0.5 on linear, 0.5 on conv:

0.8 on linear, 0.8 on conv:

But it always overfits. I'm using Adam(lr=0.001); CrossEntropyLoss. Why does it happen?

Can you post your model code?

Without seeing your code, I could be missing something. However, simple CNNs like the one you posted above, are notoriously difficult to train. It could be possible that you just haven't found the correct hyper parameters. Consider creating a script that runs through many different combinations of learning rates, l2 regularization terms, and dropouts. If that doesn't work, it might be the case that you just need more data.

I have added code of model. When my model was bigger - it wasn't learning. I will try to experiment with hyper parameters, but I think it won't help. Everyone tells me, that I need more data. But how much data is good for my task? Thank you for your answer.

Bigger CNNs should be accompanied with skip connections, which improve gradient flow and smooth the loss surface significantly. Also, generally, if you find yourself backed into a corner after you have rigorously tested your training pipeline, it helps to shift from a build mindset to and understanding one. What is the distribution of image in the train and test set? What are my convolutional layers learning? What does a 2d projection of the output of my final linear layer look like?


text:
Are all (natural) knowledge representation formalisms created equal?
In a recent arXiv paper (accepted by AAAI-25):
https://arxiv.org/abs/2412.11855
(Title: A Theory of Formalisms for Representing Knowledge)
the authors present a seemingly intriguing result, claiming that all universal (or natural and equally expressive) knowledge representation formalisms are recursively isomorphic. To establish this result, the authors developed a framework to capture the class of possible knowledge representation formalisms and define what constitutes a universal knowledge representation formalism.
I am wondering if this work has any limitations. More specifically:

Is the framework proposed for knowledge representation formalisms defined in a reasonable and comprehensive manner?

Is the claim of recursive isomorphisms among universal (or natural and equally expressive) knowledge representation formalisms correct? If so, what are its implications for AI?

In the conclusion of this paper, the authors claim:

For the debate between symbolic AI and connectionist AI, the existence of recursive isomorphisms between knowledge representation formalisms (KRFs) implies that for any knowledge operator (e.g., gradient descent) in one KRF, we can effectively find an operator in another (isomorphic) KRF to perform the same transformation. From a theoretical perspective, all these representation methodologies either pave the way to AGI or none, with core challenges being universal and advancements in one methodology benefiting others.

However, it appears that knowledge representation formalisms based on neural networks have achieved a dominant position in contemporary AI. Therefore, does the above claim potentially overstate its significance?

Indeed the theoretical recursive isomorphism of knowledge representation formalisms (KRFs) may overstate the theoretical equivalence, overlooking practical differences in efficiency, scalability, and applicability. Thus while the claim is intriguing, it may not adequately address the strengths of the dominant connectionist methods in AI especially for fast pattern recognition in perceptual applications. Each formalism has their own applicability.

With the rise of deep learning, the symbolic AI approach has been compared to deep learning as complementary "...with parallels having been drawn many times by AI researchers between Kahneman's research on human reasoning and decision making – reflected in his book Thinking, Fast and Slow – and the so-called "AI systems 1 and 2", which would in principle be modelled by deep learning and symbolic reasoning, respectively." In this view, symbolic reasoning is more apt for deliberative reasoning, planning, and explanation while deep learning is more apt for fast pattern recognition in perceptual applications with noisy data.

In fact there's significant effort from symbolic AI's KRF to try to build a large KB about the world, focusing on implicit knowledge such as Cyc. It's an example of limited practical applicability and scalability of the pure symbolic AI formalism.

Cyc is a long-term artificial intelligence project that aims to assemble a comprehensive ontology and knowledge base that spans the basic concepts and rules about how the world works. Hoping to capture common sense knowledge, Cyc focuses on implicit knowledge... Cyc's ontology grew to about 100,000 terms in 1994, and as of 2017, it contained about 1,500,000 terms. The Cyc knowledge base involving ontological terms was largely created by hand axiom-writing; it was at about 1 million in 1994, and as of 2017, it is at about 24.5 million. In 2008, Cyc resources were mapped to many Wikipedia articles... The Cyc inference engine performs general logical deduction. It also performs inductive reasoning, statistical machine learning and symbolic machine learning, and abductive reasoning...  Machine-learning scientist Pedro Domingos refers to the project as a "catastrophic failure" for the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own.

In the conclusion of the paper, the authors argue that "the existence of recursive isomorphisms between KRFs implies that for any knowledge operator (e.g., gradient descent) in a KRF we can effectively find an operator in another (isomorphic) KRF to perform the same transformation." So, in this sense, are efficiency, scalability, and applicability truly inherent advantages of neural networks?

Since SGD+minibatch can train large stacked NNs such as transformers effectively without overfitting, NNs do have an advantage for training efficiency and scalability. Note the authors in your paper mentions the condition for recursive isomorphism: up to an offline compilation.  In theory this condition seems not a big concern, but as Cyc shown in my answer, this could have a big practical scalability issue if it wants to match latest LLMs' common sense and implicit knowledge representation and generalization, since all those 'hand axiom-writings' are offline compilation. Hope it clarifies.

Thank you very much for your response! So, my understanding is that whether a KRF that is recursively isomorphic to neural networks possesses good efficiency, scalability, and applicability depends on the computational efficiency of the aforementioned recursive isomorphism (i.e., the offline compilation). For instance, if this recursive isomorphism can be computed in linear time, then the relevant techniques of neural networks can be effectively converted to the target KRF.

You're on the right track. Intuitively it's not surprising for such isomorphism since anything can be programmed in declarative functional programming languages can be achieved in imperative procedural programming languages so long as they're Turing-complete, albeit some cases one type language can be much more efficient. Similarly there's a famous theoretical Curry-Howard Isomorphism between declarative constructive logic and procedural computation. The lesson so far is that everyday world common sense and implicit knowledge belongs to thinking-fast pattern recognition tasks including LLMs.

The real challenge here to transform SGD to its corresponding method in Cyc like symbolic AI KRF, so far we haven't found an efficient scalable self-evolving method without those offline compilation of hand axiom-writings for each microtheory within the whole world ontology. For example, if new data fact from a database gives a new type of fruit, the symbolic KRF's fruitFn() function has to be modified and recompiled which is clearly an overfitting case in the sense of statistical inference, thus is not generalizable compared to LLMs KRF.

I am uncertain whether the Curry-Howard Isomorphism plays the same role. It appears that this isomorphism concerns the relationship between proofs and computation, whereas the recursive isomorphism discussed in the aforementioned work, when placed in a similar context, pertains to the relationship between declarative and procedural representations of knowledge. These do not seem to be the same thing.

Constructive logic + type systems such as Coq/Agda does represent knowledge in a Turing-complete declarative way and most logics are inherently explicit declarative representation of knowledge, thus most logics are hard to capture implicit knowledge which can be better handled by connectionist procedural functional models. For example after seeing a cat and a dog which are explicit declarative knowledge representations it's hard for logic to infer an animal which is in between. Thus the recursive isomorphism in OP could be viewed as the type system isomorphic to NN KRF which approximates TM.

I understand your point, but such an analogy still seems somewhat superficial. Whether the two kinds of isomorphisms are indeed the same requires a more rigorous argument. Anyway, thank you for your response:)


text:
Should research on artificial consciousness be regulated?
With the advancement of a series of AI technologies, including ChatGPT and o3, it appears that the realization of Artificial General Intelligence (AGI) is imminent. Consequently, it might be time to consider the relationship between AI and humans. From a human perspective, the optimal role for AI would be as an unconscious tool to enhance our productivity, much like steam engines, electric motors in traditional industries, and computers in the information age.
Therefore, a natural question arises: Should we regulate research on consciousness within the field of artificial intelligence? This could be one of the most direct ways to prevent potential future scenarios where AI might lead to human extinction.

Regarding whether AI should merely serve as a tool, a recent paper discusses this issue:
https://www.tandfonline.com/doi/full/10.1080/0020174X.2024.243693
Why AGI could not be (just) a tool: goals, life, and general intelligence

It is widely believed that AGI has the potential to be a wonderful tool that humans can use to meet our needs, solve our problems, and improve our lives. Against this view, we argue that any entity with truly general, human-level intelligence would have the capacity to lead its own life, with its own purposes and integrated hierarchy of goals. And thus any true AGI could not be merely a tool, even if it turned out to be extremely helpful for human beings. If we are correct, there is a dilemma at the heart of the ambition to build AGI as a valuable tool. On the one hand, any mere tool that we might build would lack capacities essential to the kind of general intelligence exhibited by human beings; it would not be genuine AGI. On the other hand, were we to create genuine AGI, then what we would have created would not be a mere tool, but something more. In this paper, we make the case for this dilemma. In so doing, we illuminate the connections between a set of core ideas: intelligence, agency, tools, and life.

I hope this information is useful to you.


text:
Is there any theoretical work on representation in machine learning?
In AI, how knowledge is represented is a crucial topic. In traditional knowledge representation and reasoning, substantial work has focused on logic-based and graph-based knowledge representation methods. In contrast, in machine learning, knowledge is more often represented in parametric forms such as real-valued vectors (e.g., in neural networks and large language models).
Could you please tell me about any theoretical work in the field of machine learning that focuses on representation (especially, knowledge representation)?

Parametric models with vectors representing the knowledge of input features are not unique in machine learning (ML) at all, they've been used in traditional statistical inference and regression models as well for a long time such as the standard linear and logistic regression models. On the other hand, many other popular ML models are non-parametric such as decision trees, K-NNs, SVMs, and Gaussian Processes, etc, and all their input features knowledge are represented as potentially high-dimensional vectors.
Each knowledge representation formalism has their unique advantage and weakness, though recently some people argue that in theory they may be ultimately isomorphic as discussed in a recent interesting post. In general there're two main knowledge representation schools known as symbolic computationalism and distributed connectionism, and they've been around for many years and not seem a hot topic in contemporary ML or DL field, though there's a hot ML field called representation learning which is not about any target knowledge but input latent features' end-to-end representation and transformation for downstream tasks instead of the traditional manual feature engineering. Comparatively speaking, the connectionist neural network parametric distributed knowledge representation is extremely implicit, fully-connected, integrated and non-reducible as described to be like a blackbox by many people. You can further read Rumelhart, Hinton, et al's (1986) "Learning internal representations by error propagation".

Although our learning results do not guaantee that we can find a solution for all solvable problems, our analyses and results have shown that as a practical matter, the error propagation scheme leads to solutions in virtually every case. In short, we believe that we have answered Minsky and Papert's challenge and have found a learning result sufficiently powerful to demonstrate that their pessimism about learning in multilayer machines was misplaced. One way to view the procedure we have been describing is as a parallel computer that, having been shown the appropriate input/output exemplars specifying some function, programs itself to compute that function in general. Parallel computers are notoriously difficult to program. Here we have a mechanism whereby we do not actually have to know how to write the program in order to get the system to do it. Parker (1985) has emphasized this point.


text:
Is natural language reasoning the right way to implement reasoning in AI?
It is well known that human reasoning, after evolving for at least several thousand years, has gradually transformed from natural language reasoning to formal reasoning. In modern science, a significant indicator of a discipline's maturity is whether its main theories can be formalized (or axiomatized, mathematized). At the inception of AI, research on reasoning was primarily based on formal reasoning, which surprisingly achieved remarkable success early on. For instance, programs developed by Newell, Simon and Shaw in 1956, as well as Wang in 1960, proved almost all theorems in "Principia Mathematica" by Whitehead and Russell. Nowadays, state-of-the-art symbolic SAT solvers have become so sophisticated that attempting to achieve SAT solving via machine learning approaches cannot compete with them (Daniel Selsam et al. ICLR 2019, Learning a SAT Solver from Single-Bit Supervision):

As we stressed early on, as an end-to-end SAT solver the trained NeuroSAT system discussed in this paper is still vastly less reliable than the state-of-the-art. We concede that we see no obvious path to beating existing SAT solvers.

Given this, I'm wondering why we overlook the achievements of traditional symbolic reasoning and invest so much effort into implementing natural language reasoning within large language models. What are the advantages of natural language reasoning over formal reasoning?

There's mainly deductive formal reasoning, forward/backward chaining, and production rules chunking focused in the traditional symbolic AI field based on various mathematical logic theories, type theories, knowledge base, semantic ontology/network, and computability/complexity theories such as your SAT solver for Boolean satisfiability problem which is NP-complete in general by the Cook–Levin theorem in complexity theory. Obviously these specialized and powerful symbolic AI solvers or theorem provers are essential and important for many specialized fields such as pure mathematics, proof assistant, formal program and software verification, etc.
However, many real world consumer-facing AI applications don't require above sophisticated formal inferencing engines, yet require laymen-understandable flexible semantic composition and continuation when prompted, and many "reasonings" in natural language are implicit, informal, inductive, heuristically (no need for best) explainable, generalizable, and context driven, etc. All these requirements are turned out to be better solved by bottom-up learning-based contextually-embedded LLMs which are inherently incompatible with formal top-down reasoning models. Therefore it's not true that AI industry overlooks the achievements of traditional symbolic reasoning, foundational LLMs just seem to have much wider applicable horizon than a few highly specialized fields where symbolic reasoning is a must.
Of course people are constantly trying to leverage each formalism's advantage together to build some hybrid model like NeuroSAT, and certainly every kind of AI model, be it learning based or reasoning based, has their own use cases.


text:
Training T5 model on Spider and Gretelai dataset
I am working on this project for educational purposes. I definitely don't understand enough to find the flaw in my reasoning I expected textToSQL to be quite simple but here I am.
I want to build natural language to SQL queries agent and the model that seemed to be the most promising (both in reading and suggested by GPT) was the flan-t5 model. I intended to use the spider dataset, but after looking through it the data didn't seem to be enough so I added in the gretelai dataset by composing a common input format.
One thing about the format: I expected it to be more... general? For example I would expect to replace the name of the table with some kind of a placeholder so the model doesn't consider it and focuses only on the tokens that cause shift in the logic. Is there a better way than what i have displayed on the example?
I am considering adding a classifier which would enhance the data, for example I could try getting classifying the question into types of "simple_sql", "aggregation", "joins" etc. and giving that to the t5 model as extra metadata. Do you think that will matter?
I have been doing this mostly on the flan-t5-small model to speed the process up but have tried the flan-t5-base model on the most promising iteration with no significant improvements
Here is my code (from a jupyter notebook):
import pandas as pd

df = pd.read_parquet("./data/gretel_train.parquet")
df_test = pd.read_parquet("./data/gretel_test.parquet")
df  = df[["id", "sql_prompt", "sql_context", "sql"]]
df_test  = df_test[["id", "sql_prompt", "sql_context", "sql"]]

import re

def split_on_top_level_commas(definition: str):
    segments = []
    current = []
    paren_depth = 0

    for char in definition:
        if char == '(':
            paren_depth += 1
            current.append(char)
        elif char == ')':
            paren_depth -= 1
            current.append(char)
        elif char == ',' and paren_depth == 0:
            segments.append("".join(current).strip())
            current = []
        else:
            current.append(char)

    if current:
        segments.append("".join(current).strip())

    return segments

def map_col_type(raw_type):
    t = raw_type.lower().strip()

    if re.match(r'^(int|integer|float|real|double|numeric|decimal|smallint|bigint)', t):
        return "number"

    if re.match(r'^(char|varchar|text|longtext|tinytext|mediumtext)', t):
        return "text"

    if re.match(r'^(date|datetime|timestamp|time)', t):
        return "date"

    return "unknown"

def parse_gretelai_context(sql_text: str) -> str:
    create_table_pattern = re.compile(
        r"CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?([a-zA-Z0-9_]+)\s*\(\s*(.*)\)",
        re.IGNORECASE | re.DOTALL
    )

    statements = [stmt.strip() for stmt in sql_text.split(';') if stmt.strip()]

    schema_map = {}

    for stmt in statements:
        match = create_table_pattern.match(stmt)
        if not match:
            continue

        table_name = match.group(1).strip().lower()
        body = match.group(2).strip()

        schema_map[table_name] = {"columns": {}}

        col_defs = split_on_top_level_commas(body)

        for col_def in col_defs:
            fk_match = re.match(
                r"FOREIGN\s+KEY\s*\((?P<fk_col>[\w_]+)\)\s+REFERENCES\s+(?P<ref_table>[\w_]+)\s*\((?P<ref_col>[\w_]+)\)",
                col_def,
                flags=re.IGNORECASE
            )
            if fk_match:
                fk_col = fk_match.group('fk_col').lower()
                ref_table = fk_match.group('ref_table').lower()
                ref_col = fk_match.group('ref_col').lower()

                if fk_col in schema_map[table_name]["columns"]:
                    schema_map[table_name]["columns"][fk_col]["fk"] = (ref_table, ref_col)
                else:
                    schema_map[table_name]["columns"][fk_col] = {
                        "type": "unknown",
                        "is_pk": False,
                        "fk": (ref_table, ref_col)
                    }
                continue

            pk_constraint_match = re.match(
                r"PRIMARY\s+KEY\s*\((?P<pk_col>[\w_]+)\)",
                col_def,
                flags=re.IGNORECASE
            )
            if pk_constraint_match:
                pk_col = pk_constraint_match.group('pk_col').lower()
                if pk_col in schema_map[table_name]["columns"]:
                    schema_map[table_name]["columns"][pk_col]["is_pk"] = True
                else:
                    schema_map[table_name]["columns"][pk_col] = {
                        "type": "unknown",
                        "is_pk": True,
                        "fk": None
                    }
                continue

            parts = col_def.split(None, 1)
            if not parts:
                continue

            col_name = parts[0].lower()
            remainder = parts[1] if len(parts) > 1 else ""
            remainder = remainder.strip()

            is_pk = False
            if re.search(r'\bPRIMARY\s+KEY\b', remainder, flags=re.IGNORECASE):
                is_pk = True
                remainder = re.sub(r'\bPRIMARY\s+KEY\b', '', remainder, flags=re.IGNORECASE).strip()

            type_pattern = re.compile(r"^(?P<type>[A-Za-z]+(?:\(\d+(,\d+)?\))?)(?P<rest>.*)$")
            type_match = type_pattern.match(remainder.upper())
            if type_match:
                col_type = type_match.group('type').lower()
                rest = type_match.group('rest').strip()
            else:
                col_type = "unknown"
                rest = remainder.upper()

            col_type = map_col_type(col_type)
            if col_name not in schema_map[table_name]["columns"]:
                schema_map[table_name]["columns"][col_name] = {
                    "type": col_type,
                    "is_pk": is_pk,
                    "fk": None
                }
            else:
                schema_map[table_name]["columns"][col_name]["type"] = col_type
                schema_map[table_name]["columns"][col_name]["is_pk"] |= is_pk

    lines = ["<Schema>"]
    for table, table_info in schema_map.items():
        lines.append(f"-- Table: {table}")
        columns = table_info["columns"]
        for col_name in sorted(columns.keys()):
            col_type = columns[col_name]["type"]
            is_pk = columns[col_name]["is_pk"]
            fk = columns[col_name]["fk"]

            annotations = []
            if is_pk:
                annotations.append("PK")
            if fk is not None:
                ref_table, ref_col = fk
                annotations.append(f"FK -> {ref_table}.{ref_col}")

            ann_str = ""
            if annotations:
                ann_str = ", " + " ".join(annotations)

            lines.append(f"     - {col_name} ({col_type}{ann_str})")

        lines.append("")
    lines.append("</Schema>")

    return "\n".join(lines)

df["sql_context"] = df["sql_context"].apply(parse_gretelai_context)
df_test["sql_context"] = df_test["sql_context"].apply(parse_gretelai_context)

def convert_to_t5_input(user_query: str, schema_str: str) -> str:
    return f"""translate English to SQL:

<Query>
{user_query}
</Query>

{schema_str}
"""

def generate_t5_prompt(row):
    user_query = row['sql_prompt']
    schema_str = row['sql_context']
    return convert_to_t5_input(user_query, schema_str)

df['x'] = df.apply(generate_t5_prompt, axis=1)
df_test['x'] = df_test.apply(generate_t5_prompt, axis=1)

df['y'] = df['sql']
df_test['y'] = df_test['sql']

df_gretel_final = df[['x', 'y']].copy()
df_gretel_final_test = df_test[['x', 'y']].copy()

import pandas as pd

spider_df = pd.read_json("./spider/evaluation_examples/examples/train_spider.json")
test_spider_df = pd.read_json("./spider/evaluation_examples/examples/dev.json")
tables = pd.read_json("./spider/evaluation_examples/examples/tables.json")

def parse_spider_df(schema_info: dict) -> str:
    table_names = schema_info.get("table_names", [])
    column_names = schema_info.get("column_names", [])
    column_types = schema_info.get("column_types", [])
    primary_keys = schema_info.get("primary_keys", [])
    foreign_keys = schema_info.get("foreign_keys", [])

    table_to_columns = {i: [] for i in range(len(table_names))}
    for col_idx, (tbl_idx, col_name) in enumerate(column_names):
        if tbl_idx == -1:
            continue

        col_type = column_types[col_idx] if col_idx < len(column_types) else "text"
        table_to_columns[tbl_idx].append((col_name.replace(" ", "_"), col_type, col_idx))

    fk_map = {}
    for source_col, target_col in foreign_keys:
        fk_map[source_col] = target_col

    def get_col_annotations(col_idx: int) -> str:
        """
        Return a string describing if this column is a primary key or foreign key referencing something else.
        """
        annotations = []
        if col_idx in primary_keys:
            annotations.append("PK")
        if col_idx in fk_map:
            target_idx = fk_map[col_idx]
            t_tbl_idx, t_col_name = column_names[target_idx]
            t_col_name = t_col_name.replace(" ", "_")
            target_table = table_names[t_tbl_idx] if t_tbl_idx >= 0 else "unknown_table"
            target_table = target_table.replace(" ", "_")
            annotations.append(f"FK -> {target_table}.{t_col_name}")
        if annotations:
            return ", " + " ".join(annotations)
        else:
            return ""

    schema_lines = []
    for tbl_idx, tbl_name in enumerate(table_names):
        tbl_name = tbl_name.replace(" ", "_")
        schema_lines.append(f"-- Table: {tbl_name}")
        for (col_name, col_type, col_idx) in table_to_columns[tbl_idx]:
            anno = get_col_annotations(col_idx)
            schema_lines.append(f"     - {col_name} ({col_type}{anno})")
        schema_lines.append("")

    schema_str = "\n".join(schema_lines).rstrip()

    result = (
        "<Schema>\n"
        f"{schema_str}\n"
        "</Schema>\n"
    )

    return result

def convert_to_t5_input(user_query: str, schema_str: str) -> str:
    return f"""translate English to SQL:

<Query>
{user_query}
</Query>

{schema_str}
"""

def generate_t5_prompt(row):
    user_query = row['question']
    schema_str = row['schema']
    return convert_to_t5_input(user_query, schema_str)

contexts = tables[["db_id"]]

contexts["schema"] = tables.apply(
    lambda row: parse_spider_df(row),
    axis=1
)

spider_df = pd.merge(spider_df, contexts, on='db_id', how='left')
test_spider_df = pd.merge(test_spider_df, contexts, on='db_id', how='left')
spider_df['x'] = spider_df.apply(generate_t5_prompt, axis=1)
test_spider_df['x'] = spider_df.apply(generate_t5_prompt, axis=1)

spider_df['y'] = spider_df['query']
test_spider_df['y'] = test_spider_df['query']

df_final = spider_df[['x', 'y']].copy()
df_final_test = test_spider_df[['x', 'y']].copy()

from evaluate import load

training_args = TrainingArguments(
    output_dir="./flan_t5_modified_2.1_text2sql_results",
    learning_rate=2e-5,
    lr_scheduler_type="cosine_with_restarts",
    per_device_train_batch_size=4,
    num_train_epochs=5,
    weight_decay=0.02,
    warmup_steps=200,
    max_grad_norm=1.0,
    logging_steps=50,
    save_steps=300,
    gradient_accumulation_steps=2,
    report_to=["tensorboard"],
    logging_dir="./logs",
    fp16=False,
    log_level="info",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset['train'],
    eval_dataset=tokenized_dataset['validation'],
    processing_class=tokenizer,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=2, early_stopping_threshold=0.002)]
)

trainer.train()

And an example that results in SELECT rank FROM department WHERE MAX = MAX or something similar:
input_text = """
translate English to SQL:

<Query>
Find the largest ranking by selecting the MAX value.
</Query>

<Schema>
-- Table: department
     - department_id (number, PK)
     - name (text)
     - creation (text)
     - ranking (number)
     - budget_in_billions (number)
     - num_employees (number)

-- Table: head
     - head_id (number, PK)
     - name (text)
     - born_state (text)
     - age (number)

-- Table: management
     - department_id (number, PK FK -> department.department_id)
     - head_id (number, FK -> head.head_id)
     - temporary_acting (text)
</Schema>
"""

inputs = tokenizer(input_text, return_tensors="pt").to("mps")

outputs = model.generate(**inputs, max_length=1000)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```


text:
chained linear regression models vs feed forward NN
I am trying to understand the difference between feedforward NN and chained linear regression models, if and why they can model nonlinear functions.

both are able to model non-linear dependencies
feedforward NN without activation functions cannot model nonlinear dependecies and number of layers does not matter in this context
chained regression models can model nonlinear dependencies

I am confused because above statements comes from chatgpt and I am not sure if all are correct I believe they are mutually contradictory, because from what I know feedforward NN without activation function can be cosidered as chained linear regression models.

I assume by Chained Linear Regression (CLR) you mean something like $f_1(f_2(x)) = y$, where the functions $f_i$ would be your Linear Regression models. In that case, CLR cannot model non-linear dependencies, as the composition of two linear functions is itself linear.
For the same reason, your second point is correct.
And to answer your question: A multilayer perceptron (i.e. feedforward NN with multiple layers) consists of multiple layers and each layer is just a matrix multiplication. In CLR, each of your functions $f_i$ is also just a matrix multiplication (the parameters of these matrices are optimized during training). So in that sense, a multilayer perceptron without non-linearities and chained linear regression are the same thing. However, the data may flow a bit differently through a chained linear regression, so in the exact implementation, the models may still differ a bit, just not in their ability to model non-linear dependencies.

"So in that sense, a multilayer perceptron without non-linearities " by non-linearities you meant without activation function, if this is correct can I simplify that CLR + activation function = feedforward NN ?

"by [without] non-linearities you meant without activation function", exactly!
To your question: "CLR + activation function(s) = feedforward NN + activation function(s)" If you say it like this, then it's correct. Note though that both feedforward NN and CLR are families of models: A feedforward neural network is everyneural network that doesn't contain recurrent connections (neurons feeding back information to an earlier layer) but it can take many forms. It only describes that information flows onyl one way. Multilayer perceptron is a better description of what you mean here I guess.


text:
How do I deal with dynamic, parameterized, action spaces?
I want to design an AI Learning Algorithm for a Student made, round based Game.
Let me first explain the Game/Environment
You have a round based HTTP Game, in which multiple Players can participate.
Each player has the option within a round to:

Buy new Robots (Or don't)

Additionally he has the option FOR EVERY ROBOT he owns to either:

Mine (If on a Planet that has a Resource)
Regenerate (Gives him Energy back, required for moving/attacking)
Attack (If Enemies are on the same planet, he can choose to attack a specific target, by providing the Id of the target.)
Move (LEFT,RIGHT,NORTH,SOUTH)
Buy one of the following ( Health Restore, Energy Restore, Mining Upgrade, MiningSpeed Upgrade, Storage Upgrade, MaxHealth Upgrade, Damage Upgrade, EnergyRegen Upgrade, MaxEnergy Upgrade )

So an example of an action within one Round could be, assuming Player A owns 2 Robots.
1. Action: [Buy 3 more Robots, Robot1: MoveLeft, Robot2: Mine]
in the next round it could look like this.
2. Action: [Robot1: Mine, Robot2: Mine, Robot3: MoveLeft, Robot4: Mine, Robot5: Buy Damage Upgrade]
To give a better overview, this is basically how a Map could look like.

As state I have:

Money a player owns
Positions and All Infos of EVERY Robot in the Game (Including Enemy Player Robots)
Everything there is to know about a Robot (e.g. Health, Upgrade Levels, current Resources in Inventory)
Positions of Planets (Including Resource amount, and which Robots are on it)

The Map has a fixed Size, but the amount of robots varies throughout the rounds played.
A next Round starts, once every Player has submitted a command for every Robot they own.
The Goal?
Make the most Money until the end of the game (you define a maximum amount of Rounds beforehands, e.g. 500)
Question
How can I possibly create an AI Player for this game?
Actions are dependent on one and another. For example if a player has 1000 Money, and he decides to spend all this Money on buying New Robots, then the actions for the robots he has, cannot include upgrading, because he used the money up already, so the action space depends on the previous actions chosen.
I know it is not a simple Environment but I would highly appreciate if someone could give me some hints about how to proceed with this.


text:
Calculating class-specific permutation feature importances for multilabel classification problem
I would like to apply the permutation feature importance technique to rank the features of a siamese network model that I trained. I am currently using this siamese network to perform some kind of classification, where I have many items coming from different categories, and I consider an item correcly classified if the network gives the highest similarity with another item from the same category. I would be interested to understand which features are determining the similarity for each class.
My thought was to actually calculate the permutation feature importances on my test data, but divided by class (so basically, if I have N features and M classes, I would get M different importances for each of the N features).
Does it make sense? Or are you aware of other suitable methods?


text:
How does the Yolo loss match bounding boxes to the ground truth?
I've been going over the YOLO paper again and I was wondering something about the loss.
Yolo divides an image into grids and then for each grid can have multiple bounding boxes to detect multiple objects whose centers fall into the same grid.
My issue with this is that there can be an issue with ordering.
For instance let's say within a grid, there are two object centers. Let's say the ground truth is the vector (a,b) where 'a' is the coordinates and class for object A and 'b' is for object B.
If the model outputs the vector (b,a) it is essentially right but due to ordering it will be punished.. is there anything that addresses this possible issue within the paper or newer versions of YOLO?
For instance DeTR uses the Hungarian method to get a bipratite matching between predictions and the closest ground truth label.
Edit:
I think I found my answer within the paper,
So before calculating the loss, yolo does do a matching between predictions and ground truth boxes. The prediction with the highest Intersection over Union (IoU) is chosen the the box "responsible" for that detection and the loss is done between that prediction and the ground truth.


text:
Implementation difference in REINFORCE algorithm, where to sum from
I have a question regarding the implementation of the REINFORCE algorithm.
In berkeley course (see slide 9) the gradient is defined as

Note that the return sums from 1. However in Sutton's book the return sums from t+1

In fact in two popular implementations, pytorch's official example (See Line 71) and tianshou's example (See Line 226), if I understand correctly, they are computing
sum_t (∇log π(a_t | s_t) * R_t)

which seems to correspond to Sutton's definition rather than Berkeley's (which will becomes
(sum_t ∇log π(a_t | s_t)) * (sum_t r_t)

).
I wonder which definition is the de facto standard in the community. Thank you.

The policy gradient objective function of your Berkeley reference is undiscounted value function of a start state in finite horizon episodic cases, while the other Sutton's reference at least for the boxed pseudocode is about discounted value function of a start state in finite horizon episodic cases as stated in the book:

From here on in our discussion we will assume no discounting ($\gamma = 1$) for the episodic case, although for completeness we do include the possibility of discounting in the boxed algorithms.

Since the undiscounted total return $G_t$ is defined as $\sum_t^T r_t(s_t,a_t)$, there's no essential notation difference here. Berkeley's equation is an approximation expressed via samples $i$, so it may not be easy to use $G_t$ without further unnecessary confusions such as $G_{t,i}$.

I think I understand now. The implementation uses a reward to go version, which can be proved to be equivalent to the original loss, see here.


text:
Single text embedding vs. bool[] for feature(s)?
I am creating a recommendation engine to recommend events (similar to meetup.com). Each event has 1 Interest and 0..N Tags (the tags are similar to SO tags). Should I enter these as a single text entry that is embedded? Or have a bool[] with it set to true for the matching entries?
The bool[] gives the AI services a cleaner set of data. But this means for an event it will have 55 features, 40 of which are the interests & tags. Will that dominate all NN comparisons - that would be bad.
Update: To explain more without violating what's allowed in a question. My approach for a recommendation engine is to create a vector of every user and of every event. The event vectors will include the Interest & Tags for the event, along with some other bools, numbers, and some text that is converted into embeddings.
Them to get recommendations for a user, I do two things: First, I do a nearest neighbor lookup of the users closest to their user vector. And then get the events that user has signed up for.
Second, if the user has signed up for events, I do a nearest neighbor of the events they signed up for and get those events.
I then take the 30 best matches across both lists of events the user has not signed up for yet, and return that.

Hello. Welcome! You're not explaining which algorithm or model you're trying to use. It seems you're using some AI service. It might be a good idea to mention which one and how you're calling this service, etc., but bear in mind that asking about how to use specific software is off-topic. More details here: https://ai.stackexchange.com/help/on-topic

@nbro Added more detail, without specifying the software used. If you need more, please ask - it's difficult figuring out how to ask without listing the services used.


text:
Application of AI for chat-rooms to process data
I have a software that is designed for students to have chat rooms with their classmates, and I want to process the datas being exchanged in the chat-rooms using AI model. The issue is, I don't know the specific model that is best fit or generally good fit for this type of datas. The chat-room can be considered as a normal discord channel without any toxicity, but I would like to learn the patterns, the trending topics or conversations, the common converstations they are having. What are the recommendations?
This is my idea and please be understanding of my extend of my knowledge. I am trying to figure out ways myself and I thought I would ask the community for directions. The software is for students that autamatically assigns a chat-room per class they are enrolled in, and their classmates are within the same chat-room(given they have same exact class).
I want to process all and any type(if the existing technology allows my greed) of data coming from the chatrooms and display the processed datas and use it as necessary. The AI model I have in mind is a passive one that simply processes the datas and outputs the processed version. You may say this is too vague, but I don't know what directions to look.

Can you please put your specific question in the title? "Application of AI" is very generic and not a question. Moreover, I'd recommend you tell us your knowledge of AI or machine learning and what you've found so far, if you have done any research, which I assume you did.

Could you look at it again and see if you can add more? @nbro

So are you looking for a tool/model that in a way summaries the chat room conversations or gives you the main topic of the conversations? Could you please further change the  title to ask this specific question? Anyway, if my assumption is correct, you may be interested in "topic modelling", although you could also try to use the OpenAI models. I think may may be able to do this job, although you will first need to fetch the data from the chat rooms before giving it to the models, which you need to do anyway.

This is a bit of guess, given you vague question, but you might be interested in doing the following:

first is to embed the separate discussions using an embedding model (word2vec, BERT, Ada2)
Then cluster these embeddings, you could use K-means clustering,  in order to figure out what the main groups of conversations are.
You could also perform NER (Named Entity Recognition), in order to determine the main key words in the debates.
Once these are done, you can analyse the data since the key words found from NER can be used to classify the exchanges (through presence, or through semantic similarity using the embeddings)

Alternatively (less technical):

you could ask an AI to summarize each exchange first.
Then given this list of summaries, ask an AI to give you a list of topics. (you did the summaries before in order to shorten the token limit)
Finally, for each chat, you can ask an AI to classify if it belongs to any one of the previously found topics.

Hopefully this vague answer will spark a lingr series of corrections that will be helpful.


text:
Tabular Q-Learning: Is a variable for "action_history" needed for backpropagating the q-value for all previous actions?
I am implementing a Tabular Q-Learning algorithm in Python and have questions regarding the use of an 'action_history' variable.

Necessity of 'action_history': In Q-Learning, the Q-value update is typically based on the immediate action and its outcome. Is there a benefit in maintaining a history of all actions taken in an episode to backpropagate the updated Q-value to these actions? Would this approach enhance learning, or does it contradict Q-Learning principles?

Q-value Update Mechanism: If an 'action_history' is beneficial, how should the Q-value be updated across multiple past actions? Is there a standard method or formula for distributing the updated Q-value to previous actions?

Q-Table Appearance After One Episode: If I set the learning to only one episode, how would the Q-table typically look? Would only the last action have a non-zero Q-value, with others being updated in subsequent episodes? For example, would it look like this:

    "('XO__XOXO_', 8)": 0.2,
    "('X___XOXO_', 1)": 0.038000000000000006,
    "('X____OXO_', 4)": 0.007220000000000002,
    "('X_____XO_', 5)": 0.0013718000000000003,
    "('______XO_', 0)": 0.00026064200000000006,
    "('______X__', 7)": 4.952198e-05,
    "('_________', 6)": 9.4091762e-06
}

or would it look like:
{
    "('XO__XOXO_', 8)": 0.2,
    "('X___XOXO_', 1)": 0,
    "('X____OXO_', 4)": 0,
    "('X_____XO_', 5)": 0,
    "('______XO_', 0)": 0,
    "('______X__', 7)": 0,
    "('_________', 6)": 0
}

I'm looking to understand the standard practices and the theoretical justification behind these aspects of Q-Learning.
Edit: I will add my Python code, in which I'm trying to implement tabular q-learning for 3x3 tictactoe. The q-table for a single episode looks like the following. I don't know how I need to update the value of the moves that led to the last move:
{
    "('_________', 3)": 0.002,
    "('___X_____', 6)": 0.002,
    "('___X__O__', 5)": 0.002,
    "('___X_XO__', 7)": 0.002,
    "('___X_XOO_', 1)": 0.002,
    "('_X_X_XOO_', 8)": 0.2
}

class Mark(enum.StrEnum):
    CROSS = "X"
    NAUGHT = "O"
    EMPTY = "_"

class Reward(enum.Enum):
    WIN = 1
    LOSE = -1
    TIE = -0.065
    NON_TERMINAL = 0.01

# Constants for Q-Learning
EPSILON = 0.9  # Exploration factor
ALPHA = 0.2  # Learning rate
GAMMA = 0.95  # Discount factor

TOTAL_EPISODES = 1  # Total number of games the agent will play

BOARD = np.array([Mark.EMPTY] * BOARD_SIZE)

def empty_count(board: np.ndarray) -> int:
    return np.count_nonzero(board == Mark.EMPTY)

def update_q_table(board, action, reward, new_board):
    board_key = "".join(board)
    new_board_key = "".join(new_board)

    old_value = Q_TABLE_DICT.get((board_key, action), 0)

    if game_over(new_board):
        # If it's a terminal state, there are no future rewards to consider
        next_max = 0
    else:
        # Estimate the optimal future value
        next_max = max(
            Q_TABLE_DICT.get((new_board_key, a), 0) for a in possible_moves(new_board)
        )

    temporal_difference = reward + (GAMMA * next_max) - old_value
    # Update the Q-value for the current state-action pair using the bellman equation
    new_q_value = old_value + (ALPHA * temporal_difference)
    Q_TABLE_DICT[(board_key, action)] = new_q_value

def train_q_learning_agent():
    for episode in range(TOTAL_EPISODES):
        board = np.array([Mark.EMPTY] * BOARD_SIZE)  # Reset the board
        current_mark = Mark.CROSS  # Start with X (or choose randomly)

        while not game_over(board):
            # Q-learning agent (X) makes a move
            # if current_mark == Mark.CROSS:
            action = choose_action_q_learning(board, training=True)
            new_board = make_move_to(board, action, current_mark)
            reward = get_reward(new_board, current_mark)
            update_q_table(board, action, reward, new_board)

            board = new_board
            current_mark = Mark.NAUGHT if current_mark == Mark.CROSS else Mark.CROSS
        if episode == 1200:
            EPSILON = 0.1

def choose_action_q_learning(board, training=True) -> int:
    if training and random.uniform(0, 1) < EPSILON:
        # Exploration: choose a random action
        return np.random.choice(possible_moves(board))
    else:
        # Exploitation: choose the best action based on current Q-table
        board_key = "".join(board)
        q_values = {
            action: Q_TABLE_DICT.get((board_key, action), 0)
            for action in possible_moves(board)
        }
        return max(q_values, key=q_values.get)

As an aside, the act of taking knowledge of outcomes and using it to assign values to previous state or actions is not called backpropagation in the literature. It is called backup, as in backup diagram. I will use that terminology in the answer. You may find the distinction useful in future when you have systems that are part of reinforcement learning that do also use backpropagation (as part of training a neural network).
Single-step Q learning is simple to implement and can be effective. After a single episode of TicTacToe, it may look like your second example, with only the state immediately preceding a reward storing any value. However this is not generally true. Q learning has some interesting "search" behaviour in environments where state can be revisited, and where non-zero rewards may occur other than at episode termination. Finding the exit to a maze, with reward of -1 per timestep, can be a good example of this, with the immediate updates per timestep in Q learning offering a large advantage over waiting until the end of the episode (as in Monte Carlo control).
However, there can be more effective versions using multiple steps, and there are several variants of this idea. A common one, and a core idea in RL, is eligibility traces, leading to the algorithm Q($\lambda$) which is a generalisation of the idea of assigning reward further back in time, but with less certainty, based on the parameter $\lambda$ which varies between $0$ for considering immediate effects only, to considering everything to the end of an episode. Single-step Q learning can be called Q(0) in reference to this algorith, and Q(1) has similar behavior to Monte Carlo control.
Working with Q($\lambda$) means storing a history of actions and states either directly or more commonly summarised in the trace (which also decides how much to backup), in order to assign value updates via a backup. It could lead to an update in TicTacToe similar to your first example, after a single episode. In some environments this can significantly increase the sample efficiency from gathering experience by the agent. However, don't underestimate the power of simple fast updates and collecting more samples, especially in simple simulations.

Is it possible to have a Q-Table that looks like the first example after a single episode when just using TD learning? Tbh, I never heard of ET before, but that is because I'm new to RL. I always thought that if there is only one episode of training that then the last move has a q-value and all its previous moves that led to the last move have a q-value that is calculated based on the last moves q-value. And one thing I also don't understand is if wether the q-values are updated from last move to first move or the other way around.

Furthermore, I still don't really get one more thing:
Let's say, I'm implementing tabular q-learning for tictactoe. Would the updating mechanism be in a forward or backward manner?  (i.e., from the start of the episode to the end), the first state-action pair that would have its Q-value updated would be the first one encountered in the episode. In the case of Tic-Tac-Toe, this would typically be the empty board state if the agent is the first to move. Or in a backward manner? In this case, the last state-action pair in the episode would be the first to have its Q-value updated.

@Hans You seem to be asking more and different questions in comments, and to have edited your first question to add code and asking for help with it in the main question. It would be better instead to leave the original question as-is, and only ask clarifying questions about this answer here. If you want to follow up with more detail, then it is better to ask a new question.


text:
tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table
i implemented the tabular q-learning algorithm for 3x3 tictactoe multiple times and everytime the number of entries in the q-table is 16,167. I wanna know how to calculate the number of 16,167. what is the formula to calculate it. the q-learning-algorithm also uses all symmetries. 0,90,180,270 degree rotations, horizontal, vertical, diagonal, antidiagonal.
i know that a 3x3 Tic-Tac-Toe board can be in any combination of three states (empty, X, O) across 9 positions which means that it has
3⁹=19,683 possible configurations
can anybody please help me get the formula to calculate it?
1st EDIT:
An example entry of my q-table:
('_XXX_O_OO', 6): 0.9024999999999993
2nd EDIT:
"('_________', 2)": 0.06166253080818989,
"('_________', 8)": 0.06166253080818989,
"('_________', 0)": 0.06166253080818989,
"('_________', 6)": 0.06166253080818989,

"('_________', 3)": 0.030324239570976224,
"('_________', 7)": 0.030324239570976224,
"('_________', 5)": 0.030324239570976224,
"('_________', 1)": 0.030324239570976224,

"('_________', 4)": 0.15534933785554575,

I save the board and all of its symmetries inside the q-table.
3rd EDIT:
my update_q_table function:
def update_q_table(
    board,
    action,
    reward,
    new_board,
    alpha,
    gamma,
    future_value,
):
    global Q_TABLE_DICT
    symmetric_states_and_actions = generate_symmetric_states_and_actions(board, action)

    old_value = Q_TABLE_DICT.get((board, action), 0)

    if game_over(new_board):
        # If it's a terminal state, there are no future rewards to consider
        next_max = 0
    else:
        next_max = future_value
    temporal_difference = reward + (gamma * next_max) - old_value
    new_q_value = old_value + (alpha * temporal_difference)
    # print(new_q_value)
    # Update the Q-value for the current state-action pair and its symmetric states
    for symmetric_state, symmetric_action in symmetric_states_and_actions:
        Q_TABLE_DICT[(symmetric_state, symmetric_action)] = new_q_value

    return new_q_value

This is going to depend on the representation and approach you used. For example, are you using state-only representation, or state/action? Please share the representation used in your Q table. Use [edit] to add it to the question

I added an example entry of my q-table

Thank you. That's not quite what I expected, so I am not quite sure how you are normalising by symmetry. Assuming you start with _________ and agent is selecting actions in $[0,8]$, the agent could play actions 1, 3, 5 or 7 which all result in the "same" board. How is that resolved? I.e. how is the canonical next state decided out of _X_______, ___X_____, _____X___ and _______X_?

Are you searching for existing entries that match by trying all possible transformations?

I added 2nd and 3rd EDIT. Do these answer your questions?

Are you searching for existing entries that match by trying all possible transformations? – No, but lookup in the dictionary via the (board, action) tuple.

Looks like you are instead storing all possible combinations, and using the symmetry to manage table updates? Oddly enough, that makes the calculation easier, because your symmetry handling is not changing the count at all :-)

Looks like you are instead storing all possible combinations, and using the symmetry to manage table updates? - Yes

cuz, i didn't know how to use canonical states for a (board, action) tuple. for minimax algorithm it worked fine, cuz I only had the board as key and either -1,0,1 (rewards) as value. but here i have a tuple of board,action.

please, let me know if you have any ideas how to get to the 16,167 :)

The bad news is that there is no simple formula for calculating the state space or state+action space here. It is relatively easy to get upper or lower bounds to it, but complex to precisely calculate. Writing a maths formula for it would look like pseudocode but written in maths notation - a procedure, rather than a simple expression.
Your calcuation of $3^9$ is an overestimate of the state space (i.e. ignoring actions), based on the full gamut of the state representation. It's an overestimate because it includes many impossible unreachable states such as XX_______ or ____O____ where the turn sequence has not been followed. You could extend this representation space analysis though, by multiplying by all 9 actions to arrive at $9 \times 3^9 = 177141$. This is an upper bound on the size of your Q table. As well as impossible states, it will include incorrect state and action combinations such as (____X____, 4).
Another way to arrive at a strict upper bound is to count all the nodes in the game decision tree. At the start, player X must make one of 9 decisions, player O must take one of the remaining 8 etc. This leads to the upper bound of $\sum_{t=0}^8 \prod_{c=9-t}^9 c$ (where $c$ is the number of choices available at time step $t$). This resolves to $986409$ - higher than the number of representations! That's because many of the combinations end up in the same state - action sequence $(4,5,0)$ is the same board as $(0,5,4)$. This will also include states that continue play after X or O have won.
You can refine calculations by filtering, but there is no simple formula for that, because you have to apply the game rules, including turn-taking and winning conditions, to determine whether a state and action is valid. You will need to script something that generates one of the larger upper bound sets, then filter logically.
I would suggest that the easiest calculation here is to start with all possible state representations and filter as follows:

Remove all states where count of X is either less than count of O, or is more than $1$ higher.
Remove all states which show either X or O (or both!) completing a line, because no action is possible from those states, so they are never added to the Q table.

Then multiply each state by the count of _, and sum to get number of actions, thus expected size of a completed Q table. Note this correctly assigns a count of $0$ for full boards for drawn games, they won't appear in the Q table (it doesn't matter if you also want to filter these out).
I have tested this with a short script and obtained the number $16167$. So you can have some confidence that your learning algorithm is exploring all options.

I would really like to see the short script you wrote to obtain the number of 16167. Would you like to publish that somewhere for others to view?

Great answer from @Neil Slater; however, I wanted to give my 2 cents.
Even though it's called "tabular" q-learning, it doesn't imply that you have to use a table to access it; yes, a table might have some advantages, but there are other data structures out there
In particular, you can use a map instead (key-value data structure, equivalent of a dictionary in python) where you pass some sort of "string representation of your board" as key, and have as value an array of N possible moves
Now the question is, do we really want to save that little memory, paying the additional cost of a map, or we can live with a sparse table with entries that are never used? I'd say the latter, as it's much easier to vectorize accesses and updates of such table
Furthermore, if memory is your concerns, consider that tictactoe has a ton of symmetries: you can flip the board horizontally, vertically, rotate it, and flip the entries (X instead of O and viceversa) which will definitely save a lot of memories


text:
Using computer vision to comprehend Piping and Instrumentation Diagrams
I'm wondering how to approach this problem. I want to create an excel document (or just a dictionary) where each instrument (circle objects FV 1031, for example) is associated with a line number (for example 2"-AL-131-14060-B4A2-IH).
 And also recognize line breaks like this:

I'm trying to find a good approach.


text:
Does ResNext split data or copy it?
I have been learning how to create ResNext neural networks, and am confused how input works with cardinality. In this answer, it seems that it's saying that the data is added together, which I assumed meant some sort of aggregation like element-wise summation, but I've ready other things
that say that ResNext undergoes split-transform-merge, the same that inception network uses. I also was looking at this Github repo that is a ResNext model in PyTorch, and while I don't completely understand it, it seemed like it just had multiple convolutions working on different sections of the data, instead of running multiple copies of the data through separate convolutions. This is the section I was looking at:
        width_ratio = out_channels / (widen_factor * 64.)
        D = cardinality * int(base_width * width_ratio)
        self.conv_reduce = nn.Conv2d(in_channels, D, kernel_size=1, stride=1, padding=0, bias=False)
        self.bn_reduce = nn.BatchNorm2d(D)
        self.conv_conv = nn.Conv2d(D, D, kernel_size=3, stride=stride, padding=1, groups=cardinality, bias=False)
        self.bn = nn.BatchNorm2d(D)
        self.conv_expand = nn.Conv2d(D, out_channels, kernel_size=1, stride=1, padding=0, bias=False)
        self.bn_expand = nn.BatchNorm2d(out_channels)

What I would have expected to see (and I am very naive about this), would be concatenating the data to itself cardinality times, and then using cardinality groups.
So, does ResNext send identical copies of data through separate branches, or split it into sections and run those sections through separate branches?

Your second link doesn't work on my end. It seems to be a Bing link -- if it's search result, can you replace it with a link to the original website?

Also, for the third link it'd be more clear if you added the specific snippet of code that you're referring to in your question.

My bad, sorry. I just copied the link from the edge search results after I found it, but apparently that doesn't actually copy the link. I fixed it and added some code.

I realized that groups in PyTorch works along the same dimension as channels, I thought it worked along the same dimension as the data. I believe the GitHub example does run the same data through each group, it just has an initial convolution to increase the channel length.


text:
Separation property of general uninform graph search from Artificial Intelligence: A Modern Approach Textbook by Peter Norvig and Stuart J. Russell
Before I start, I'm deeply sorry if my grammar is incorrect or hard to understand, since I haven't trained my English for a long time. Thanks you for spending time and effort to read and answer my question.
I'm trying to understand the solution for exercise 3.13 of Artificial Intelligence: A Modern Approach Textbook by Peter Norvig and Stuart J. Russell. However, the author's solution from his solution manual has a lot of ideas that are unclear to me. So please help we to correct my understanding and explain what the author really means in his solution.

The exercise 3.13
The GRAPH-SEARCH algorithm has another nice property: the frontier separates the state-space graph into the explored region and the unexplored region, so that every path from the initial state to an unexplored state has to pass through a state in the frontier. Prove that GRAPH-SEARCH satisfies the graph separation property.
Here is the GRAPH-SEARCH algorithm:
function GRAPH-SEARCH(problem) returns a solution, or failure
  initialize the frontier using the initial state of problem
  initialize the explored set to be empty
  loop do
    if the frontier is empty then return failure
    choose a leaf node and remove it from the frontier
    if the node contains a goal state then return the corresponding solution
    add the node to the explored set
    expand the chosen node, adding the resulting nodes to the frontier
      only if not in the frontier or explored set

The solution from solution manual
At the start of the search, the frontier holds the initial state; hence, trivially, every path from the initial state to an unexplored state includes a node in the frontier (the initial state itself).
Now, we assume that the property holds at the beginning of an arbitrary iteration of the GRAPH-SEARCH algorithm in Figure 3.7. We assume that the iteration completes, i.e., the frontier is not empty and the selected leaf node n is not a goal state. At the end of the iteration, n has been removed from the frontier and its successors (if not already explored or in the frontier) placed in the frontier. Consider any path from the initial state to an unexplored state; by the induction hypothesis such a path (at the beginning of the iteration) includes at least one frontier node; except when n is the only such node, the separation property automatically holds. Hence, we focus on paths passing through n (and no other frontier node). By definition, the next node n′ along the path from n must be a successor of n that (by the preceding sentence) is already not in the frontier. Furthermore, n′ cannot be in the explored set, since by assumption there is a path from n′ to an unexplored node not passing through the frontier, which would violate the separation property as every explored node is connected to the initial state by explored nodes (see lemma below for proof this is always possible). Hence, n′ is not in the explored set, hence it will be added to the frontier; then the path will include a frontier node and the separation property is restored.
Lemma: Every explored node is connected to the initial state by a path of explored nodes.
Proof: This is true initially, since the initial state is connected to itself. Since we never remove nodes from the explored region, we only need to check new nodes we add to the explored list on an expansion. Let n be such a new explored node. This is previously on the frontier, so it is a neighbor of a node n′ previously explored (i.e., its parent). n′ is, by hypothesis is connected to the initial state by a path of explored nodes. This path with n appended is a path of explored nodes connecting n′ to the initial state.

My understanding about the solution

"We assume that the iteration completes, i.e., the frontier is not empty and the selected leaf node n is not a goal state". The author means the arbitrary iteration that he assumes to hold the property, which when completes will not lead to end of loop.
"By definition, the next node n′ along the path from n must be a successor of n that (by the preceding sentence) is already not in the frontier". The author means at the end of the iteration, a new node n' will be added to the search tree with the Parent pointer pointing to node n.
"By definition, the next node n′ along the path from n must be a successor of n that (by the preceding sentence) is already not in the frontier. Furthermore, n′ cannot be in the explored set, since by assumption there is a path from n′ to an unexplored node not passing through the frontier, which would violate the separation property as every explored node is connected to the initial state by explored nodes (see lemma below for proof this is always possible). Hence, n′ is not in the explored set, hence it will be added to the frontier; then the path will include a frontier node and the separation property is restored". Here, the author is trying to prove that n' must be added to the frontier by proving that n' cannot already be at the frontier nor in the explored set. However, if n' is added to search tree as successor of node n, then n' mustn't be in explored set or already at frontier. So why did the author has to reprove n'?
"Furthermore, n′ cannot be in the explored set, since by assumption there is a path from n′ to an unexplored node not passing through the frontier, which would violate the separation property as every explored node is connected to the initial state by explored nodes". How does this help proving n' isn't in explored set?

Deeply thank for your answers and replies.

This post seems to contain multiple questions and seems to be unnecessary long. Can you please edit the post to leave just one question and move the others to a different post? Also, please, put your question in the title. "Separation property..." is not a question.


text:
VAE ( variational autoencoder) for timeseries anomaly detection ,
I am implementing VAE based anomaly detection for multivariate timeseries using keras,
I have ELBO (Evidence lower bound) which is combination of
$$-\ D_{KL}\left({\ q}_\varphi\left(z\middle| x^i\right)\parallel p_\theta\left(z\right)\ \right)\ +\ \ E_{q_\varphi(z|x^i)}\left[\log{p_\theta\left(x\middle| z\right)}\right]$$
The loss function is combination of minimizing the KL[Q(z|x)|p(z)] and maximizing log [P(X|z)]
how do I implement the above loss function for the multivariate timeseries of shape (batch_size,time_steps,features) using Tensorflow ?
appreciate any suggestions

Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.

Hi yeah i am basically struggling with  the implementation part of Loss function for VAE and my data is multivariate timeseries data.


text:
What best practices for VAE do you know?
The data is binary voxel data of shape (60, 36, 60). I want to compress such data into lower latent space to be able to use the denoising diffusion model. What best practices (model architecture, depth, etc.) do you suggest to get the most out of the VAE?
Here is my implementation of VAE:
# encoder
voxel_size = (60, 36, 60, 1)
encoder_inputs = tf.keras.Input(shape=voxel_size)
x = tf.keras.layers.Conv3D(64, 3, padding='same')(encoder_inputs)
x = VAEResidualBlock(64, name='residual_0')(x)
x = VAEResidualBlock(64, name='residual_1')(x)
x = tf.keras.layers.Conv3D(64, 3, strides=2, padding='same')(x)
x = VAEResidualBlock(128, name='residual_2')(x)
x = VAEResidualBlock(128, name='residual_3')(x)
x = tf.keras.layers.Conv3D(128, 3, strides=2, padding='same')(x)
x = VAEResidualBlock(256, name='residual_4')(x)
x = VAEResidualBlock(256, name='residual_5')(x)
x = VAEResidualBlock(256, name='residual_8')(x)
x = VAEAttentionBlock(256, name='attention_0')(x)
x = VAEResidualBlock(256, name='residual_9')(x)
x = GroupNormalization(256, name='group_normalization_0')(x)
x = tf.keras.activations.swish(x)
x = tf.keras.layers.Conv3D(8, 3, padding='same')(x)
z_mean = tf.keras.layers.Conv3D(4, 1, padding='valid', name='z_mean')(x)
z_log_var = tf.keras.layers.Conv3D(4, 1, padding='valid', name='z_log_var')(x)
encoder = tf.keras.Model(encoder_inputs, [z_mean, z_log_var], name='encoder')
encoder.summary()

# decoder
latent_inputs = tf.keras.Input(shape=(15, 9, 15, 4))
x = tf.keras.layers.Conv3D(4, 1, padding='valid')(latent_inputs)
x = tf.keras.layers.Conv3D(256, 3, padding='same')(x)
x = VAEResidualBlock(256)(x)
x = VAEAttentionBlock(256)(x)
x = VAEResidualBlock(256)(x)
x = VAEResidualBlock(256)(x)
x = VAEResidualBlock(256)(x)
x = VAEResidualBlock(256)(x)
x = tf.keras.layers.UpSampling3D(2)(x)
x = tf.keras.layers.Conv3D(256, 3, padding='same')(x)
x = VAEResidualBlock(128)(x)
x = VAEResidualBlock(128)(x)
x = VAEResidualBlock(128)(x)
x = tf.keras.layers.UpSampling3D(2)(x)
x = tf.keras.layers.Conv3D(128, 3, padding='same')(x)
x = VAEResidualBlock(64)(x)
x = VAEResidualBlock(64)(x)
x = VAEResidualBlock(64)(x)
x = GroupNormalization(64)(x)
x = tf.keras.activations.swish(x)
decoder_outputs = tf.keras.layers.Conv3D(1, 3, activation='sigmoid', padding='same')(x)
decoder = tf.keras.Model(latent_inputs, decoder_outputs, name='decoder')
decoder.summary()

# sampler
class Sampler(tf.keras.layers.Layer):
    def call(self, z_mean, z_log_var):
        batch_size = tf.shape(z_mean)[0]
        z_size = tf.shape(z_mean)[1]
        epsilon = tf.random.normal(shape=tf.shape(z_mean))
        # epsilon = tf.random.normal(shape=(batch_size, z_size))
        # print(z_mean.shape, epsilon.shape)
        return z_mean + tf.exp(0.5*z_log_var) * epsilon

# VAE
class VAE(tf.keras.Model):
    def __init__(self, encoder, decoder, **kwargs):
        super().__init__(**kwargs)
        self.encoder = encoder
        self.decoder = decoder
        self.sampler = Sampler()
        self.total_loss_tracker = tf.keras.metrics.Mean(name='total_loss')
        self.reconstruction_loss_tracker = tf.keras.metrics.Mean(name='reconstruction_loss')
        self.kl_loss_tracker = tf.keras.metrics.Mean(name='kl_loss')

    @property
    def metrics(self):
        return [self.total_loss_tracker,
                self.reconstruction_loss_tracker,
                self.kl_loss_tracker]

    def train_step(self, data):
        with tf.GradientTape() as tape:
            z_mean, z_log_var = self.encoder(data)
            z = self.sampler(z_mean, z_log_var)
            reconstruction = decoder(z)
            reconstruction_loss = tf.reduce_mean(tf.reduce_sum(tf.keras.losses.binary_crossentropy(data, reconstruction),
                                                                               axis=(1, 2, 3)))
            kl_loss = -0.5 * (1+z_log_var-tf.square(z_mean)-tf.exp(z_log_var))
            total_loss = reconstruction_loss + tf.reduce_mean(kl_loss)
        grads = tape.gradient(total_loss, self.trainable_weights)
        self.optimizer.apply_gradients(zip(grads, self.trainable_weights))
        self.total_loss_tracker.update_state(total_loss)
        self.reconstruction_loss_tracker.update_state(reconstruction_loss)
        self.kl_loss_tracker.update_state(kl_loss)
        return {'total_loss': self.total_loss_tracker.result(),
                'reconstruction_loss': self.reconstruction_loss_tracker.result(),
                'kl_loss': self.kl_loss_tracker.result()}

vae = VAE(encoder, decoder)
vae.compile(optimizer=tf.keras.optimizers.Adam(), run_eagerly=True)
vae.fit(voxel_arrays, epochs=150, batch_size=4,
        callbacks=[tf.keras.callbacks.EarlyStopping(monitor='total_loss',
                                                            patience=5),
                   tf.keras.callbacks.ModelCheckpoint('/home/r.abdrakhmanov/diffvox/models/vae.keras',
                                                              monitor='total_loss',
                                                              mode='min',
                                                              save_best_only=True,
                                                              save_weights_only=True)])

But the best possible total_loss so far:
Epoch 26/150
362/362 [==============================] - 552s 2s/step - total_loss: 20168.3750 - reconstruction_loss: 20089.0879 - kl_loss: 79.2998
Epoch 27/150
362/362 [==============================] - 555s 2s/step - total_loss: 19650.8145 - reconstruction_loss: 19571.0508 - kl_loss: 79.7697
Epoch 28/150
362/362 [==============================] - 540s 1s/step - total_loss: 19981.1680 - reconstruction_loss: 19898.7871 - kl_loss: 82.3754
Epoch 29/150
362/362 [==============================] - 550s 2s/step - total_loss: 19577.8027 - reconstruction_loss: 19502.1582 - kl_loss: 75.6499
Epoch 30/150
362/362 [==============================] - 535s 1s/step - total_loss: 20013.8477 - reconstruction_loss: 19934.2266 - kl_loss: 79.6151
Epoch 31/150
362/362 [==============================] - 528s 1s/step - total_loss: 19767.3184 - reconstruction_loss: 19687.5156 - kl_loss: 79.7937
Epoch 32/150
362/362 [==============================] - 516s 1s/step - total_loss: nan - reconstruction_loss: nan - kl_loss: nan
Epoch 33/150
362/362 [==============================] - 517s 1s/step - total_loss: nan - reconstruction_loss: nan - kl_loss: nan
Epoch 34/150
362/362 [==============================] - 527s 1s/step - total_loss: nan - reconstruction_loss: nan - kl_loss: nan


text:
Why does the latent space in Stable Diffusion have a shape of 64x64x3?
Since the encoding is performed by a Variational Autoencoder, the VAE encoder must output some mean and log variance that we can use to sample a latent vector of shape (latent_dim,). But then how does Stable Diffusion use a vector of shape 64x64x3?
Reference (https://arxiv.org/pdf/2112.10752.pdf, page 24, table 12):

Can you provide a reference that tells that the latent space has the dimensionality you say?

check updated question

Can you provide a citation/link to the paper where you got the table? I assume it's from Rombach et al., 2022?

There's not really a restriction on the shape for variational autoencoders. If you really wanted a 1D vector, you could just flatten the matrix and get a vector of size 64 * 64 * 3.
The 64x64x3 size makes sense if you consider why they encode images into a latent dimension in the first place.
Consider this passage from the latent diffusion paper:

(i) By leaving the high-dimensional image space, we obtain DMs which are computationally much more efficient because sampling is performed on a low-dimensional space. (ii) We exploit the inductive bias of DMs inherited from their UNet architecture [71], which makes them particularly effective for data with spatial structure and therefore alleviates the need for aggressive, quality-reducing compression levels as required by previous approaches [23, 66].

and

This is in contrast to previous works [23, 66], which relied on an arbitrary 1D ordering of the learned space z to model its distribution autoregressively and thereby ignored much of the inherent structure of z.

The purpose of the latent encoding is to compress images to a lower-dimensional space that's easier to work with---but you also want to make sure that you aren't losing information during the compression process.
A 2D representation would better models spatial structures that subsequent UNet layers are designed to work with.

Thank you for the feedback! But if we consider 3D (binary voxel) data instead of images, then what structure of latent representation would be useful: 2D or 3D? The issue is that I can't find any 3D latent diffusion implementations.

Yeah, I'm not too familiar with this as well. Although this paper (https://snap-research.github.io/3DVADER/paper.pdf) seems to do things in a different way. Their latent has three dimensions (whereas you'd expect four if you were to extend the 2D diffusion model).