---
tags:
- weight-space-learning
- neural-network-autoencoder
- autoencoder
- transformer
datasets:
- maximuspowers/muat-fourier-5
---

# Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures.
It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

## Model Description

- **Architecture**: Transformer encoder-decoder
- **Training Dataset**: maximuspowers/muat-fourier-5
- **Input Mode**: signature
- **Latent Dimension**: 256

## Tokenization

- **Chunk Size**: 64 weight values per token
- **Max Tokens**: 512
- **Metadata**: True

## Training Config

- **Loss Function**: cosine
- **Optimizer**: adam
- **Learning Rate**: 0.0001
- **Batch Size**: 16

## Performance Metrics (Test Set)

- **MSE**: 0.299696
- **MAE**: 0.303521
- **RMSE**: 0.547445
- **Cosine Similarity**: 0.8642
- **R² Score**: 0.0638