decoderesearch/qwen-3.5-saes

Qwen-3.5 SAEs for use with the SAELens library

This repository contains the following SAEs:

Qwen-3.5-0.8b

• qwen-3.5-0.8b/btk-mat-layer-5-k-100
• qwen-3.5-0.8b/btk-mat-layer-11-k-100
• qwen-3.5-0.8b/btk-mat-layer-17-k-100

Qwen-3.5-0.8b-base

• qwen-3.5-0.8b-base/btk-mat-layer-5-k-100
• qwen-3.5-0.8b-base/btk-mat-layer-11-k-100
• qwen-3.5-0.8b-base/btk-mat-layer-17-k-100

Qwen-3.5-4b

• qwen-3.5-4b/btk-mat-layer-3-k-100
• qwen-3.5-4b/btk-mat-layer-15-k-100
• qwen-3.5-4b/btk-mat-layer-27-k-100

Qwen-3.5-4b-base

• qwen-3.5-4b-base/btk-mat-layer-3-k-100
• qwen-3.5-4b-base/btk-mat-layer-15-k-100
• qwen-3.5-4b-base/btk-mat-layer-27-k-100

Load these SAEs using SAELens as below:

from sae_lens import SAE

sae = SAE.from_pretrained("decoderesearch/qwen-3.5-saes", "<sae_id>")

About these SAEs

These SAEs are all Matryoshka BatchTopK SAEs trained on the Pile Uncopyrighted using SAELens.

The SAEs each have width 65k latents, and 2 inner Matryoshka prefixes of 2k latents and 16k latents.

Reproducing training

For complete details of how these SAEs were trained, refer to the runner_cfg.json in each SAE directory. This config includes all hyperparameters passed to the SAELens trainer when training the SAEs.

Citation

@misc{decode2026qwen35saes,
  author       = {Chanin, David and Lin, Johnny},
  title        = {{Qwen-3.5} Sparse Autoencoders},
  year         = {2026},
  organization = {Decode Research},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/decoderesearch/qwen-3.5-saes}}
}

Acknowledgements

These SAEs were trained thanks to compute provided by Modal.