Potpov/grimoire-checkpoints

Grimoire — VSQ + ART checkpoints

Final checkpoints for Grimoire, a two-stage conditional SVG generative model.

• VSQ (stage 1): a vector-quantized SVG tokenizer/autoencoder (ResNet encoder → FSQ codebook → differentiable vector decoder rendered with diffvg).
• ART (stage 2): an autoregressive transformer over VSQ tokens, conditioned on a frozen BERT text embedding (class VQ_SVG_Stage2).

Layout

<dataset>/<stage>/last.ckpt      # PyTorch-Lightning checkpoint (weights under "state_dict")
<dataset>/<stage>/config.yaml    # config to rebuild the model before loading

dataset	VSQ (stage 1)	ART (stage 2)
figr8	✅	✅
mnist_bw	✅	✅
mnist_color	✅	✅
fonts	✅	—
emoji	✅	—

Matching datasets: grimoire-figr8, grimoire-mnist, grimoire-emoji.

Loading & inference (with the code checked out)

Stage 1 — VSQ reconstruction (figr8 / fonts, single-layer):

python scripts/hf_inference_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir figr8/vsq \
    --dataset_csv_path <csv with a file_path column> --outpath out/

Stage 2 — ART text→SVG generation (loads VSQ + trained ART from HF, samples an SVG):

python scripts/hf_generate_demo.py --hf_repo Potpov/grimoire-checkpoints \
    --art_subdir figr8/art --vsq_subdir figr8/vsq --prompt "a star" --outfile out/gen.svg

Emoji — layered/colored VSQ reconstruction (the VSQ_layers + hydra path is different):

python scripts/hf_inference_emoji_demo.py --hf_repo Potpov/grimoire-checkpoints --subdir emoji/vsq \
    --emoji_dir <dir with preprocessed_v2/> --outfile out/emoji.svg

All three are verified working (torch 2.0.1 / cu118 / diffvg-from-source).

Notes

• Build the VSQ from its own vsq/config.yaml. mnist_color/* configs are the repo configs.
• emoji/vsq is emoji_VSQ_hydra_small_noout (hydra decoder, single_code_representation=true, num_segments=16, pred_color=true) — the properly-trained emoji VSQ; it reconstructs emojis faithfully. (An earlier push used the older VSQ_EMOJI_COLOR, which under-reconstructed.)
• Loading the ART (stage-2) checkpoints: strip only the leading model. prefix (a blanket replace("model.","") also deletes the inner .model. in transformer.model.layers.* → the transformer loads as random weights) and remap .ff.3.→.ff.2. (x-transformers version shift). scripts/hf_generate_demo.py:_remap_key does this. VSQ checkpoints load with a plain leading-prefix strip.

Requirements

torch 2.0.1 (cu118) · diffvg (from source) · see the code repo's install.sh / requirements.txt.