0xSero/glm-local-vision-checkpoint

GLM-Local-Vision Checkpoint (Mixed Dataset Step 1000)

Vision projector retrofit for GLM-5.2 (753B MoE, text-only LLM), trained following Harry Partridge's projector-only alignment method. The projector maps MoonViT vision-tower features (1152-dim) into GLM's embedding space (6144-dim) so the frozen backbone can ground on UI screenshots, app interfaces, and general visual content.

What this is

• 44.8M-parameter vision projector (only trainable component)
• Architecture: LN(1152) → Linear(1152, 6144) → GELU → Linear(6144, 6144)
• MoonViT (Kimi-K2.6 vision tower, ~450M params) stays frozen
• GLM-5.2 backbone (256 experts pruned to 32, NF4 quantized, ~194GB) stays frozen
• Image convention: |begin_of_image| + |image|×N + |end_of_image|
• Coordinate convention: [0, 1000) normalized integers, top-left origin
• Training format: UI-TARS-style click(start_box=[x, y])

Checkpoint details

Field	Value
Checkpoint step	1000 (mixed) ≈ step 1750 cumulative (750 screenshot + 1000 mixed)
Training loss	2.42 (mixed, still declining)
Dataset	116,241 train (30,850 screenshots + 85,391 art/design/color/hardware)
Val set	6,119 examples
Warm-start	from screenshot step-750 checkpoint (loss 1.280)
Epoch	1/2 (mixed, over halfway through epoch 1)
Optimizer	AdamW, lr=5e-4
Global batch	64 (microbatch=8, grad-accum=8)
Throughput	~57 s/step, 94 tok/s on 4× H100
Date	2026-07-19

Projector-only eval (50 mixed val samples, CPU)

The projector produces highly structured, spatially-differentiated embeddings compared to a random-init baseline. The mixed dataset (screenshots + art + color + hardware photos) broke the projector out of its screenshot-only plateau — token diversity jumped 17.6× and spatial variance hit a new all-time high of 7.41, with ratio crossing 5000×.

Metric	Trained	Random	Ratio
Output norm	303.01	14.90	20.3×
Spatial variance	7.4132	0.0015	5072×
Token diversity	0.3177	—	—

Eval progression (mixed dataset)

Checkpoint	Spatial var	Token diversity	Ratio	Norm
Screenshot step 750	4.82	0.018	3282×	—
Mixed step 50	4.32	0.050	2904×	—
Mixed step 100	5.13	0.088	3333×	—
Mixed step 150	5.65	0.131	3717×	—
Mixed step 200	5.81	0.158	3927×	—
Mixed step 250	5.89	0.198	3912×	355.30
Mixed step 300	6.12	0.194	4071×	361.79
Mixed step 350	6.78	0.229	4396×	349.77
Mixed step 400	6.86	0.240	4667×	342.69
Mixed step 450	7.09	0.261	4631×	335.54
Mixed step 500	6.92	0.280	4511×	323.51
Mixed step 550	6.83	0.274	4549×	318.68
Mixed step 600	6.63	0.273	4444×	314.43
Mixed step 1000	7.41	0.318	5072×	303.01

Token diversity climbed sharply through step 250 (0.018 → 0.198, 11×), plateaued at ~0.19 (step 300), then jumped +18% at step 350 (0.194 → 0.229) — breaking the plateau — climbed to 0.280 (step 500), then flat through compression phase (steps 500-600: 0.273-0.280), then broke out again at step 1000: 0.318 (+16.4% from step 600). Spatial variance peaked at 7.09 (step 450), declined through compression (6.63 at step 600), then smashed a new all-time high at step 1000: 7.41 (+11.8% from step 600). Ratio crossed 5000× for the first time. Output norm kept declining (314 → 303) — the projector is expressing much richer structure at tighter magnitude. The compression→breakout cycle is the textbook pre-grokking signature: tighten, then expand with richer structure.

Full-inference grounding eval (GPU, 50 screenshot val examples)

End-to-end test: MoonViT → projector → GLM backbone generates click(start_box=[x,y]) autoregressively. Measures Euclidean distance between predicted and ground-truth click coordinates in [0, 999] space.

Metric	Value
Parse rate	46/50 (92.0%)
Mean distance	563.7 / 999
Median distance	566.9 / 999
Min distance	25.0
Accuracy @50	2/46 (4.3%)
Accuracy @100	4/46 (8.7%)
Accuracy @200	7/46 (15.2%)

92% parse rate confirms the projector successfully bridges vision features into GLM's token space — GLM reliably emits well-formed click coordinates. Mean distance ~564/999 means clicks land in the right general region but not precisely on target. This is the baseline from the mixed-dataset projector (heavy on art/color/hardware, lighter on screenshots); a screenshot-focused finetune should sharpen accuracy. The near-perfect min distance (25/999) shows the capacity for precise grounding exists.

Files

• projector.safetensors — the 44.8M-param projector weights (BF16)
  • ln.weight/bias: LayerNorm (1152)
  • fc1.weight/bias: Linear (1152 → 6144)
  • fc2.weight/bias: Linear (6144 → 6144)

Usage

from safetensors.torch import load_file
import torch.nn as nn

class VisionProjector(nn.Module):
    def __init__(self, in_dim=1152, out_dim=6144):
        super().__init__()
        self.ln = nn.LayerNorm(in_dim)
        self.fc1 = nn.Linear(in_dim, out_dim)
        self.fc2 = nn.Linear(out_dim, out_dim)
    def forward(self, x):
        return self.fc2(torch.nn.functional.gelu(self.fc1(self.ln(x))))

proj = VisionProjector().to(torch.bfloat16)
state = load_file("projector.safetensors")
proj.load_state_dict(state)
proj.eval()

Pair with a MoonViT vision tower (1152-dim output) and the GLM-5.2 backbone to splice projected embeddings into |image| token slots.

Method credit

Training follows the projector-only alignment approach described by Harry Partridge — only the projector trains, the entire LLM backbone stays frozen, and a sharp loss drop ("grokking") marks alignment.