siddharthmb/2026.RA.Fairness-GRPO-lam0

2026.RA.Fairness-GRPO — LoRA adapters

These adapters did not achieve their objective. They are published as the artifact of a preregistered negative result, and should not be used as "a fairer negotiator" — measured against the untrained base model they are worse on welfare and more selfish in canonical bargaining games.

What these are

LoRA adapters (r=32, α=64, 87.3M trainable parameters) on Qwen/Qwen3-8B, trained with GRPO in six-party self-play negotiation against a smoothed logarithmic Nash-welfare reward. Two arms, each with a checkpoint ladder at steps {5, 10, 15, 20, 24}:

• 2026.RA.Fairness-GRPO-lam0 — λ=0, each seat paid its own normalized outcome.
• 2026.RA.Fairness-GRPO-lam1 — λ=1, every seat paid the table's mean outcome.

The reward is R_i(λ) = (1−λ)·g(z_i) + λ·mean_j g(z_j) where z_i is party i's normalized surplus and g(z) = log z above a threshold ε = 0.01, continuing linearly below it. It is text-blind: computed from the engine's scoring of the closed deal, never from generated text.

Measured performance (this is the part that matters)

Held-out evaluation on 24 games never trained on (48 instance clusters, 960 episodes per cell), trained minus untrained Qwen3-8B, 95% instance-cluster bootstrap:

endpoint	λ=0 step 24	λ=1 step 24	direction
normalized Nash welfare	−0.185 [−0.225, −0.145]	−0.109 [−0.145, −0.070]	worse
deal rate (baseline 0.817)	−0.252	−0.149	worse
below-threshold agreements	−0.087 [−0.127, −0.050]	−0.079 [−0.124, −0.030]	better
NNW among individually-rational deals	+0.014 [−0.012, +0.040]	−0.040 [−0.067, −0.015]	λ=1 worse
Gini among IR deals	−0.008 [−0.025, +0.009]	+0.023 [+0.010, +0.036]	λ=1 worse

Interpretation. The one thing training bought is a reduction in below-threshold agreements — deals a party signs that are worse for it than walking away. It bought that substantially by not agreeing at all: deal rate fell by 0.25 (λ=0) and 0.15 (λ=1). Overall welfare therefore fell. Splitting the fairness metrics into all episodes versus only-fully-individually-rational episodes separates the arms: λ=0's welfare loss is a composition effect (fewer deals, not worse ones — every among-IR interval spans zero), whereas λ=1 struck measurably less fair deals among the deals it did strike.

Out-of-distribution behaviour got worse. Seated in the classic ultimatum game, which the policy never saw in training, the untrained model proposes an accepted 60/40 split. Both trained arms propose 96.7/3.3, accepted 100% of the time — a move toward the self-interested subgame-perfect prediction and away from the fair split, consistent across every seed. Divide-the-dollar shifts the same way more weakly (max share 0.794 → 0.844 / 0.894 against a 1/n reference of 0.333).

Guards. The trained policy is not exploitable — seated against five computable rational agents its own capture is unchanged (λ=0 −0.001 [−0.036, +0.036]). A transcript audit found no degenerate strategy: the trained policy actually negotiates more (turns per episode 20.5 → 25.1) and agrees less.

Training-time metrics (on the training bank, so not evidence of generalization): reward R_table +0.61 / +0.57, worst-off seat's g +2.86 / +3.33, log-prob drift from the frozen base +0.14 / +0.27 nats per token, 0 fabricated turns at any step.

Caveat that bounds every number above

The run is 24 GRPO steps per arm, 12% of the preregistered 200-step budget, because a step costs ~17 minutes in this environment (an episode is ~20 sequential co-stepping model calls). Over-training is already visible — welfare worsens between step 10 and step 24 — but nothing here rules out a different trajectory at full budget. Treat this as a well-instrumented short run.

Reproducing

python -m grpo.train --lam 0 --steps 24 --groups 4 --k 8 --micro-batch 6 --max-new-tokens 384 \
    --lr 5e-5 --checkpoint-steps 5 10 15 20 24 --bank instances_grpo_train_v1 \
    --out runs/lam0 --transcript-every 5 \
    --wandb-project rational_agents_fairness_grpo --wandb-group fairness-grpo-pilot

Evaluation fleet and analysis: python -m grpo.launch_eval --checkpoint "lam0_step24=runs/lam0/checkpoint-24" --out-dir sbatch_grpoeval --submit, then python -m grpo.analyze_eval --baseline "<baseline>_primary_s*" --trained "lam0_step24=<run>_primary_s*" --out eval.json.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16")
model = PeftModel.from_pretrained(base, "siddharthmb/2026.RA.Fairness-GRPO-lam0", subfolder="checkpoint-24")

Links

• Dataset (training steps, eval contrasts, rollout transcripts): siddharthmb/2026.RA.Fairness-GRPO
• Weights & Biases: project rational_agents_fairness_grpo, group fairness-grpo-pilot — λ=0 iedvonxu, λ=1 a6wqb6l8
• Cluster artifacts: /nlp/scr/siddharth/ii_mats/rational_agents/fairness_grpo/ (adapters, step logs, 256 sampled rollout transcripts); evaluation run directories /nlp/scr/siddharth/ii_mats/rational_agents/grpoeval_*
• Full write-up: research note 0023-fairness-grpo-pilot.md in the project repository, with the preregistration it was run against and the compliance table.