Beyond the Aggregate Score: Per-Country Domain Shift in the GWHD Wheat Head Detection Model Zoo

Saumya Saksena

dronefreak

Follow

Background

A few weeks ago I open-sourced 9 object detection models — YOLOv8, YOLOv11, YOLOv26, and RF-DETR, spanning nano through x-large variants — fine-tuned on the Global Wheat Head Dataset (GWHD) 2021, a dense, single-class wheat-head detection benchmark assembled from field images captured across 6 countries and 18 research institutions to maximize genotype, growth-stage, and imaging-condition diversity. All 9 models were trained and evaluated under one shared pipeline (DetectionBench), with the usual reproducibility artifacts: mAP/precision/recall, per-class breakdowns, training configs, and qualitative showcases.

The headline result was straightforward: YOLOv11x tops the leaderboard at 74.25% mAP@50 (34.92% mAP@50:95, 83.37% precision), with YOLOv26s offering the best efficiency tradeoff — 70.49% mAP@50 at just 22.8 GFLOPs (10.0M params), within 4 points of the top model while using ~8.6x fewer FLOPs than YOLOv11x's 196.0 GFLOPs.

However, based on community feedback, someone pointed out that:

A useful next comparison would be per-country or genotype slices, since domain shift can matter more than the aggregate score in field deployment.

These metrics weren't included in the model cards — at least not at first. This article is that follow-up: a per-country stratified evaluation across all 9 models, and what it actually implies.

Methodology

GWHD's images come from 18 contributing institutions across 6 countries (Australia, China, Japan, Mexico, Sudan, US), each represented by one or more "domains" in the dataset's own terminology. I compiled a per-image manifest (domain_metadata.json) mapping every test-split image to its contributing country and growth stage — this mapping is not part of the original GWHD release; it was built for this analysis specifically, from per-domain metadata associated with the dataset.

For each of the 9 trained models, I re-ran evaluation restricted to one country's images at a time. Critically, this reuses the exact same evaluation engine already producing each model's published aggregate score — Ultralytics' own model.val() for the YOLO family, supervision.MeanAveragePrecision for RF-DETR — just filtered to a per-country image subset rather than the full test split. The per-country numbers are therefore directly reconcilable with the aggregate numbers already published, not a second, independently-implemented metric that could subtly disagree with them.

One test image (of 1,382 total) had no resolvable country in the source metadata — a documented upstream duplicate-filename quirk where the same raw filename appears under two different domain entries — and was excluded from every breakdown below rather than guessed at. All figures here are computed on the remaining 1,381 test images, split unevenly across countries:

That imbalance turns out to matter a lot for interpreting the aggregate leaderboard, as the results below show.

Results: mAP@50 per country

(mAP@50:95 per country is available in each model's country_breakdown.json; omitted here for brevity — the pattern below holds for both metrics.)

Finding 1: China is the strongest country for every single model

Without exception, all 9 models — regardless of architecture, size, or training recipe — score highest on the China subset (79.8–92.4% mAP@50). This is the one fully consistent result across the whole zoo, and it's worth being honest about why it's not especially surprising: China is one of the larger, more visually consistent domains in the dataset (2 institutions, 200 images), and consistency within a domain tends to make evaluation on it easier regardless of what a model learned elsewhere.

Finding 2: YOLO and RF-DETR mostly disagree on where they struggle most

This is the more interesting, genuinely model-dependent result. Looking at each model's worst-performing country:

4 of the 6 YOLO variants (YOLOv26s, YOLOv8m, YOLOv8s, YOLOv8n) are weakest on the US subset — which, at 605 of 1,381 images, is also 43.8% of the entire test set. That means the published aggregate mAP for those four models is disproportionately anchored to the one region they handle worst, not a "typical" country performance. The two exceptions are notable in their own right: YOLOv11x's worst country is actually Australia, and YOLOv26m's is Japan — neither model shares the majority pattern.

All three RF-DETR variants, by contrast, are weakest on Australia (20.3% of the test set) — a completely different failure mode from most of the YOLO family, despite RF-DETR Medium's overall aggregate score (67.1% mAP@50) sitting in the same range as several mid-sized YOLO models.

Finding 3: robustness to domain shift is largely independent of raw accuracy

YOLOv26m has the narrowest country-to-country spread of all 9 models — 22.8 points, from 88.99% on China down to 66.21% on Japan — despite not being the top-scoring model overall (that's YOLOv11x, with a nearly identical 23.1-point spread). Both models are meaningfully more consistent across domains than everything else in the zoo.

At the other end, RF-DETR Nano has by far the widest spread of any model — 44.4 points, swinging from 79.8% on China down to just 35.4% on Australia. That's a larger single-model spread than the entire model-to-model range on China (79.8–92.4%, a 12.6-point spread). In other words: for RF-DETR Nano specifically, which country your deployment images come from matters more than which model in the zoo you picked.

This is the practical takeaway the reviewer's question was really getting at: the single aggregate mAP number would have completely hidden that reliability gap. YOLOv26s and YOLOv8m post nearly identical aggregate scores (70.49% vs. 69.55% mAP@50 — under a point apart), yet their country-to-country spreads differ by more than 3 points (25.3 vs. 28.6). Two models can rank almost identically on the leaderboard while having measurably different reliability across domains, and that's information a single aggregate row can't carry.

Caveats

• Sample sizes vary a lot by country. Sudan (30 images) and Japan (60 images) are small subsets; a handful of hard or easy images can move their mAP more than for US (605 images) or Australia (281 images). Treat the Sudan/Japan numbers as noisier than the rest.
• This is country-level, not genotype-level. The reviewer's comment mentioned genotype slices specifically; the per-image manifest this analysis is built from doesn't carry genotype labels, only country/institution/growth-stage. Genotype-stratified evaluation would need a different metadata source.
• Growth-stage stratification is next, using the same re-run-per-subset methodology, once a labeling inconsistency in the source metadata ("Post-flowering" vs. "Post-Flowering" — almost certainly the same stage, differing only in capitalization) is normalized so it doesn't fragment into two near-duplicate groups.

Where to find this

• Every model card now has a Per-Country Performance section with this table for that specific model, plus country_breakdown.json (raw results) and domain_metadata.json (the country/growth-stage mapping used to compute them) staged in the repository — see the model collection.
• Dataset: dronefreak/GWHD
• Code: DetectionBench

Credits

Full credit for the underlying dataset goes to Etienne David, Mario Serouart, Simon Madec, and the Global Wheat Head Detection consortium (Plant Phenomics, 2021; 2020). This release and the analysis above are an unofficial, independent evaluation built on top of their work, not affiliated with the original authors.

Thanks to the reviewer who pushed on this — it surfaced a real, actionable finding about model reliability across domains that the aggregate leaderboard alone would never have shown.