CVPR runs on benchmark evidence: reviewers at the 2026 edition sorted 16,092 submissions
largely by asking "do the tables prove the sentence?" This skill designs an experimental
program that answers the four questions every vision review implicitly asks — does it
work, why does it work, when does it fail, and what does it cost.
versions, standard splits. A new task may justify a new benchmark, but then the
benchmark itself becomes a contribution needing validation (and, if claimed as one,
public release by camera-ready — verified 2026 policy).
baseline. The comparison that kills papers in review is the one you omitted because
it was too strong.
schedule wherever possible — or tabulate the mismatch explicitly. Beating a
ResNet-era method with a ViT-L and calling it method innovation is the single most
common CVPR review objection.
papers; asterisk and footnote the re-runs.
Vision reviewers treat ablations as the paper's proof of understanding. Structure the
grid so each row removes or replaces exactly one design decision:
# ablation-matrix.txt — one experiment per line, one variable per experiment
A0 full method (reference row)
A1 - temporal attention → per-frame baseline tests the core claim
A2 - our loss → standard L1 is the loss or the architecture doing it?
A3 - pretrain → from scratch how much rides on initialization?
A4 swap: our module in baseline X does the gain transfer?
A5 sensitivity: key hyperparameter sweep is A0 a lucky point?
Rows A4 (transplant) and A5 (sensitivity) separate memorable ablation sections from
perfunctory ones. Every ablation row cited in prose belongs in the 8-page body; the
long grid goes to the supplement.
Cherry-picked grids convince nobody at a venue that invented the genre. The credible
pattern:
| Qualitative element | Purpose |
|---|---|
| Random (or id-listed) sample grid | Shows typical, not best-case, behavior |
| Side-by-side vs. two strongest baselines | Same inputs, aligned crops, labeled columns |
| Failure cases with a taxonomy | "Fails under occlusion and low light" beats silence |
| Video for anything temporal | In the supplement — external links are banned |
State the selection rule in the caption ("first 8 validation images", "random seed 0").
A stated rule converts pretty pictures into evidence.
The 2026 cycle made compute visible venue-wide via the mandatory Compute Reporting Form
(hardware + verification sections required; deeper compute sections optional but tied
to recognition badges). Align the paper with the form: report params, FLOPs, latency
(named hardware, batch size, resolution), and training GPU-hours for your method and
re-run baselines where you can. "Real-time" with no hardware named contradicts your own
CRF and reviewers can now check.
A method shown on one dataset is a result about that dataset. Cheap robustness
evidence reviewers reward: evaluate the trained model on a second domain without
retuning; report cross-dataset transfer; if your field has corruption/shift suites, run
them. One honest sentence about where transfer degrades is worth more than a defensive
omission — it becomes your limitations paragraph (see cvpr-writing-style).
At CVPR-scale training costs, experiment selection is a resource-allocation problem.
Plan the program backward from the paper's skeleton:
efficiency table, qualitative grid, one transfer result).
have" grid costing more than the flagship.
claim runs first, not last — discovering in October that A1 ≈ A0 should redirect
the project, not decorate it.
small matched-setting re-run, and having compute standing by converts a weakness
into a mini-table.
cvpr-reproducibility) as it runs; the CRFoptional sections then fill themselves.
Multi-seed everything is unaffordable at modern budgets; the defensible pattern is
multi-seeded cheap experiments with mean ± std, a single flagged flagship run, and no
headline claims resting on differences smaller than observed seed noise (full protocol
in cvpr-reproducibility). For generative work, repeat evaluation sampling; for
detection, fix and disclose the exact mAP implementation.
withheld; see cvpr-reproducibility for submission-server discipline).
cycle).
[Evidence audit] works / why / fails / costs — covered?
[Comparison risks] unmatched: <backbone/pretrain/resolution/schedule>
[Ablation] rows isolating single factors: <n>; transplant + sensitivity present?
[Qualitative] selection rule stated? failures taxonomized?
[Efficiency] params/FLOPs/latency/GPU-hours vs CRF: consistent?
[Priority additions] <ordered by review impact>