Design experiments that test the paper's claim, not experiments that showcase the
system. At RSS the evaluation section is where the scientific claim either becomes
falsifiable or is exposed as marketing.
Write the claim, then derive what evidence its logical form demands:
| Claim form | Evidence the form demands |
|---|---|
| "X causes the improvement" | Ablation removing only X; everything else frozen |
| "the gain transfers" | Held-out tasks/objects/platforms named before running |
| "method M has property P" | The condition where P would break, deliberately tested |
| "T is the bottleneck" | Failure attribution showing T dominates the failure mass |
| "faster/cheaper at equal quality" | Matched-quality comparison, not best-vs-default |
An experiment matrix that cannot be traced back to a claim row is footage, not
evidence — cut it or move it to the supplement.
distribution, and stopping rule written down before the campaign. Post-hoc criteria
are how demo bias enters honest labs.
auditable; "extensive trials" is not.
interventions. A human silently rescuing the robot between trials is part of the
system and must appear in the protocol.
to small samples and let the language match their width. Twenty trials support "in
our setting"; they do not support "reliably in general."
regimes hardware cannot safely visit); hardware earns the claim's headline.
physical. A sim-only paper must scope its claim to simulation explicitly.
silently.
A distribution over failure causes is often more scientifically valuable than the
success rate above it. Build one:
planning, timing, slip, hardware fault, other).
from memory drifts optimistic.
categories is telling you how it works.
Hardware time is the binding constraint, so budget it explicitly:
conditions: 2 methods x 3 environments x (1 ablation + 1 transfer) = 10
trials/condition: 25 -> 250 attempts
minutes/attempt (incl. reset): 6 -> 25 robot-hours
overhead (faults, recalibration, re-runs): x1.5 -> ~38 robot-hours
robot access: 4 h/day -> ~10 working days, before any surprise
If the arithmetic says the matrix cannot finish before the January freeze, shrink
the matrix, not the per-condition counts — fewer conditions with auditable n beat
a full grid of anecdotes. Reserve the final week for zero data collection.
Non-negotiables for the results section, independent of subfield:
when n is small; never a bare mean.
proposed method, and say so in one sentence.
make fast; if reimplementing, report your version's score on a setting where the
original published numbers exist.
A draft evaluates a new whole-body controller on eight household tasks and reports
success rates — a demo matrix. The claim, once written down, is "torque-limit
awareness, not trajectory optimality, drives the reliability gain." The redesign:
(1) an ablation running the same controller with torque-awareness disabled, same
eight tasks; (2) a stress condition where torque limits bind hard (heavy objects)
versus one where they never bind — the claim predicts the gain concentrates in the
first; (3) failure attribution distinguishing torque-saturation failures from
tracking failures. Three conditions replaced five decorative tasks, and the paper
gained a falsifiable spine without new hardware.
[Claim -> condition map] <each claim row -> experiment>
[Protocol status] pre-registered / drifting / post-hoc
[Counts] <per-condition n, denominators, interventions>
[Sim/real ledger] clean split / mixed (fix)
[Failure attribution] <categories + dominant mass>
[Decisive missing run] <the one experiment to add next>