A PLDI evaluation answers one question: does the claimed mechanism cause the
claimed effect on programs that matter? Everything in the design flows from
making that causal link auditable. The community's shared rubric is SIGPLAN's
Empirical Evaluation checklist (see pldi-reproducibility for the measurement
hygiene); this skill covers the design choices above the hygiene layer.
setjmp, which our restriction rejects (§4.4)" builds trust; silent dropping
destroys it.| Weak move | Defensible move |
|---|---|
| Compare against -O0 or an untuned build | Strongest published configuration of the standard toolchain (state version + flags) |
| Reimplement a rival technique quickly | Use the authors' artifact where one exists; note deviations |
| Compare only against your own prior system | Add the external baseline reviewers will name in review |
| Report best-of-N runs | Report distribution over all N runs |
| One aggregate number | Aggregate + per-benchmark table, so wins and losses show |
The reviewers most likely to be assigned your paper wrote the baselines. Assume the baseline's author reads your flags line.
The claim "our escape signatures cause the speedup" needs the experiment where signatures are replaced by the prior summary while everything else stays fixed. Design one ablation per mechanism named in the contributions list; a mechanism with no ablation is a mechanism the paper does not actually test.
Runtime, compile time, and memory are all first-class at PLDI. A technique that buys 1.1x runtime with 3x compile time must say so in the abstract, not in a footnote. Report all three, each with repetitions and dispersion, even when one of them is "no change" — especially when it is "no change."
Run this list before the deadline; it is roughly what a PLDI review's evaluation section says when it goes badly:
[ ] Is the delta bigger than the noise band? (CI overlap check per benchmark)
[ ] Does the effect survive on a second microarchitecture?
[ ] Are the flags/version of every baseline stated and current?
[ ] Is there a benchmark where we lose, and do we explain it?
[ ] Does the ablation exist for every mechanism we claim credit for?
[ ] Is warmup/steady-state handling stated per benchmark family?
[ ] Could the speedup come from an unrelated engineering change? (same-codebase control)
A paragraph explaining the two programs where the technique regresses — with a cause, not a shrug — routinely appears in accepted PLDI papers and in Distinguished Paper profiles. Reviewers read it as instrument calibration. Deleting the losing rows reads as the opposite.
[Suite] chosen + justified? versions pinned? exclusions listed?
[Baselines] strongest config? external baseline present? flags stated?
[Ablations] mechanism -> ablation experiment (n/n covered)
[Currencies] runtime / compile time / memory each measured with variance?
[Objection pass] items failing from the checklist above