Use this before the evaluation is frozen. An MLSys evaluation exists to answer one
compound question: *does the named mechanism deliver the claimed system-level payoff on
workloads that matter, at acceptable cost, for understood reasons?* Each clause needs its
own evidence, and this venue — which published the MLPerf Training methodology in its own
proceedings — holds measurement to benchmark-committee standards.
"Unrepresentative workload" is the modal fatal objection at this venue. Defend the
choice explicitly:
published request traces. Bespoke workloads need a characterization section showing
their statistics (request rates, sequence lengths, arrival burstiness, model mix).
bursty and uniform load; the uniform case bounds your claim, it does not weaken it.
non-win row is the cheapest credibility purchase available.
artifact badges (see mlsys-artifact-evaluation).
configurations — and tune them with the same budget you gave your own system.
Document both budgets; asymmetric tuning is treated as invalidating.
behavior); reviewers use it to sanity-check absolute numbers.
rather than asserting it.
Never report throughput alone. Each headline claim carries four coordinates:
| Coordinate | Report as | Classic gaming pattern it prevents |
|---|---|---|
| Throughput/goodput | Under a stated latency constraint | Batch-size inflation that destroys tails |
| Latency | p50 and p99 (p99.9 for serving), distributions not means | Mean-only tables hiding stragglers |
| Memory | Peak device + host, at the measured configuration | Wins that only fit on 80GB parts, unstated |
| Cost/efficiency | $ or joules per unit work, with price/meter basis | "Faster" systems that are 4x more expensive |
Quality (accuracy/perplexity or task metric) rides along whenever the technique could
plausibly change model output — quantization, sparsity, scheduling with dropping,
approximation of any kind. "No quality change" is a claim requiring a measurement.
Systems papers stack techniques; reviews ask which component pays.
and say which.
include the no-lookahead scheduler, not just the whole-system-off row.
a mechanism needing per-workload hand-tuning must say so.
# experiment-matrix.yaml — one row per plotted point, generated, never hand-edited
sweep:
models: [llama-class-8b, llama-class-70b, moe-16e]
hardware: [1xA100, 8xA100-nvlink] # state interconnect explicitly
load: [uniform-qps, bursty-trace-A]
precision: [bf16, int4]
trials: 5
warmup_steps: 20
metrics: [goodput@p99<250ms, p50, p99, peak_mem_gb, gpu_hours, quality]
seeds: [1, 2, 3, 4, 5]
single-configuration win invites the "will this survive scale?" objection you cannot
answer in the four-day response window.
and variance rules live in mlsys-reproducibility).
Systems papers rarely need hypothesis-testing machinery, but they always need variance
honesty; the floor this venue's reviewers apply:
IQR) and say which in the caption.
report the tie honestly; a "win" inside noise is worse than a reported tie because
one reviewer rerun destroys it.
requests that p99 is estimated from hundreds of samples, not three.
(placement, thermal state, co-tenants) dominates latency variance and is controlled
by measurement design, not seeds — the split is detailed in mlsys-reproducibility.
Run this table exercise before burning GPU-hours: for each claim in the abstract, write
the figure that will support it, the workload it needs, and the strongest objection a
systems reviewer could raise — then check the matrix covers that objection. Claims
without a planned figure get cut from the abstract, not padded later.
appendix table.
[Evaluation readiness] strong / adequate / weak
[Workload defense] <named anchors + characterization + non-win case present?>
[Baseline symmetry] <versions pinned, tuning budgets equal + documented?>
[Quartet coverage] <throughput/latency-tails/memory/cost per headline claim>
[Attribution] <mechanism -> ablation row>
[Decision-critical missing run] <one experiment>