Skills Artificial Intelligence ML System Evaluation Methodology Guide

ML System Evaluation Methodology Guide

v20260724
mlsys-experiments
This comprehensive guide instructs researchers on how to design rigorous and credible evaluations for Machine Learning systems. It covers critical areas such as workload selection, baseline comparison discipline, reporting four key coordinates (throughput, latency tails, memory, cost), structuring ablations, and generating scaling evidence. It ensures measurements meet academic benchmarking standards.
Get Skill
375 downloads
Overview

MLSys Experiments

Use this before the evaluation is frozen. An MLSys evaluation exists to answer one
compound question: *does the named mechanism deliver the claimed system-level payoff on
workloads that matter, at acceptable cost, for understood reasons?* Each clause needs its
own evidence, and this venue — which published the MLPerf Training methodology in its own
proceedings — holds measurement to benchmark-committee standards.

Workload selection: the make-or-break choice

"Unrepresentative workload" is the modal fatal objection at this venue. Defend the
choice explicitly:

  • Anchor on recognizable workloads: MLPerf-family tasks, widely used open models, or

published request traces. Bespoke workloads need a characterization section showing
their statistics (request rates, sequence lengths, arrival burstiness, model mix).

  • Cover the axis your mechanism exploits: a scheduler for bursty load must be tested on

bursty and uniform load; the uniform case bounds your claim, it does not weaken it.

  • Include at least one workload where you expect little or no gain, and show it. A

non-win row is the cheapest credibility purchase available.

  • State workload licensing/provenance — traces you cannot release constrain your future

artifact badges (see mlsys-artifact-evaluation).

Baseline discipline

  • Compare against the current strongest systems, version-pinned, in their recommended

configurations — and tune them with the same budget you gave your own system.
Document both budgets; asymmetric tuning is treated as invalidating.

  • Include the "do nothing" configuration where meaningful (e.g., default framework

behavior); reviewers use it to sanity-check absolute numbers.

  • If a relevant baseline cannot run in your setting, demonstrate the incompatibility

rather than asserting it.

The reporting quartet

Never report throughput alone. Each headline claim carries four coordinates:

CoordinateReport asClassic gaming pattern it prevents
Throughput/goodputUnder a stated latency constraintBatch-size inflation that destroys tails
Latencyp50 and p99 (p99.9 for serving), distributions not meansMean-only tables hiding stragglers
MemoryPeak device + host, at the measured configurationWins that only fit on 80GB parts, unstated
Cost/efficiency$ or joules per unit work, with price/meter basis"Faster" systems that are 4x more expensive

Quality (accuracy/perplexity or task metric) rides along whenever the technique could
plausibly change model output — quantization, sparsity, scheduling with dropping,
approximation of any kind. "No quality change" is a claim requiring a measurement.

Ablation = attribution

Systems papers stack techniques; reviews ask which component pays.

  • One ablation row per named mechanism, cumulative or leave-one-out — pick one scheme

and say which.

  • Ablate the insight, not only the code: if the claim is "lookahead scheduling wins,"

include the no-lookahead scheduler, not just the whole-system-off row.

  • Sensitivity sweeps over the parameters a deployer must set (window sizes, thresholds);

a mechanism needing per-workload hand-tuning must say so.

Scaling and sensitivity evidence

# experiment-matrix.yaml — one row per plotted point, generated, never hand-edited
sweep:
  models:    [llama-class-8b, llama-class-70b, moe-16e]
  hardware:  [1xA100, 8xA100-nvlink]        # state interconnect explicitly
  load:      [uniform-qps, bursty-trace-A]
  precision: [bf16, int4]
  trials: 5
  warmup_steps: 20
  metrics: [goodput@p99<250ms, p50, p99, peak_mem_gb, gpu_hours, quality]
  seeds: [1, 2, 3, 4, 5]
  • Plot trends, not points: gain versus model size, versus load, versus GPU count. A

single-configuration win invites the "will this survive scale?" objection you cannot
answer in the four-day response window.

  • Log everything; regenerate every figure from logs by script (measurement mechanics

and variance rules live in mlsys-reproducibility).

Statistical floor for systems numbers

Systems papers rarely need hypothesis-testing machinery, but they always need variance
honesty; the floor this venue's reviewers apply:

  • Five or more trials for any number in a comparison table; report spread (stdev or

IQR) and say which in the caption.

  • Overlapping spreads mean the difference is not a finding — either run more trials or

report the tie honestly; a "win" inside noise is worse than a reported tie because
one reviewer rerun destroys it.

  • For latency distributions, trials are not enough: within-run percentiles need enough

requests that p99 is estimated from hundreds of samples, not three.

  • Seed variation matters only where model quality enters the claim; systems noise

(placement, thermal state, co-tenants) dominates latency variance and is controlled
by measurement design, not seeds — the split is detailed in mlsys-reproducibility.

Evaluation-design review before running

Run this table exercise before burning GPU-hours: for each claim in the abstract, write
the figure that will support it, the workload it needs, and the strongest objection a
systems reviewer could raise — then check the matrix covers that objection. Claims
without a planned figure get cut from the abstract, not padded later.

Common reject patterns to design against

  • Microbenchmark gains that never appear end-to-end; always include the end-to-end run.
  • Gains measured only at one scale/hardware and claimed generally.
  • Quality silently degraded by the optimization, discovered by a reviewer from your own

appendix table.

  • Cost omitted where the honest accounting would erase the win.

Output format

[Evaluation readiness] strong / adequate / weak
[Workload defense] <named anchors + characterization + non-win case present?>
[Baseline symmetry] <versions pinned, tuning budgets equal + documented?>
[Quartet coverage] <throughput/latency-tails/memory/cost per headline claim>
[Attribution] <mechanism -> ablation row>
[Decision-critical missing run] <one experiment>
Info
Name mlsys-experiments
Version v20260724
Size 6.35KB
Updated At 2026-07-28
Language