技能 人工智能 ML系统评估方法论指南

ML系统评估方法论指南

v20260724
mlsys-experiments
本指南为机器学习系统设计了全面、严谨的评估框架。它详细指导作者如何选择代表性工作负载、与当前最佳基线进行公平比较、报告关键性能指标(吞吐量、p99延迟、内存和成本),并构建消融和扩展性证据,确保实验结果具有高可信度。
获取技能
375 次下载
概览

MLSys Experiments

Use this before the evaluation is frozen. An MLSys evaluation exists to answer one
compound question: *does the named mechanism deliver the claimed system-level payoff on
workloads that matter, at acceptable cost, for understood reasons?* Each clause needs its
own evidence, and this venue — which published the MLPerf Training methodology in its own
proceedings — holds measurement to benchmark-committee standards.

Workload selection: the make-or-break choice

"Unrepresentative workload" is the modal fatal objection at this venue. Defend the
choice explicitly:

  • Anchor on recognizable workloads: MLPerf-family tasks, widely used open models, or

published request traces. Bespoke workloads need a characterization section showing
their statistics (request rates, sequence lengths, arrival burstiness, model mix).

  • Cover the axis your mechanism exploits: a scheduler for bursty load must be tested on

bursty and uniform load; the uniform case bounds your claim, it does not weaken it.

  • Include at least one workload where you expect little or no gain, and show it. A

non-win row is the cheapest credibility purchase available.

  • State workload licensing/provenance — traces you cannot release constrain your future

artifact badges (see mlsys-artifact-evaluation).

Baseline discipline

  • Compare against the current strongest systems, version-pinned, in their recommended

configurations — and tune them with the same budget you gave your own system.
Document both budgets; asymmetric tuning is treated as invalidating.

  • Include the "do nothing" configuration where meaningful (e.g., default framework

behavior); reviewers use it to sanity-check absolute numbers.

  • If a relevant baseline cannot run in your setting, demonstrate the incompatibility

rather than asserting it.

The reporting quartet

Never report throughput alone. Each headline claim carries four coordinates:

CoordinateReport asClassic gaming pattern it prevents
Throughput/goodputUnder a stated latency constraintBatch-size inflation that destroys tails
Latencyp50 and p99 (p99.9 for serving), distributions not meansMean-only tables hiding stragglers
MemoryPeak device + host, at the measured configurationWins that only fit on 80GB parts, unstated
Cost/efficiency$ or joules per unit work, with price/meter basis"Faster" systems that are 4x more expensive

Quality (accuracy/perplexity or task metric) rides along whenever the technique could
plausibly change model output — quantization, sparsity, scheduling with dropping,
approximation of any kind. "No quality change" is a claim requiring a measurement.

Ablation = attribution

Systems papers stack techniques; reviews ask which component pays.

  • One ablation row per named mechanism, cumulative or leave-one-out — pick one scheme

and say which.

  • Ablate the insight, not only the code: if the claim is "lookahead scheduling wins,"

include the no-lookahead scheduler, not just the whole-system-off row.

  • Sensitivity sweeps over the parameters a deployer must set (window sizes, thresholds);

a mechanism needing per-workload hand-tuning must say so.

Scaling and sensitivity evidence

# experiment-matrix.yaml — one row per plotted point, generated, never hand-edited
sweep:
  models:    [llama-class-8b, llama-class-70b, moe-16e]
  hardware:  [1xA100, 8xA100-nvlink]        # state interconnect explicitly
  load:      [uniform-qps, bursty-trace-A]
  precision: [bf16, int4]
  trials: 5
  warmup_steps: 20
  metrics: [goodput@p99<250ms, p50, p99, peak_mem_gb, gpu_hours, quality]
  seeds: [1, 2, 3, 4, 5]
  • Plot trends, not points: gain versus model size, versus load, versus GPU count. A

single-configuration win invites the "will this survive scale?" objection you cannot
answer in the four-day response window.

  • Log everything; regenerate every figure from logs by script (measurement mechanics

and variance rules live in mlsys-reproducibility).

Statistical floor for systems numbers

Systems papers rarely need hypothesis-testing machinery, but they always need variance
honesty; the floor this venue's reviewers apply:

  • Five or more trials for any number in a comparison table; report spread (stdev or

IQR) and say which in the caption.

  • Overlapping spreads mean the difference is not a finding — either run more trials or

report the tie honestly; a "win" inside noise is worse than a reported tie because
one reviewer rerun destroys it.

  • For latency distributions, trials are not enough: within-run percentiles need enough

requests that p99 is estimated from hundreds of samples, not three.

  • Seed variation matters only where model quality enters the claim; systems noise

(placement, thermal state, co-tenants) dominates latency variance and is controlled
by measurement design, not seeds — the split is detailed in mlsys-reproducibility.

Evaluation-design review before running

Run this table exercise before burning GPU-hours: for each claim in the abstract, write
the figure that will support it, the workload it needs, and the strongest objection a
systems reviewer could raise — then check the matrix covers that objection. Claims
without a planned figure get cut from the abstract, not padded later.

Common reject patterns to design against

  • Microbenchmark gains that never appear end-to-end; always include the end-to-end run.
  • Gains measured only at one scale/hardware and claimed generally.
  • Quality silently degraded by the optimization, discovered by a reviewer from your own

appendix table.

  • Cost omitted where the honest accounting would erase the win.

Output format

[Evaluation readiness] strong / adequate / weak
[Workload defense] <named anchors + characterization + non-win case present?>
[Baseline symmetry] <versions pinned, tuning budgets equal + documented?>
[Quartet coverage] <throughput/latency-tails/memory/cost per headline claim>
[Attribution] <mechanism -> ablation row>
[Decision-critical missing run] <one experiment>
信息
分类 人工智能
名称 mlsys-experiments
版本 v20260724
大小 6.35KB
更新时间 2026-07-28
语言