MICRO's evaluation culture is instrument-centric: the community knows exactly what
each measurement tool can and cannot prove, and reviewers score the *match between
claim and instrument* before they look at the numbers.
| Instrument | Proves | Cannot prove | Typical tools |
|---|---|---|---|
| Analytical / trace model | First-order potential, limit studies | Interaction effects, timing | custom, trace-driven models |
| Cycle-level simulation | Relative performance of mechanisms | Absolute wall-clock, physical cost | gem5, ChampSim, Sniper, Ramulator/DRAMsim3 |
| Power/area models on top of sim | Energy and area trends | Sign-off-quality numbers | McPAT, CACTI, Accelergy |
| RTL + synthesis | Timing closure, real area/power at a node | Full-system performance | Verilog/Chisel + synthesis flow |
| FPGA prototype | Functional correctness at scale, OS interaction | ASIC frequency/power | FireSim, custom boards |
| Silicon measurement | Everything, for that one chip | Generality across designs | perf counters, power rails |
Rule: claim one rung below your strongest instrument. Cycle-level simulation
plus McPAT supports "reduces memory-stall cycles by X% with ~Y mm² estimated
overhead"; it does not support "improves datacenter TCO." The lineage here is the
venue's own: CACTI 6.0 (MICRO 2007) and McPAT (MICRO 2009) were published at MICRO
precisely because the community polices modeling fidelity.
default: wide OoO, capable branch predictor, competent multi-stream prefetcher,
realistic DRAM timing. A win over gem5's out-of-the-box config is a non-result.
your simulator with its published parameters — not just numbers copied from its
paper under a different config.
what fraction of the headroom you capture.
additional cache/predictor capacity in at least one comparison — the
"iso-storage" check reviewers ask for in rebuttal anyway.
subset), PARSEC/GAP for multithreaded, MLPerf or named DNNs for accelerators,
plus the domain traces your mechanism targets.
budgets per region. Cherry-picked "representative regions" without a stated
selection rule are a rebuttal magnet.
and never geomean-only (it hides the regressions reviewers will hunt for).
# the only defensible summary statistic for speedups
from math import prod
def geomean(xs):
return prod(xs) ** (1.0 / len(xs))
speedups = per_workload_ipc_new / per_workload_ipc_base # elementwise
print(f"geomean {geomean(list(speedups)):.3f}, "
f"min {min(speedups):.3f} ({worst_workload}), "
f"regressions: {(speedups < 1.0).sum()} of {len(speedups)}")Every design decision named in the mechanism section owes the evaluation section a
figure or table row:
to show each earns its area.
core counts, LLC capacities, DRAM bandwidth. The mechanism should degrade
gracefully at the sweep edges — cliffs demand explanation in the text.
access pattern designed to defeat it. Reporting a bounded loss builds more trust
than an unbroken win column.
Two credibility checks reviewers apply that authors often skip:
windows on memory-bound workloads mostly measure the warmup transient. State
warmup and detailed lengths, and show at least once that doubling them does not
move the headline.
results, show your baseline's absolute behavior is sane: IPC or MPKI within
the published envelope for the same suite and a comparable config. A baseline
whose branch predictor achieves 2x the published MPKI of the modeled design
invalidates the comparison silently.
For multicore results, disclose how heterogeneity was handled: mix construction
rule, per-mix repetitions, and whether throughput is weighted speedup, harmonic
mean, or raw IPC sum — each answers a different question and reviewers check that
the metric matches the claim (fairness claims need a fairness metric).
labeled as estimates.
claims justified.
paragraph.
[Instrument] <rung used> — claim height matches: yes / no (quote the overclaim)
[Baseline strength] product-like config / best-prior reimplemented / oracle bound /
iso-storage check: present-absent each
[Workloads] suite + subset rule + sampling disclosure: complete / gaps listed
[Summary stats] per-workload + geomean + regression count: yes / no
[Ablations] each mechanism component covered: list of unablated components
[Sensitivity] parameters swept vs parameters hardcoded
[Overheads] storage / area / power / latency / energy: quantified-missing each