技能 编程开发 学术论文严谨实验设计与评估

学术论文严谨实验设计与评估

v20260724
oopsla-experiments
本指南旨在为撰写和评估学术论文提供一套严谨的实验框架。它强调了“声称类型”与“证据类型”之间的匹配关系(例如,声称性能提升时,必须提供基准测试证据)。内容涵盖如何建立可靠的基线、定义工作负载规则以及进行统计学上的细节核查,确保论文达到顶级学术会议的要求。
获取技能
265 次下载
概览

OOPSLA Experiments

OOPSLA's evaluation question is not "is there a big table?" but "does the
evidence type match the claim type?" The venue's published scope runs
from mathematical formalisms to empirical studies, and its exemplars span
benchmark suites, measurement methodology, corpus mining, and language
experience reports (resources/exemplars/library.md) — so the first design
act is choosing the right instrument, and the second is executing it to the
SIGPLAN Empirical Evaluation Guidelines standard that reviewers apply
checklist-in-hand (oopsla-reproducibility operationalizes the pillars).

Claim-type → evidence-type routing

Claim typePrimary evidenceCommon OOPSLA failure
"Faster / cheaper"Benchmarks vs strongest baseline, variance reportedWeak baseline; startup vs steady-state conflated
"More expressive / safer"Formal result + programs witnessing the boundaryExpressiveness asserted by example only
"Programmers benefit"User study or field data with a designAnecdote from the authors' own use
"Occurs in practice"Corpus study with stated selection ruleConvenience sample of famous repos
"The design generalizes"Second instantiation (language/runtime/domain)Single-host generalization claims
"Semantics is right"Mechanization or proofs + conformance testsCalculus untethered from the implementation

A paper may need two rows; it rarely supports five. Cutting a claim is
cheaper than defending its missing evidence through a Major Revision.

Baselines and workloads that survive scrutiny

  • The baseline is what a skeptical expert would actually use today, tuned the

way its own paper tunes it — document versions and flags in the paper.

  • Workload selection needs a rule (suite version, corpus filter, sampling

frame) stated before results; exclusions listed with reasons. Curated-only
workload sets are the most quietly fatal reviewer finding.

  • For LLM-era tooling claims, hold out for contamination: date-split corpora,

and report sensitivity to prompt/configuration where relevant.

  • Negative controls: include a configuration where your mechanism should

not help and show that it doesn't. Nothing signals honesty faster.

Sizing experiments to the round calendar

The two-round system changes experimental economics. Between an R1 verdict
and the R2 resubmission there are only months, so:

Design now, before Round N:
  - matrix of runs a reviewer could plausibly demand (extra baseline,
    larger corpus, second platform) with wall-clock + hardware cost each
  - keep the harness parameterized so a demanded cell is a config change
  - archive raw results per run (the ledger of oopsla-reproducibility)
Payoff: a Minor Revision executes in days; a Major Revision's
expectation list maps to known cells instead of new engineering.

Analysis floor

  • Repetition counts, warmup handling, and dispersion (CI or IQR) for every

performance number; significance or effect size where comparisons are the
claim.

  • Summaries chosen deliberately (geomean for ratios) and stated.
  • Human-subject work: sample size rationale, task design, and ethics/IRB

status — the venue takes the human-aspects lane seriously enough to review
it by social-science norms.

  • Case studies report failures encountered, not only successes; an

experience report with zero friction reads as marketing
(oopsla-writing-style).

Output format

[Routing] claim → evidence rows used + mismatches found
[Baseline audit] strongest-sensible test: pass / gaps
[Workload rule] stated / absent; exclusions justified: yes/no
[Demand matrix] anticipated reviewer demands with cost estimates
[Analysis floor] repetitions/dispersion/summary-statistic compliance
信息
分类 编程开发
名称 oopsla-experiments
版本 v20260724
大小 4.09KB
更新时间 2026-07-28
语言