Reproducibility at NAACL is enforced through a document, not a badge: the Responsible NLP checklist travels with the submission, reviewers read it against the paper, and answers contradicted by the PDF are grounds for rejection without review under current ARR policy. The working stance: every checklist answer is a claim you are prepared to defend in the author response.
| Moving part | Pin it as | Why it decays |
|---|---|---|
| Hosted LLM APIs | Model identifier + query date range | Providers swap weights behind stable names |
| Open-weights models | Exact checkpoint hash or revision tag | "Latest" changes under you |
| Decoding | Temperature, top-p, max tokens, seed policy, n samples | Unstated sampling makes numbers unrepeatable |
| Prompts | Verbatim strings, all variants, selection rule | "We used a standard prompt" reproduces nothing |
| Tokenizers / normalization | Version + Unicode normalization form | Silent retokenization shifts multilingual scores |
| Eval metrics | Implementation + version (not just the metric name) | Scorer variants disagree by whole points |
| Data splits | Published split files or generation script + seed | Ad-hoc splits are unrecoverable |
Papers committed to NAACL disproportionately evaluate across languages, and multilingual pipelines decay in language-specific ways: normalization that strips combining diacritics, sentence splitters that fail on Spanish inverted punctuation, tokenizers that fragment agglutinative morphology (Nahuatl, Quechua, Guaraní), and translation-based baselines whose MT system version was never recorded. Log per-language preprocessing explicitly — a single global "we lowercase and tokenize" line hides exactly the steps that differ across the languages you claim to cover.
# repro-manifest.yml — include in the supplement
models:
- id: example-lm-7b, revision: a1b2c3d, dtype: bf16
- id: hosted-model-x, api_dates: 2026-05-02..2026-05-19
decoding: {temperature: 0.0, max_tokens: 512, samples: 1}
prompts: prompts/ # verbatim, one file per task x language
data:
- name: task_es, split_files: splits/es/, license: CC-BY-4.0
- name: task_gn, split_files: splits/gn/, license: community-terms
scoring: eval/score.py (chrF++ via sacrebleu 2.4.x, signature logged)
hardware: 4x A100-80GB, ~310 GPU-hours total
seeds: [13, 42, 2026] # every table reports mean/sd over these
known_gaps: human eval not re-runnable; transcripts included
The known_gaps line is the point: an honest boundary between re-runnable
and merely documented is what distinguishes a defensible checklist from a
hopeful one.
Reproducibility rigor is usually sold as ethics; at NAACL it is also tactics. When a reviewer asks "would the result hold with a different prompt phrasing?" a team with a pinned manifest and an experiment ledger answers inside the window with numbers; a team without one answers with adjectives. Concretely, the manifest converts three recurring review moments:
Meta-reviews reward the second answer pattern visibly; the checklist is read as a proxy for whether the authors could defend any number under pressure.
Name the level in the paper. NAACL reviewers penalize mismatch between claimed and actual level far more than they penalize level 3 honestly held — especially when community data-governance terms, common in Americas-language work, are the stated reason.
[Checklist audit] <answer -> evidence location -> holds/contradicted>
[Pin table status] <each moving part -> pinned/missing>
[Per-language gaps] <language -> unlogged preprocessing or scorer>
[Release level] re-runnable / verifiable / documented (+ reason)
[Fixes before upload] <ordered>