EACL Artifact Evaluation

Use this to turn a paper's evidence into artifacts that survive review and become a public release. EACL runs through ACL Rolling Review, so the artifact lives two lives: an anonymized supplement attached at ARR submission, and a public release after commitment acceptance. Both are audited against the Responsible NLP checklist. Reopen the current checklist before packaging.

The two lives of an EACL artifact

Do not conflate them: the review supplement must contain no author-identifying strings, while the public release must contain exactly the identifying and licensing information the supplement omitted.

What belongs in an EACL artifact

• Code to reproduce the headline tables, with a top-level entry point.
• Data: the dataset or a loader plus a documented path to it; if redistribution is restricted, document access precisely rather than implying release.
• Prompts and decoding settings verbatim for any LLM-based result — these are part of the method, not an afterthought.
• Model outputs retained so scores can be re-computed without re-running expensive models.
• Annotation materials: guidelines, interface, pay information, and inter-annotator agreement.

Anonymized-supplement checklist

[ ] No author names in paths, file headers, LICENSE, or notebook metadata
[ ] Git history stripped or repo re-initialized
[ ] No personal hosting URLs (Drive/Dropbox) that identify authors
[ ] Prompts + decoding params included verbatim
[ ] Model outputs included for re-scoring
[ ] A README that reproduces at least one reported table
[ ] Smoke-checked (see resources/code/README.md)

Run the shared smoke checker before upload:

python3 ../../../shared-resources/ml-conference-methods/code/check_repro_package.py /path/to/anonymous-supplement

Licensing and documentation for the public release

• Choose a license appropriate to code (e.g. permissive) and data (respecting upstream dataset terms); the paper text should state it.
• Document intended use and known limitations of any released dataset — required by the checklist and expected by the European community's data-governance norms.
• Version the release with a tag that matches the camera-ready, so the Anthology PDF and the repo cannot drift.

Multilingual and lower-resource specifics

• If the artifact covers lower-resourced languages, document provenance and speaker/annotator context carefully; thin documentation of a low-resource dataset is a common EACL reviewer concern.
• Keep language codes and scripts explicit (ISO codes, script variants) so the artifact is usable by others working on those languages.

Output format

[Artifact stage] Anonymized supplement / Public release
[Contents] <code/data/prompts/outputs/annotation coverage>
[Anonymization] <pass/fail with specific leaks>
[Reproduces] <which reported table the README regenerates>
[Licensing + docs] <license, dataset terms, intended-use note>
[Gaps] <what a reviewer could still not reproduce>