RecSys Reproducibility

Reproducibility is unusually load-bearing at RecSys for two reasons. First, the venue runs a dedicated Reproducibility Track for papers that repeat, refute, or re-scope prior results and for datasets and frameworks that enable future reproduction. Second, the field's own literature documented how often reported recommender gains fail to survive a fair re-evaluation, so reviewers of regular papers read reproducibility signals as a proxy for whether the gains are real.

Why recommender results drift: the usual suspects

A reproducibility-strong RecSys paper closes each row with an artifact, not a promise: the config file is the documentation.

Minimum reporting block for any empirical RecSys paper

Put this in the paper (it fits in ~0.3 page and pre-empts three review objections):

• Datasets with versions and filtering; train/validation/test usage; the split protocol.
• Tuning protocol: search space, budget, selection metric, validation split — symmetric across systems, baselines included.
• Metrics: which ones, the cutoff, and whether ranking is over the full item catalog or a sampled candidate set.
• Seeds: how many, and whether tables show mean, sd, and a significance test on close results.
• For any deployment or off-policy claim: the estimator, its assumption, and the logged propensities.

The Reproducibility Track specifically

A reproduction paper is first-class work here, not a lesser contribution.

[reproduction scope]  which prior papers, which claims, which datasets
[matched setup]       identical splits, metrics, and — critically — equal tuning budget
[divergence report]   where results reproduce, where they do not, and the diagnosed cause
[new context]         a new domain, dataset, or baseline that tests generality
[artifact]            a runnable pipeline others can extend

Honest divergence is the contribution: "we could not reproduce claim X under matched tuning, and here is why" is publishable when the diagnosis is rigorous.

Vignette: a session-model reproduction

Consider reproducing three published session recommenders. Its reproducibility spine: the exact dataset versions and k-core filter, a temporal split shared across all systems, an equal tuning budget so no method is advantaged by search, full-ranking metrics at a fixed cutoff, five seeds with mean ± sd, and one honest paragraph on the one result that did not reproduce and the preprocessing difference that explains it.

Degrees of reproducibility

• Turnkey: one command regenerates each ranking table from logged seeds.
• Scripted: scripts exist but need documented manual steps or an external dataset download.
• Descriptive: prose detailed enough that a competent reader could rebuild the pipeline.

For RecSys, the offline pipeline should be turnkey because reviewers actually re-run it; a production A/B result may stay descriptive with the deviation documented. State the achieved level honestly rather than overpromising a turnkey build that fails on a clean machine.

Output format

[Claim inventory] <claim -> evidence location>
[Reproducibility status] strong / partial / weak
[Drift risks] <dataset version / split / baseline tuning / sampled metrics / seeds / propensities>
[Paper fixes] <must appear in main PDF>
[Repository fixes] <anonymous-repo additions>