技能 数据科学 可视化研究可复现性评估

可视化研究可复现性评估

v20260724
vis-artifact-evaluation
本指南详细介绍了可视化研究领域(如IEEE VIS)的科学可复现性标准。它指导作者如何为作品构建完整的可复现性包,以通过独立志愿者在干净环境下的测试。核心内容包括GRSI要求、使用Docker容器封装代码、提供数据溯源、以及建立图表与脚本的明确映射关系,确保研究结果的真实性和可追溯性。
获取技能
155 次下载
概览

VIS Artifact Evaluation

Use this for the reproducibility track. IEEE VIS does not use ACM-style artifact badges; its recognition is the Graphics Replicability Stamp Initiative (GRSI) — issued to TVCG papers as the TVCG Replicability Stamp — plus the conference's Open Practices program. Two things to internalize: the stamp is earned by an independent volunteer actually reproducing your results from a public archive, and the review artifact (anonymized, for your paper's reviewers) is not the same deliverable as the stamp artifact (de-anonymized, permanently archived).

What GRSI certifies (verify the current process)

Mechanism What it certifies What earns it
TVCG Replicability Stamp (GRSI) An independent volunteer reproduced the paper's results from your code/data A public repository + a single documented build/run path that regenerates the key figures/results
Open Practices disclosures Transparent reporting of open code, data, preprints, preregistration Filling the camera-ready Open Practices form honestly and posting the materials

Unlike a graded badge ladder, the stamp is binary: an evaluator either reproduces your results or does not. The failure mode is therefore always "it did not build/run on their machine," never "the idea was weak" — design for a stranger's clean environment.

What a GRSI volunteer opens first

Claim type First thing reproduced Common failure caught
A visualization technique/algorithm The build + a script that regenerates a key figure Undocumented deps; only-builds-on-authors'-GPU
A system/tool The install and a demo on bundled sample data Requires a private server, API key, or paid license
A perceptual/empirical study The analysis scripts that turn raw responses into the paper's stats Numbers in the PDF no script reproduces; raw data missing
A rendering result The pipeline + reference images with a comparison Non-deterministic output with no tolerance/seed documented

Assume the evaluator gives your package a bounded time budget on a clean machine. The first build and the first regenerated figure must succeed.

Packaging plan

[Container]   ship a Dockerfile or a pinned environment (requirements/lockfile, exact toolchain
              versions); avoid "install these 40 things by hand" and undocumented GPU/driver needs
[README]      one-screen orientation: what it is, how to build, how to run the demo, how to
              regenerate each figure/result, expected runtime and outputs
[Mapping]     an explicit table: paper figure/result -> script -> expected output
[Data]        the actual dataset or stimuli (or documented access), not just a pointer
[Determinism] seeds, tolerances, and reference images for anything stochastic or GPU-dependent
[License]     an OSI-approved license so others can reuse the visualization code
[Archive]     deposit in a DOI-issuing repository (OSF, Zenodo, Software Heritage) for permanence

Anonymized review artifact vs. stamp artifact

  • At submission (if double-blind): the supplemental code/data/video is anonymized for the paper's reviewers — no owner strings, lab names, institutional URLs, or identity-revealing demo links, and no live repository that discloses authors.
  • After acceptance: replace anonymized placeholders with the public, licensed, DOI-issuing archive; this is the version a GRSI volunteer reproduces and the camera-ready cites.

Worked vignette: stamping a technique + system paper

A paper contributes a new graph-layout technique and an interactive system. To target the stamp: ship a Docker image with the layout code pre-built; a run_demo.sh that lays out a small bundled graph and writes the teaser figure in under a minute; a reproduce/ directory whose scripts regenerate each quantitative figure from logged benchmark data; a figure-to-script mapping in the README; the benchmark graphs themselves with provenance; and an MIT/BSD license. State honestly which figures are turnkey and which need the full (slow) benchmark run.

Calibration

  • The GRSI review is independent of and after camera-ready; do not conflate them, and do not block the paper on the stamp result.
  • The exact GRSI submission process, the Open Practices requirements, and whether any element is mandatory vary by cycle — confirm on the current Open Practices page and the GRSI site.

Output format

[Target recognition] TVCG Replicability Stamp / Open Practices disclosures
[Artifact role] anonymized review artifact / public stamp artifact
[Contents] <code/data/stimuli/determinism aids/license>
[Clean-machine test] does build + demo + one regenerated figure succeed? yes/no
[Figure mapping] <figure/result -> script -> expected output present? yes/no>
[Fixes before archiving] <ordered list>
信息
Category 数据科学
Name vis-artifact-evaluation
版本 v20260724
大小 5.21KB
更新时间 2026-07-29
语言