UIST does not have a single evaluation orthodoxy; it has a matching rule. The evaluation must measure the claim the artifact makes, and different artifact types make categorically different claims. The classic failure is the ritual study: a 12-participant SUS-and-task-time exercise bolted onto a toolkit paper whose actual claim — "this abstraction lets developers build a class of things" — no usability score can support.
| Artifact claim | Primary evidence | Supporting evidence |
|---|---|---|
| "This technique outperforms the status quo" (input, pointing, text entry) | Controlled within-subjects comparison against the real incumbent | Learning-curve data over sessions |
| "This hardware enables new sensing/actuation" | Technical characterization: accuracy, range, SNR, latency across conditions | Small usage session showing humans can operate it |
| "This toolkit lowers the floor / raises the ceiling" | Demonstration portfolio + developer study or expert walkthrough | Code-size/API comparisons, workshop deployments |
| "This pipeline makes X automatic" | Quantitative accuracy on held-out real inputs + failure taxonomy | End-to-end examples spanning input diversity |
| "This system supports task Y better" | Task-based study instrumented around Y | Log analysis, think-aloud excerpts |
Two smaller lanes are legitimate at UIST and often misused: demonstration-only evidence carries a paper when the enabling novelty is extreme (the demo is the result), and expert sessions (3-6 domain professionals in deep sessions) beat 20 novices when the system targets professionals.
The technical evaluation is the half UIST reviewers read most skeptically, because they build things too:
Compare against what a skilled practitioner would actually use today — the shipping technique, the standard library, the commercial device — not a strawman reimplementation. When no incumbent exists (genuinely new capability), say so and substitute ablations: which component of the pipeline buys which capability.
uist-workflow).Interactive systems are pipelines, and reviewers want the credit assignment:
uist-author-response).| Recurring objection | Pre-emption |
|---|---|
| "The comparison baseline is a strawman" | Use the shipping incumbent; state its version and settings |
| "Latency/accuracy measured under ideal conditions" | Report the envelope and the tails, not the demo-day sweet spot |
| "The study tasks were designed around the system's strengths" | Include at least one task chosen from prior work's protocol |
| "n = 12 cannot support this claim" | Match claim size to power; concede small effects |
| "The applications are all variations of one scenario" | Diversify the portfolio across input contexts and user types |
| "No failure analysis" | A measured failure boundary section, always |
For each contribution claim C1..Cn in the introduction:
[ ] name the evaluation section that tests it
[ ] name the measure and the condition range
[ ] name the comparison point (incumbent / ablation / none-with-justification)
[ ] name where dispersion is reported (SD, CI, percentiles)
Orphan claims -> weaken the claim or add evidence
Orphan studies -> cut, or promote to a claim the intro actually makes
Run this audit against the video figure too: any capability the video shows that no
section measures is a question a reviewer will ask in exactly those words (rebuttal
implications in uist-author-response).
Put the strongest number in the abstract with its condition attached. In the body,
one summary table of the technical characterization beats four bar charts; per-cell
detail goes to the appendix, raw harness and logs to the supplement (packaging rules
in uist-artifact-evaluation and uist-reproducibility).
[Claim type] technique / hardware / toolkit / pipeline / task-support
[Evidence match] matched / partial mismatch / ritual-study warning
[Envelope coverage] <conditions characterized vs conditions claimed>
[Baseline honesty] real incumbent / ablation / missing
[Weakest link] <the claim most exposed in review + cheapest fix>