Psychological Science holds analyses to high credibility standards: effect sizes with confidence intervals for major results, full disclosure of how the data were handled, and a clean confirmatory vs. exploratory separation. Analysis scripts and data are shared and can be checked.
psci-open-science-and-transparency).A preregistered two-study package on selective attention. Study 1 (N = 240, between-subjects) tests whether a brief mindfulness induction reduces attentional capture by emotional distractors. The confirmatory analysis is a single preregistered contrast on reaction-time cost.
Confirmatory (preregistered) — Study 1
Effect: induction vs. control on capture cost (ms)
d = 0.34, 95% CI [0.08, 0.59], t(238) = 2.66, p = .008
Sensitivity: holds with/without the 6 preregistered RT-outlier exclusions
(d shifts 0.34 → 0.31), and under log-RT (d = 0.33)
Exploratory (labeled) — Study 1
Trait-anxiety × condition interaction surfaced post hoc; reported as
exploratory, flagged for confirmation in Study 2's preregistration
Confirmatory (preregistered) — Study 2 (N = 300, direct + extension)
Replicates direct effect (d = 0.29, 95% CI [0.06, 0.51]) and
preregisters the anxiety moderation that was exploratory in Study 1
Why this passes Psychological Science scrutiny: every confirmatory number carries an effect size and a CI; the anxiety interaction is honestly demoted to exploratory and then promoted to confirmatory only after preregistration in Study 2; the sensitivity line pre-empts the "fragile-to-exclusions" reviewer.
| Reviewer pushback | What it signals here | Psychological Science fix |
|---|---|---|
| "p = .048 — too close to the line, and the CI nearly spans zero" | post-credibility-revolution distrust of just-significant single tests | report the CI prominently, add the Study 2 replication, lead with the pooled estimate |
| "Which exclusions were preregistered?" | suspicion of undisclosed forking paths | give the disclosure table: rule, count, preregistered vs. post hoc, and the estimate with vs. without |
| "Means hide the distribution" | bar-of-means aesthetic distrusted | recompute and show effect size + CI; route exhibit to psci-tables-figures |
| "Is this confirmatory?" | HARKing concern | point to the preregistration timestamp; relabel anything generated after data as exploratory |
| "Reviewer 2 could not rerun your code" | reproducibility gate | ship a fresh-session run log; see psci-open-science-and-transparency |
Run the battery, don't just enumerate it. Full map:
execution-with-mcp. Psychological Science is short-format experimental psychology with strong open-science norms; preregister, run randomization inference, and report effect sizes with family-wise corrections.
romano_wolf (step-down FWER) or
benjamini_hochberg — report the adjusted threshold.oster_delta / sensemakr.wild_cluster_bootstrap (few clusters), twoway_cluster / conley;
multilevel data → cluster at the right level.audit_result(result_id) lists the missing checks and the
exact suggest_function for each.etable / did_summary_to_latex from the handle — no retyped numbers.Keep the decisive checks in the body and the exhaustive battery in the supplement. See the executed chain in the JF execution walkthrough.
【Main result】effect size + confidence interval + meaning
【Disclosure】N-determination + all exclusions + all conditions + all measures reported? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Inference】assumptions/diagnostics, MHT handled?
【Reproducible】scripts + data dictionary + fresh-session check? [Y/N]
【Next】psci-tables-figures
../../resources/external_tools.md — effectsize, emmeans, metafor, JASP/jamovi, reproducible-report tooling../../resources/official-source-map.md — statistical and disclosure requirements