Skill UI

A comprehensive guide for designing, executing, and reporting high-stakes research studies in Software Engineering (SE), tailored to meet the stringent standards of top academic conferences like ICSE. It covers best practices for evidence-claim matching, advanced statistical rigor (non-parametric tests, effect sizes), handling AI model ablations, and implementing threats-driven design to ensure findings are robust, valid, and maximally reproducible.

This guide provides rigorous standards for designing, validating, and reporting security research, particularly for high-stakes academic conferences like IEEE S&P. It covers critical areas including adaptive adversary modeling, measurement validity (sampling, ethics), statistical reporting, establishing strong baselines/ablations, and adhering to ethical constraints (IRB, authorization) to ensure evidence is robust and reproducible.

This guide provides comprehensive guidelines for designing, conducting, and auditing empirical studies in software engineering. It emphasizes achieving scientific rigor by matching every claim to specific, measured evidence, ensuring fair baselining (equal budget), and applying appropriate advanced statistics (non-parametric tests, effect sizes). Essential reading for researchers submitting papers based on performance evaluations.

A guide for authors submitting to theoretical computer science venues (ITCS). It strictly defines what constitutes valid evidence for a mathematical claim, emphasizing that complete proofs are paramount. The guidelines detail how to use worked examples, separating objects, and appropriately scoped computations, while strongly advising against using empirical benchmarks or performance metrics.

This guide provides comprehensive methodological advice for academic submissions in Information Systems (IS), particularly for journals like JAIS. It covers selecting and rigorously defending diverse research designs—including behavioral experiments, causal inference (Economics of IS), design science, qualitative studies, and literature reviews. Use it when you need to validate the suitability of your methodology against your research claims and underlying phenomena.

A comprehensive guide for designing and presenting rigorous experiments for submissions to the Journal of Software (JOS). It guides users through setting clear Research Questions (RQ), selecting real-world datasets and baselines, applying advanced statistical testing (effect size, significance), conducting ablation studies, and addressing threats to validity (internal, external, conceptual). This ensures the evidence strongly supports the claims, meeting top-tier academic standards.

A comprehensive guide detailing the high standards required for presenting security evidence at major conferences. It covers methodologies for proving claims—including attacks, defenses, and detection systems—across live internet measurements, controlled testbeds, and emulation. Key requirements include adaptive adversary modeling, rigorous measurement hygiene, and adherence to ethical review protocols to ensure results are reproducible and relevant to real-world networks.

This guide provides a framework for designing and auditing the experimental rigor of academic papers, particularly in systems and programming languages. It emphasizes the crucial link between a claim type (e.g., 'Faster') and the required evidence type (e.g., benchmarks, user studies, formal proofs). Learn how to establish robust baselines, define clear workload rules, size experiments for conference cycles, and ensure statistical compliance for maximum impact.

A comprehensive guide for structuring academic system evaluation papers (e.g., OSDI). It dictates how to frame experiments around core research questions, select robust baselines, utilize realistic workloads (including adversarial tests), and apply rigorous measurement disciplines (e.g., reporting distributions, p99s) while maintaining conciseness within strict page limits.

This guide provides best practices for designing and auditing academic evaluations of programming language features and compilers, following standards used in venues like PLDI. It covers selecting defensible benchmark suites, establishing strong baselines, performing targeted ablations, and measuring multiple performance metrics (runtime, compile time, memory) to ensure research claims are reproducible and scientifically sound.

This guide outlines the rigorous standards for presenting 'evidence' in highly theoretical computing venues. It emphasizes that results must be supported by deep mathematical proofs, establishing matching upper and lower bounds (tightness), and meticulously auditing all underlying model assumptions (e.g., timing, failure modes). It shifts focus from empirical benchmarks to formal correctness.

Provides detailed guidelines for submitting empirical papers to POPL. It teaches authors how to correctly match specific claims (e.g., soundness, precision) with appropriate evidence types (proofs, case studies, effort data). The core principle emphasized is that evidence must demonstrate realizability but never substitute for the formal theorem.