Systematic Search & Synthesis (revedres-literature-synthesis)
When to trigger
- The protocol is fixed and it is time to search the literature exhaustively
- Searching feels ad hoc; you cannot yet report a reproducible PRISMA flow
- You have hundreds of records and need a defensible screening trail
- A reviewer at RER is likely to ask "why did you omit study X / database Y?"
Search to a documented PRISMA flow, not to memory
An RER systematic review's credibility rests on a reader's belief that you found everything that meets your criteria — and can prove it. Execute the protocol, logging every number for the PRISMA flow diagram (identification → screening → eligibility → included).
-
Run the registered search. Search every database in the protocol (ERIC, PsycINFO, Education Source, Web of Science, Scopus, ProQuest Dissertations) with the recorded strings, plus grey-literature and hand-searches of key journals. Record hits per source and the search date.
-
Deduplicate and log. Report records identified, duplicates removed, and records screened — exact counts.
-
Dual independent screening. Two screeners at title/abstract, then full-text, against the eligibility criteria; record exclusions with reasons at full-text (required by PRISMA). Report inter-rater reliability (Cohen's κ or % agreement) and how conflicts were resolved.
-
Supplement to saturation. Backward (reference lists of included studies) and forward (who cites them) snowballing; ask whether new searches still surface eligible studies. Document where they stop.
-
Extract into a structured dataset. Apply the codebook to every included study — this is the raw material for the framework, the tables, and the meta-analysis.
From extraction to synthesis (not summary)
Summarizing is restating each study; synthesizing is making the studies answer your question together. Maintain a coding dataset as you extract:
| Column |
What to capture |
| Study |
author–year; the included report (watch for multiple reports of one sample) |
| Sample/context |
learners, setting, grade/level, country — for moderator analysis and scope claims |
| Design |
RCT / quasi-experiment / correlational / qualitative — for risk-of-bias and weighting |
| Construct/measure |
exactly what was measured (so non-commensurable outcomes are not pooled) |
| Effect / finding |
effect size + variance (meta-analysis) or coded finding (narrative synthesis) |
| Risk of bias |
your appraisal on the a-priori tool (you do not re-run the study; you judge it) |
| Dependencies |
shared samples / multiple effects per study (drives the variance model) |
This dataset feeds the organizing framework, the forest plot and coding tables, and the even-handed treatment of conflicting evidence. You appraise the primary studies (you are the field's reviewer-of-record); you do not re-collect their data.
Education-specific search hazards
The education literature is scattered across disciplines and document types, which creates predictable holes:
-
Cross-disciplinary indexing. Relevant work hides in psychology (PsycINFO), economics (EconLit/NBER), sociology, and policy databases — searching only ERIC misses it. Map your constructs to each field's vocabulary.
-
Grey literature is large and consequential. Dissertations (ProQuest), technical and foundation reports, and What Works Clearinghouse / IES products carry many null and small-sample results; omitting them biases pooled effects upward.
-
Terminology drift. The same construct is named differently across eras and subfields (e.g. "self-regulation" vs. "metacognition" vs. "executive function") — build a thesaurus of synonyms into the search string.
-
Multiple reports of one study. Program evaluations spawn several papers on the same sample; collapse them to one unit or model the dependency, or you double-count.
Document how you handled each, so a reviewer sees the gaps were anticipated, not missed.
Checklist
Anti-patterns
- Searching from memory or one database (predictable, fatal coverage gaps at RER)
- A PRISMA diagram whose numbers do not reconcile (a red flag reviewers check)
- Single-screener inclusion with no reliability statistic
- Excluding grey literature without acknowledging the publication-bias risk it creates
- Pooling outcomes that measure different constructs into one "effect of X"
- Re-analyzing or "correcting" a primary study's raw data — you appraise, you do not re-collect
Output format
【Databases + date】<sources searched, search date>
【PRISMA counts】identified / dedup / screened / full-text / excluded-w-reasons / included
【Screening reliability】κ or % agreement; conflicts resolved by <method>
【Snowballing】backward + forward to saturation? Y/N
【Grey literature】included? Y/N — publication-bias implication noted? Y/N
【Coding dataset】codebook applied; dependent effects + shared samples flagged? Y/N
【Coverage risks】<any eligible study/database a reviewer could name as missing>
【Next step】→ revedres-organizing-framework (impose the conceptual spine on the corpus)
