The Replication Laboratory

The replication crisis and the AI opportunity

The scientific community faces a well-documented replication crisis. Studies across psychology, medicine, economics, and machine learning have found that a substantial fraction of published findings cannot be reproduced by independent researchers. Most failures stem not from fraud but from undocumented implementation details, missing preprocessing steps, unreported hyperparameter choices, hardware-dependent numerical precision, and data access barriers.

Traditional academic incentives actively discourage replication work. Graduate students need novel publications, not confirmatory studies. Journals prefer exciting new results over careful replications. The result: published claims accumulate without systematic verification.

AI-assisted replication changes the equation in several ways:

Container-per-session

The key architectural decision: every interaction between roles occurs in a fresh Docker container with fresh context. This enforces genuine cognitive separation — the Advisor reviewing a phase report truly does not have access to the RA’s context from producing it. This isn’t enforced by prompt engineering (“pretend you don’t know”) but by infrastructure — the context literally does not exist in the new container.

This solves multiple problems simultaneously:

Git as shared memory

The project’s git repository is the only persistent state. It serves as long-term memory (each session bootstraps by reading the state file and recent reports), communication channel (the Advisor and RA “talk” by writing documents and committing them), audit trail (every commit is tagged with role, phase, and timestamp), and handoff mechanism (the state file records who has the baton).

a8f3d2c [advisor] Final replication report approved
e7c4b1a [advisor] Synthesis: Final report generated
9b2e5f3 [ra] Phase 4: Comparison analysis complete
7d1c8a4 [advisor] Phase 4 review: APPROVED
5e9f2b1 [ra] Phase 3: Baseline model training complete
3a7d4c2 [advisor] Phase 3 review: APPROVED - results within tolerance
1f8e6b3 [ra] Phase 2: Data preprocessing complete
4c2a9d1 [advisor] Phase 2 review: APPROVED
8e5b3f2 [ra] Phase 1: Environment setup complete
2d7f1c4 [advisor] Tactical workplan created
6b9e4a3 [human] Plan approved: proceeding with execution
9c1d8f5 [advisor] Replication plan created
7a4e2b1 [init] Project initialized: feng-2024-concept-binding

Three roles, infrastructure-enforced separation

The system prompts are the highest-leverage component — carefully designed instructions that encode all architectural discipline and behavioral patterns. Each role has a distinct persona, responsibilities, decision authority, and workflow. Critically, each runs in its own ephemeral container — the separation is not a prompt instruction but a hard infrastructure boundary.

Scope enforcement

The Advisor’s central job is preventing scope creep. Common failure modes: the RA finds an interesting discrepancy and wants to investigate 17 variations; the RA notices additional claims beyond stated scope; the RA gets distracted by tooling improvements. The Advisor catches these in review and redirects focus, logging tangential findings in an appendix for future work rather than pursuing them.

The human as third voice

The human participates asynchronously by dropping comment files into comments/ at any time. At each phase boundary, the Advisor checks for new comments and incorporates relevant ones. The human uses the same file-mediated protocol as the agents — a third voice in the conversation, not a supervisor peering over the agent’s shoulder.

From initialization to completion

A replication project moves through a well-defined state machine. Each transition is a container session; each session reads the current state, does its work, commits results, and updates the state file for the next session.

Graduated autonomy

The orchestrator enforces configurable human checkpoints  — phase transitions where execution pauses for approval. The default for v0.1: plan approval and final report review require human oversight; all intermediate phase transitions (RA → Advisor review, Advisor → RA execution) are autonomous.

SLURM integration

Research replication often requires long-running GPU jobs. The RA never waits for SLURM jobs — it uses a fire-and-forget pattern: submit the batch script via SSH to the headnode, record the job ID in the state file, commit, and exit. The orchestrator polls sacct every 60 seconds. When the job completes (or fails), it launches a new RA session with that information. If the job fails, the RA decides how to handle it: resubmit with different resources, debug, or escalate.

Components

Container system

Ubuntu 24.04 LTS base with Claude Code, Pandoc + LaTeX toolchain (texlive, pdflatex), the Python scientific stack (numpy, pandas, scipy, matplotlib, scikit-learn, PyTorch, transformers), and standard build tools. The build script detects the available container runtime and supports Docker, Podman, and Singularity/Apptainer for HPC environments.

At runtime, the container gets three volume mounts: the project repository (read-write), system prompts (read-only), and SSH keys for SLURM access (read-only). It can read and write project files, SSH to the SLURM headnode, and access the internet for package downloads. It cannot modify files outside the workspace, access the host filesystem, or persist state between sessions.

CLI tool

The human interface is a Python dispatcher with four commands:

replab init \
  --paper ./feng-2024-concept-binding.pdf \
  --code https://github.com/feng/binding-ids \
  --scope "Replicate core binding ID mechanism (Sections 3-4)" \
  --compute-budget "50 GPU-hours" \
  --involvement "checkpoints-only"

Document generation

Iterative documents (replication plan, phase reports, reviews, tactical workplan) use Markdown rendered to PDF via Pandoc — fast to produce, easy to diff in git, easy for AI agents to write correctly. The final replication report uses LaTeX for archival quality, publication-ready figures, and proper bibliography management. This gives the “all documentation produces PDFs” property without forcing the RA to fight with LaTeX during iterative work.

State file

The .replication-state.toml file is the single source of truth for project state. It tracks the current phase and role, completed phases, pending SLURM jobs with their IDs and status, budget allocation and usage (tokens and GPU-hours), checkpoint configuration, and whether the project is awaiting human approval.

[project]
name = "feng-2024-concept-binding"
status = "executing"

[execution]
current_phase = "phase3-baseline"
current_role = "ra"
phases_completed = ["tactical_workplan", "phase1-setup", "phase2-data"]

[pending_jobs]
job_87654 = { phase = "phase3-baseline", status = "RUNNING",
              partition = "gpu_requeue" }

[budgets]
total_token_budget = 500000
tokens_used = 127000
total_compute_hours = 50.0
compute_hours_used = 15.0

Four papers, four peer reviews

The Replication Laboratory has completed four end-to-end replications, each producing a structured final report and an independent peer review. The results span the full range of outcomes — from strong confirmation to significant non-replication — demonstrating that the system produces honest assessments rather than confirmatory exercises.

Status and next steps

The v0.1 proof of concept is complete: four papers replicated end-to-end with full Advisor/RA/Reviewer pipelines, producing structured reports and independent peer reviews. The infrastructure — container system, CLI, orchestrator, three-role prompt suite — is validated and operational.