Research Summary
Build a deep research agent as a stateful workflow, not a long chat loop
Executive takeaways
- The right mental model is a research operating system, not a prompt: planning, evidence gathering, source ranking, contradiction handling, synthesis, and citation verification should all be explicit stages.
- The best default architecture is supervisor/planner + bounded research workers + external typed state + search/extract-first retrieval + browser fallback + citation verifier.
- A single-agent loop is acceptable for an MVP, but it breaks down quickly on coverage, provenance, citation discipline, and long-horizon runs.
- Externalize state early: research brief, open questions, source registry, evidence notes, claims, contradictions, citations, checkpoints, costs, and tool traces should live outside the model context.
- Use breadth-first then depth-first evidence gathering: map the space first, then deepen only on the highest-value gaps and contradictions.
- Treat citations as first-class structured objects. Every claim should trace back to a note, source, retrieval timestamp, and exact quote/span or extracted artifact.
- Browser automation should be a fallback, not the default. Start with search + extraction, then use the browser only for high-value dynamic or blocked pages.
- Strong systems use explicit stop gates: enough source diversity, key claims supported, contradictions checked, and diminishing returns reached.
- Evaluation should cover not just answer quality but citation correctness, coverage, contradiction handling, replayability, cost, and latency.
Recommended architecture
Minimal architecture: typed single-researcher loop
Use this when you want the simplest reliable first version.
Components:
- ResearchController — owns state transitions, retries, budgets, and checkpoints.
- Structured Research Agent — chooses the next action from a typed action set.
- Search tool — query in, ranked results out.
- Extract tool — URL in, normalized document out; browser fallback on failure.
- Evidence store — source registry, notes, claims, contradictions, citations.
- Report generator — writes from typed evidence, not the whole browsing transcript.
- Citation verifier — checks every material claim against source snippets/spans.
Flow:
1. turn user request into a research brief
2. broad search and extraction
3. note extraction into typed evidence objects
4. gap and contradiction analysis
5. targeted follow-up searches
6. synthesis into claim table and outline
7. draft report
8. verify/repair citations
9. finalize
Advanced architecture: supervisor + parallel evidence workers + verifier
Use this when you need better breadth, lower wall-clock time, or production reliability.
Components:
- Supervisor / lead researcher — decomposes the question, manages budgets, chooses when to broaden, deepen, or stop.
- Planner — builds the initial question tree, source classes, and evaluation criteria.
- Research workers — each handles a narrow subquestion in an isolated context and returns only structured evidence packets.
- Source ranker — scores authority, recency, independence, specificity, and bias.
- Evidence graph / state store — models Sources, Documents, Notes, Claims, Questions, Contradictions, Citations, WorkerRuns, ToolCalls.
- Contradiction handler — forces the system to resolve or explicitly preserve disagreement.
- Synthesis agent — builds the outline from supported claims.
- Citation verifier — attaches and validates citations claim-by-claim.
- Trace/replay layer — stores model inputs/outputs, tool observations, state diffs, costs, and versions.
This is the architecture most likely to feel like “real deep research” instead of fancy browsing.
Custom-harness design
1) Single-agent loop vs planner/executor split
Single-agent loop
Pros:
- fine for narrow questions and short reports
Cons:
- context fills with noisy observations
- bad at breadth and provenance
- citation bookkeeping gets messy
Planner/executor split
Pros:
- better breadth/depth control
- easier evaluation and replay
- easier to specialize prompts or models by phase
Cons:
- needs explicit merge logic and schemas
Recommendation: start with one structured controller that plans explicitly, then graduate to supervisor + workers + verifier.
2) One long-running context vs state externalization
Long context is fast to prototype but expensive, fragile, and hard to replay.
Externalized state gives you durability, inspectability, compaction, and reproducibility.
Recommendation: keep only a compact state summary in-context:
- top claims/evidence summaries
Everything else belongs in storage.
3) Serial tools vs parallel workers
Serial is easier to debug but slow and more vulnerable to early-path bias.
Parallel workers improve breadth and latency, but require dedupe, ranking, and merge discipline.
Recommendation: parallelize evidence gathering, not final judgment. Workers should return structured packets, never final prose.
4) Browser-first vs search/extract-first
Search/extract-first is cheaper, faster, more structured, and better for dedupe and citations.
Browser-first is slower and flakier, but essential for dynamic pages, forms, JS-heavy sites, screenshots, and interactive verification.
Recommendation: use a cascade:
1. search API
2. extraction
3. crawl/map if needed
4. browser fallback
5. screenshot/vision only when visual evidence matters
5) Prompt-only control vs typed state / finite-state machine
Prompt-only control is flexible but brittle.
Typed state with a flexible FSM gives you deterministic gates, retries, checkpointing, and testability.
Recommended phases:
Do not let the model jump to final output until evidence gates pass.
Control-loop pseudocode
Typed action schema
type (search | extract | browse | note | synthesize | verify_citation | stop)
Observation schema
status (ok | retryable_error | fatal_error)
Minimal loop sketch
def deep_research(task: str, budget: Budget) -> Report:
state = ResearchState.new(task=task, budget=budget)
checkpoint(state)
state.brief = llm_structured(
prompt="Turn the task into a research brief, subquestions, and source strategy.",
input={"task": task},
schema=ResearchBrief,
)
state.phase = "BROAD_SEARCH"
checkpoint(state)
while not should_stop_research(state):
decision = llm_structured(
prompt=RESEARCH_CONTROLLER_PROMPT,
input={
"brief": state.brief,
"phase": state.phase,
"open_questions": state.open_questions(),
"claims": state.claim_summaries(),
"sources": state.source_summaries(),
"budget": state.budget_status(),
"allowed_actions": allowed_actions(state.phase),
},
schema=NextAction,
)
obs = execute_action_with_retries(decision, state)
append_trace(decision, obs)
if obs.status == "retryable_error":
state.errors.append(obs)
continue
if obs.status == "fatal_error":
state.failures.append(obs)
state = transition_after_failure(state, obs)
checkpoint(state)
continue
state = reduce_observation(state, decision, obs)
state = update_phase(state)
checkpoint(state)
outline = llm_structured(
prompt="Create an outline from supported claims and preserve uncertainty.",
input=state.evidence_brief(),
schema=ReportOutline,
)
draft = llm_structured(
prompt="Write the report using only supported claims and notes.",
input={"outline": outline, "claims": state.supported_claims()},
schema=DraftReport,
)
verified = verify_citations(draft, state)
draft = repair_or_qualify_unsupported_claims(draft, verified)
return finalize_report(draft, verified)
Advanced supervisor / worker sketch
def advanced_deep_research(task: str, budget: Budget) -> Report:
state = ResearchState.new(task, budget)
brief = planner.create_brief(task)
state.set_brief(brief)
subquestions = planner.decompose(task)
state.add_questions(subquestions)
checkpoint(state)
breadth_jobs = [
WorkerJob(
question=q,
instructions="Map the source landscape. Prefer primary sources. Return evidence packets only.",
max_searches=5,
max_extracts=8,
)
for q in select_parallelizable_questions(state)
]
packets = run_workers_parallel(breadth_jobs, max_workers=6, timeout_s=180)
for packet in packets:
validate_packet(packet)
state.merge(packet)
dedupe_sources(state)
rank_sources(state)
gaps = analyst.find_gaps(state)
contradictions = analyst.find_contradictions(state)
deep_jobs = [
WorkerJob(
question=t.question,
instructions="Resolve this gap or contradiction with higher-quality sources and exact quotes/spans.",
max_searches=4,
max_extracts=6,
)
for t in prioritize(gaps + contradictions)
]
deep_packets = run_workers_parallel(deep_jobs, max_workers=4, timeout_s=240)
for packet in deep_packets:
validate_packet(packet)
state.merge(packet)
dedupe_sources(state)
rank_sources(state)
update_claim_statuses(state)
if not evidence_gates_pass(state) and state.budget.can_continue():
return continue_research(state)
outline = synthesis_agent.make_outline(state)
draft = synthesis_agent.write_report(outline, state.claims, state.notes)
citation_result = citation_agent.verify_and_attach(draft, state.sources, state.notes)
draft = repair_draft(draft, citation_result)
save_replay_trace(state)
return final_editor.polish(draft)
Eval and testing
Measure at least these:
Task success
- did it answer the actual question?
- did it satisfy required sections and constraints?
- did it make a usable recommendation?
Factual grounding
- are material claims source-supported?
- are unsupported inferences labeled?
Citation correctness
- does the citation point to the right source?
- does the source actually support the claim?
- is the claim stronger than the evidence?
Coverage
- did it answer the major subquestions?
- did it include important source classes?
- did it represent competing views?
Contradiction handling
- were they resolved, scoped, or preserved as uncertainty?
Robustness / efficiency
- long runs without context collapse
- cost per useful source or accepted citation
Regression suite
Build frozen fixtures for:
- primary-source-required task
- ambiguous/no-clean-answer task
For each fixture, store:
- expected must-have sources
- expected forbidden/low-quality sources when relevant
- golden outline rather than exact golden prose
Pitfalls
- Premature synthesis — the agent decides it is done after a handful of searches.
- Context hoarding — stuffing pages into the prompt instead of extracting notes.
- Citation laundering — attaching thematically related sources that do not support the actual sentence.
- Search monoculture — overreliance on one engine or one family of sources.
- Browser overuse — paying high latency/cost for pages an extractor could handle.
- Worker chaos — parallel agents duplicate work or return incompatible shapes.
- No source ranking — weak blogs count the same as docs, papers, or primary reports.
- No contradiction pass — output presents one side as consensus.
- No replay — impossible to debug why the run missed obvious sources.
- Vector DB misuse — embeddings treated as source-of-truth memory without provenance.
- Stale technical docs — outdated implementation details get recycled as current truth.
- Missing failure policy — extraction failures silently become research omissions.
- Cost runaway — workers recurse without marginal-value checks.
Short implementation roadmap
Phase 1 — MVP
- define typed schemas for Action, Observation, Source, Note, Claim, Citation, ResearchState
- implement search, extract, and browser fallback tools
- implement a single-agent FSM with explicit phases
- add SQLite or Postgres checkpoints
- add URL canonicalization and dedupe
- add final citation verification
- build 10 frozen regression fixtures
Phase 2 — reliable deep research
- add contradiction detection
- add breadth-first / depth-first policy
- add semantic retrieval over notes/docs
- add trace viewer or replay logs
- add critique / repair pass
Phase 3 — advanced multi-agent system
- add worker queue and reducer
- add per-worker isolated context
- add parallel source-class sweeps
- add human review hooks for high-impact runs
- add continuous regression over frozen and live tasks
A strong default build
If I were building this in an OpenClaw-like custom harness, I would start with:
- Python or TypeScript orchestrator
- Pydantic or Zod schemas for all actions and state
- SQLite/Postgres for run state and checkpoints
- blob storage for fetched artifacts
- search API + extractor + browser fallback
- one supervisor LLM using structured outputs
- optional worker pool behind a typed
spawn_research_worker tool
- a dedicated citation verifier
- LangGraph-like checkpoint/replay semantics whether or not you use LangGraph
- OpenAI Agents SDK-like tracing and handoffs whether or not you use that SDK
- STORM-like separation between research/prewriting and final prose
- Anthropic-like separation between research agents and citation verification