How to tell if the document tree is actually working
Does the structure really steer a coding agent — or is it just the model being capable? An honest look at the evidence, plus copy-paste commands to check your own sessions.
If you've read The Document Tree, here's the question that should bother you: it feels like the docs keep the agent on track — but how would you know? Maybe a capable model would do fine without any of it. The honest worry:
Is the structure actually being loaded and used — or is it just the model being powerful, and the docs are decoration?
You can't run a clean controlled trial on your own project. But you can get real signal, and most of it is one habit away. Here's what I found, and how to check your own.
The evidenceWhat the transcripts showed
I went back through the recorded sessions of one project — a small identity-verification web app I built solo with a coding agent (~6 days, 47 sessions). Coding agents like Claude Code keep a transcript of every session as a local .jsonl log, so every file the agent reads and every line it writes is on the record. Three things stood out.
1. The linked docs get read, not just written. Across five recent sessions the agent opened the devlog 9 times, the root file 6, and pulled in product-decision / deployment / route docs and historical specs on demand. The lazy-loading isn't theoretical — the leaves get opened when a task needs them.
2. They steer the reasoning — provably. The agent repeatedly cited decisions that live only in the docs, not in the code:
- It pulled a product rule recorded only in a doc — "snapshots are deferred to a later phase; for now, link to the live post" — to steer how it built a feature.
- It caught a proposed URL scheme that contradicted the spec, citing the exact spec line numbers.
A model can't invent "spec line 126." To produce that, it had to open the file. That's the cleanest proof: for a decision that exists only in a doc, honoring it means the doc did the work.
3. It pushes back — even on my own prompts. The behavior I value most showed up on its own:
- It declined a search feature I asked for because "that would violate one of your own locked decisions."
- It refused to offer a made-up identifier because it "would violate the anti-squatting invariant the design had settled (§3)."
- Before answering open questions it would say "let me check what's already decided, so my answer doesn't conflict with a locked decision" — and go read the docs first.
HonestyWhat this proves — and what it doesn't
It does kill the "maybe the docs are never even loaded" worry: they demonstrably are, often, and they change what the agent does — including overriding me. It does not prove the counterfactual: I can't isolate how much better the outcome was than a no-docs run, because there's no A/B. The value is clearest for decisions that exist only in the docs (the model couldn't have known them otherwise), and smallest on a tiny project a model can hold in its head. Treat it as a case study, not a benchmark.
Do it yourselfHow to tell on your own project
You don't have to take my word for it — run these on your own transcripts. Your sessions live under ~/.claude/projects/<encoded-cwd>/*.jsonl (the project path with slashes turned into dashes). One caveat: Claude Code prunes transcripts after cleanupPeriodDays (default 30) — raise it in ~/.claude/settings.json if you want a longer window to look back on.
Test 1 — are the docs actually read? Count the agent's reads/edits of doc files:
cat ~/.claude/projects/<your-project>/*.jsonl \
| jq -rc '.message.content[]? | select(.type=="tool_use")
| [.name, (.input.file_path // "")] | @tsv' \
| grep -E 'docs/|CLAUDE\.md|devlog' | sort | uniq -c | sort -rnLots of Reads on your docs = lazy loading is happening. Zero = the leaves are being ignored, and your pointers need to be more compelling.
Test 2 — do the docs steer reasoning? Surface the moments the agent cited a doc or flagged a conflict:
cat ~/.claude/projects/<your-project>/*.jsonl \
| jq -rc 'select(.type=="assistant") | .message.content[]?
| select(.type=="text") | .text' \
| grep -iE 'conflicts? with|contradicts|locked decision|per .*\.md|the spec says'Judge the hits by specificity: a quote naming a concrete rule or a spec line proves the doc was read; a vague "this might clash with your architecture" is the model guessing.
Test 3 — the cold-context test (the strongest). Start a fresh session and ask, without letting it read the code:
"From the docs only — what does this project's decision say about <X>?" — where X is a policy/product choice recorded only in a doc, not derivable from the code.
If it answers correctly, the doc did the work — a no-memory session has no other way to know. (This is the same move as sending in a fresh agent to review your docs, from the main essay.)
And the one habit that makes all of this visible going forward: tell your root file to name the docs it consulted before planning ("per docs/architecture.md…"). Then steering — or its absence — is something you can see in every session, not just guess at.
Bottom line
Is the structure loaded, and does it change what the agent does — including overriding the user? Yes, with receipts. Is it strictly better than a big model with no docs? Unproven, and honestly hard to prove. Run the tests on your own project and decide from your own evidence.
🌳