Stage 0 — context assembly

How the four layers compose

The four layers of stage 0 are not alternatives to each other. They answer different questions, they disagree usefully, and the packer treats agreement between them as evidence.

Layer 1 asks whether a sequence of characters appears anywhere. It is nearly free, it never misses anything literally present, and it cannot tell code from a comment or from a string that happens to contain the same words.

Layer 2 asks whether a shape of code appears — a function declaration, a call to a particular method — by matching the parsed structure rather than the text. It is immune to formatting, line breaks and comments. It is not immune to naming: two spellings of the same thing are two different queries to it.

Layer 3 asks what a name refers to. It builds an index by analysing the project the way a compiler would, following imports out into installed dependencies. It is the only layer that can state that two differently named variables are the same object, and the only one that can answer the question the plan validator asks — whether a symbol named in a plan actually exists.

Layer 4 does not search at all. It decides what gets sent: ranking hits from strongest evidence to weakest, removing duplicates, widening each survivor out to its enclosing function so it reads in isolation, and filling a fixed budget in rank order. A location found independently by all three search layers ranks highest, and in practice that is usually the definition you were looking for.

The layers get progressively more expensive and progressively more certain, which is the order you want: the cheap ones cast a wide net and the expensive one adjudicates.

Layer 1 — lexical

Layer 1 is the cheapest and widest net in context assembly. It answers exactly one question, with ground truth: does this pattern occur, and where? A string either occurs at a byte offset or it does not. No model is involved and none is needed.

It runs ripgrep and parses the JSON Lines stream rather than the human output. The human output is for humans: it elides, colourises and reflows. The JSON stream is a documented interface carrying byte offsets and submatch spans.

Every flag, and why

What the records show

The last 25 L1 calls. Patterns are shown as a hash, not as text, for the same reason nothing else here carries content.

The corpus column is what makes the result hash mean anything: two runs are only comparable when both searched the same code.

Layer 2 — structural

Layer 1 matches characters. Layer 2 matches parse trees. The difference is not academic: searching one repository on this host for the text 'export function' returned nothing from the project's own source, because that codebase is CommonJS and writes 'function' — while returning 1573 hits from its vendored dependencies. A structural query asks for a function declaration and gets one, regardless of formatting, line breaks, or whether the text happens to sit inside a comment.

Layer 2 runs ast-grep in two modes. A pattern answers 'where is this code shape', and a kind answers 'where is every function' — which is what the packing layer needs in order to expand a match out to its enclosing definition.

It is also where the argument for Layer 3 becomes measurable rather than theoretical. In the sibling harness, the pattern telemetry.record(...) matches 29 call sites. The pattern tel.record(...) matches zero — and so does self.record(...) — even though all of them are the same method on the same object, bound to a different name at the call site. Layer 2 cannot know that. Resolving a name to the thing it refers to is symbol resolution, and that is Layer 3.

What the records show

Layer 3 — symbols

Layer 1 matches characters. Layer 2 matches parse trees. Layer 3 knows what a name refers to.

The gap it closes is the one Layer 2 leaves open, and it is measurable. A structural search for telemetry.record(...) finds 29 call sites and a search for tel.record(...) finds none — same method, same object, a different local name at the call site. Indexing this project instead produces one symbol with one definition and fourteen references, spanning both spellings. The contract validator planned for stage 1 stands on exactly this: 'every symbol the plan names must resolve' is not a question the first two layers can answer.

Layer 3 is different in kind. The first two read the code and answer in milliseconds. This one analyses it: it is slow, it needs the project's dependencies installed because resolution follows imports out into them, it can fail on code that does not typecheck — which is the state code is in during a repair loop — and above all it produces a snapshot rather than an answer.

What the records show

Layer 4 — the packer

The first three layers find things. The packer decides what actually gets sent, and it is the only part of stage 0 whose output another stage consumes directly.

It takes the hits from all three layers, ranks them, removes duplicates, widens each one out to the code that makes it comprehensible, and fills a hard budget. A location found by several layers keeps the best rank and records every layer that contributed, because agreement between independent methods is itself evidence — the highest-ranked chunk in a typical run is one that all three layers found.

Ranking runs from strongest evidence to weakest: a resolved definition, then a resolved reference, then a structural match, then a textual one. The budget is filled in that order rather than by fitting the most chunks, because packing small chunks first would maximise the count while burying the thing you actually needed. Under a budget tight enough for two chunks, the definition is still the first one in.

The budget is measured in bytes, not tokens. Counting tokens exactly requires asking the model that will read them, which would make the one stage that is supposed to be free and offline cost money and depend on a network. Bytes are approximate — roughly three and a half to four per token for source code — and saying so is honest. Exact counting belongs at the point where the bundle actually reaches a model.

What the records show

A red fill means the bundle was truncated — reported, never silent. A bundle that quietly dropped what did not fit is indistinguishable from one that covered everything.

A note on determinism

Three search tools have been added to this pipeline. All three emit their results in an order that varies between runs, and none of that is documented anywhere prominent.

The mechanism is the same in each case: they search files concurrently and report matches in whatever order the work finishes. The results are correct and complete every time. Only the ordering moves. For most uses that is entirely harmless, which is exactly why it is not advertised.

It is not harmless here. This pipeline hashes its own results so that two runs over the same code can be compared, which is the only practical way to notice that something has broken without inspecting everything by hand. A hash over a shuffled list is a different hash. Left alone, the alarm built to detect real breakage would have fired continuously on nothing at all.

One of the three tools has a flag that fixes it. The other two do not, so the ordering is imposed in this codebase instead, after results come back. That turned out to be the better pattern in general: a guarantee that depends on someone else's tool keeping a promise is a guarantee their next release can remove.

Across the layers: 22 distinct queries, 73 records carrying a corpus fingerprint, 0 queries whose result changed while the corpus did not.