Blog · 2026-08-01
How the ErrLookup scanner works
Every record on this site started as a line of library source code. Here is the pipeline that turns a repository into documented, searchable, citable error knowledge — and the design decisions that keep it cheap, honest, and repeatable.
Step 1: deterministic candidate extraction (zero tokens)
The scanner does not ask a language model to "find the errors." It first runs a structural
grep over a pinned commit of the repository — throw new, raise,
panic!, errors.New, thiserror's #[error(...)]
attributes, HTTP error responses, error-code constants — across nine language families.
Test files, mocks, and generated code are excluded. The output is a list of concrete
candidate sites: file, line, matched code, extracted string literal.
This pass costs nothing and it changes what we ask the model to do. Language models are mediocre at exhaustively searching a large codebase but very good at judging a concrete site with its surrounding context. Feeding 80 candidates per message also flips the economics: one dense request replaces dozens of exploratory agent turns.
Step 2: batched analysis phases
Each repository then flows through LLM phases, ten to eighty items per call:
| Discovery | Classify each candidate: user-facing or internal? Exact message with template placeholders preserved, error type, code, class, HTTP status. |
| Enrichment | What the error means, what triggers it, real-world situations, ordered solutions, an example fix. |
| Defense | How a consumer avoids it: validation snippets, type guards, try/catch patterns, prevention tips — in the library's language. |
| Verify | A separate pass reviews the assembled records for gaps and emits patches; records re-validate against the schema after every patch. |
Phases are checkpointed per repository and per commit SHA. A crashed run resumes at the first incomplete phase with zero re-spend — phase outputs are persisted, so even a failed database write can be reassembled without calling a model again.
Step 3: model routing
Nothing in the pipeline assumes a specific model. Providers are pluggable subprocesses — a CLI in print mode, or any agent speaking ACP (the Agent Client Protocol) — and each phase can route to a different one. We benchmarked frontier and low-cost models on the same repository and found extraction quality at parity, so bulk phases run on an inexpensive model while the verify pass gets a stronger one. The result of that comparison ships in the repository as a markdown report.
Step 4: honest assembly and atomic publishing
Deterministic code — never the model — computes stable record IDs, URL slugs, message regex patterns (with ReDoS-safety fallbacks), and extracts the throwing source region with a GitHub permalink pinned to the analyzed SHA. Every record passes schema validation on write; failures are logged as rejects, never silently dropped.
Exports are atomic: the dataset is written to a temp directory, every file re-validated, then renamed over the previous version with a monotonically increasing dataset version. Consumers — this site, the MCP server, answer engines reading llms.txt — never observe a torn dataset.
What keeps it trustworthy
Every page carries the analyzed commit SHA and a link to the exact source line. Records are clearly labeled as AI-assisted analysis. When the scanner is wrong, file an issue — the fix lands in the next dataset version, and the version history makes the change auditable.