stablyai/orca · error · Error

benchmark produced an empty TOC; document generation is brok

Error message

benchmark produced an empty TOC; document generation is broken

What it means

Inside the typing-burst loop, each generated document is parsed with buildMarkdownTableOfContentsLike and the resulting TOC must be non-empty. An empty TOC means the synthetic document no longer contains recognizable headings — i.e. the document generator (DOC_HEADINGS / heading format) is broken, and the benchmark would silently measure zero work.

Source

Thrown at config/scripts/markdown-toc-parse-benchmark.mjs:112

  const mid = Math.floor(sorted.length / 2)
  return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid]
}

function measureDoc(headings, parasPerHeading) {
  const baseDoc = buildDocument(headings, parasPerHeading)
  // Simulate typing: each content change appends one character so `content`
  // changes identity every time (matching the memo dependency in the editor).
  const perChange = []
  for (let i = 0; i < WARMUP + CONTENT_CHANGES; i += 1) {
    const doc = `${baseDoc}\nedit-${i}`
    const t0 = performance.now()
    const toc = buildMarkdownTableOfContentsLike(doc)
    const elapsed = performance.now() - t0
    if (i >= WARMUP) {
      perChange.push(elapsed)
    }
    if (toc.length === 0) {
      throw new Error('benchmark produced an empty TOC; document generation is broken')
    }
  }
  const total = perChange.reduce((sum, value) => sum + value, 0)
  return {
    bytes: Buffer.byteLength(baseDoc, 'utf8'),
    headings,
    perChangeMedianMs: median(perChange),
    perChangeMaxMs: Math.max(...perChange),
    burstTotalMs: total
  }
}

const sizes = [
  { headings: Math.round(DOC_HEADINGS / 8), paras: PARAGRAPHS_PER_HEADING },
  { headings: Math.round(DOC_HEADINGS / 2), paras: PARAGRAPHS_PER_HEADING },
  { headings: DOC_HEADINGS, paras: PARAGRAPHS_PER_HEADING }
]

View on GitHub (pinned to 1136503c6a)

Solutions

Print baseDoc to stderr and manually run remark-parse on it to confirm headings are detected
Verify the heading template starts with '# ' (ATX) and is not indented or inside a code fence
Pin remark/remark-gfm/remark-frontmatter versions if a transitive bump changed heading parsing
Cross-check against src/renderer/src/components/editor/markdown-table-of-contents.ts which this mirrors

Defensive patterns

Strategy: validation

Validate before calling

const sample = buildMarkdownTableOfContentsLike(baseDoc)
if (sample.length === 0) {
  console.error('baseDoc heading sample:\n', baseDoc.slice(0, 500))
  throw new Error('document generator produced no headings before running the burst')
}

Prevention

Print baseDoc to stderr before the burst loop so an empty TOC has context
Pin remark/remark-gfm/remark-frontmatter versions in package.json
Keep buildMarkdownTableOfContentsLike in lockstep with markdown-table-of-contents.ts

When it happens

Trigger: buildMarkdownTableOfContentsLike(doc) returns [] for the base doc. Happens if the heading emitter produces lines that remark does not treat as headings (wrong leading '#' count, frontmatter swallowing the first heading, remarkGfm/remarkFrontmatter plugin misconfiguration).

Common situations: Editing buildMarkdownTableOfContentsLike (e.g. bumping remark versions) changes heading detection; the heading template string is changed to something that is not ATX heading syntax; or remarkFrontmatter is configured with a format that eats the first heading.

Related errors

AI-assisted analysis of stablyai/orca@1136503c6a (2026-08-12). Data as JSON: /api/errors/2f5b417ca1261334. Report an issue: GitHub.