tobi/qmd · warning

[qmd] AST parse failed for ${filepath}, falling back to rege

Error message

[qmd] AST parse failed for ${filepath}, falling back to regex: ${err instanceof Error ? err.message : err}

What it means

getASTBreakPoints parses a file with tree-sitter to find function/class boundaries for chunking; if parsing throws, it warns '[qmd] AST parse failed for {filepath}, falling back to regex' and returns an empty breakpoint list. The caller then chunks the file with the regex strategy, so the file is still indexed — only the AST-aware boundaries are lost for that one file.

Source

Thrown at src/ast.ts:320

    const seen = new Map<number, BreakPoint>();

    for (const cap of captures) {
      const pos = cap.node.startIndex;
      const score = SCORE_MAP[cap.name] ?? 20;
      const type = `ast:${cap.name}`;

      const existing = seen.get(pos);
      if (!existing || score > existing.score) {
        seen.set(pos, { pos, score, type });
      }
    }

    tree.delete();
    parser.delete();

    return Array.from(seen.values()).sort((a, b) => a.pos - b.pos);
  } catch (err) {
    console.warn(`[qmd] AST parse failed for ${filepath}, falling back to regex: ${err instanceof Error ? err.message : err}`);
    return [];
  }
}

// =============================================================================
// Health / Status
// =============================================================================

/**
 * Check which tree-sitter grammars are available.
 * Returns a status object for each supported language.
 */
export async function getASTStatus(): Promise<{
  available: boolean;
  languages: { language: SupportedLanguage; available: boolean; error?: string }[];
}> {
  const languages: { language: SupportedLanguage; available: boolean; error?: string }[] = [];

View on GitHub (pinned to dbfd0b4736)

Solutions

1. Treat it as a soft warning first: the file still gets indexed via regex chunking; only act if search quality for that file matters
2. Check the file for encoding problems: `file -i <path>`, remove BOM/NUL bytes, and confirm it is valid UTF-8
3. If the file is minified/generated, exclude it from AST chunking (use the regex strategy for that path or exclude it from the collection)
4. If many files fail after a dependency upgrade, refresh/pin the tree-sitter grammar package (see the related grammar-load warning)

Example fix

# before
qmd collection add . --name src --chunk-strategy auto  # warns on gen/file.ts

# after: keep generated code out of AST chunking
qmd collection add . --name src --chunk-strategy auto --mask 'src/**/*.ts,!src/gen/**'

import { readFileSync } from 'node:fs';

function isSafeForAstParse(filepath: string): boolean {
  const buf = readFileSync(filepath);
  if (buf.includes(0)) return false;                 // NUL bytes
  if (buf.length > 2 * 1024 * 1024) return false;    // oversized/generated files
  const s = buf.toString('utf8');
  if (s.includes('\uFFFD')) return false;            // invalid UTF-8
  return true;
}

function looksParseableSource(text: string): boolean {
  return !text.includes('\u0000') && text.length < 1_000_000 && !text.includes('\uFFFD');
}

When it happens

Trigger: Indexing a .ts/.js/.py/.go/.rs file with AST chunking where tree-sitter's parser throws on the content: pathological or deeply nested code, files with unusual encodings/BOMs or invalid bytes, extremely large files that trip parser limits, or a stale grammar producing internal errors for that syntax.

Common situations: Minified or machine-generated files with extremely long lines; files with mixed encodings or stray NUL bytes after a bad merge; a corrupted source file; grammar edge cases after a tree-sitter upgrade. The warning is per-file and benign unless it affects many files.

Related errors

• [qmd] AST grammar unavailable for ${language}: ${message}
• Failed to parse ${configPath}: ${error}
• Reranker unavailable — skipping reranking (${detail}). Use -

AI-assisted analysis of tobi/qmd@dbfd0b4736 (2026-08-28). Data as JSON: /api/errors/7330c9c7eb7fdaa5. Report an issue: GitHub.