⚠ Error rate too high (${activeErrorCount()}/${processed}) —

Error message

⚠ Error rate too high (${activeErrorCount()}/${processed}) — aborting embedding

What it means

While embedding chunks into the vector store, the batch loop aborts early when the active error rate exceeds 80% of attempted chunks (after at least BATCH_SIZE processed). All remaining un-attempted chunks are recorded as failures ('embedding aborted because error rate was too high') and the warning '⚠ Error rate too high (x/y) — aborting embedding' is printed. This is a circuit breaker: it prevents burning time/API quota when embeddings are systematically failing.

Source

Thrown at src/store.ts:2146

      const totalBatchChunkBytes = batchChunks.reduce((sum, chunk) => sum + chunk.bytes, 0);
      let batchChunkBytesProcessed = 0;

      for (let batchStart = 0; batchStart < batchChunks.length; batchStart += BATCH_SIZE) {
        // Abort early if session has been invalidated (e.g. max duration exceeded)
        if (!session.isValid) {
          const remainingChunks = batchChunks.slice(batchStart);
          for (const chunk of remainingChunks) recordFailure(chunk, "LLM session expired before embedding chunk");
          console.warn(`⚠ Session expired — skipping ${remainingChunks.length} remaining chunks`);
          break;
        }

        // Abort early if active error rate is too high (>80% of attempted chunks failed)
        const processed = chunksEmbedded + activeErrorCount();
        if (processed >= BATCH_SIZE && activeErrorCount() > processed * 0.8) {
          const remainingChunks = batchChunks.slice(batchStart);
          for (const chunk of remainingChunks) recordFailure(chunk, "embedding aborted because error rate was too high");
          console.warn(`⚠ Error rate too high (${activeErrorCount()}/${processed}) — aborting embedding`);
          break;
        }

        const batchEnd = Math.min(batchStart + BATCH_SIZE, batchChunks.length);
        const chunkBatch = batchChunks.slice(batchStart, batchEnd);
        const texts = chunkBatch.map(chunk => formatDocForEmbedding(chunk.text, chunk.title, embedModelUri));

        try {
          const embeddings = await session.embedBatch(texts, { model });
          for (let i = 0; i < chunkBatch.length; i++) {
            const chunk = chunkBatch[i]!;
            const embedding = embeddings[i];
            if (embedding) {
              insertEmbedding(db, chunk.hash, chunk.seq, chunk.pos, new Float32Array(embedding.embedding), model, now, chunk.expectedTotalChunks, fingerprint);
              chunksEmbedded++;
              successesSinceRetry++;
              clearFailure(chunk);
            } else {

View on GitHub (pinned to dbfd0b4736)

Solutions

Run `qmd doctor` to diagnose model/device issues, then free memory (close GPU processes) or run on CPU before re-running `qmd embed`
Check the recorded failure reasons for the failed chunks (they include the per-chunk error) to confirm whether it is memory, missing model, or context length
If the model files are missing/corrupt, reinstall model assets (`bun install`, or clear the model cache to force re-download)
Re-run `qmd embed` after fixing the root cause — chunks already embedded are not redone, only the failed ones retry

Example fix

# before
qmd embed   # aborts: Error rate too high (81/100)

# after
qmd doctor                       # confirm device/model issue
# free VRAM or switch to CPU, then:
qmd embed

Defensive patterns

Strategy: retry

Validate before calling

// Pre-flight before qmd embed: cheap model sanity check
import { spawnSync } from 'node:child_process';

function embeddingModelOk(): boolean {
  const r = spawnSync('qmd', ['doctor'], { encoding: 'utf8' });
  return r.status === 0 && !/model|device|vram/i.test(r.stdout.split('\n').filter(l => /fail|error/i.test(l)).join(''));
}
if (!embeddingModelOk()) throw new Error('Fix model/device issues before embedding (qmd doctor)');

Type guard

function canEmbed(freeVramMb: number | null): boolean {
  return freeVramMb === null || freeVramMb >= 1000; // embeddinggemma needs headroom
}

Try / catch

// Re-run pattern: failed chunks are recorded; after fixing the root cause,
// re-running qmd embed retries only failed chunks.
do {
  run('qmd embed');
} while (lastRunAbortedHighErrorRate() && ++attempt < 2);

Prevention

Run qmd doctor before qmd embed on new machines
Free VRAM/RAM or force CPU inference before embedding large collections
Embed in smaller batches (add collections incrementally) to isolate failures
Inspect recorded chunk failure reasons after an abort to distinguish memory vs model-file issues

When it happens

Trigger: Running `qmd embed` (or embedding during indexing) when the embedding model (embeddinggemma via node-llama-cpp) fails for most batches — out of VRAM/RAM, model file missing or corrupt, native library crash per call, or context length overflows for every chunk. Once processed >= BATCH_SIZE and activeErrorCount() > 0.8 * processed, the loop records the remaining chunks as failed and breaks.

Common situations: Embedding a large collection on a machine with too little VRAM; the embedding model was never fully downloaded; a node-llama-cpp ABI mismatch makes every inference call throw; extremely long chunks that exceed the model's context window across the board.

Related errors

AI-assisted analysis of tobi/qmd@dbfd0b4736 (2026-08-28). Data as JSON: /api/errors/de886a08c65bfe23. Report an issue: GitHub.