{"record":{"id":"de886a08c65bfe23","repo":"tobi/qmd","slug":"error-rate-too-high-activeerrorcount-pro","errorCode":null,"errorMessage":"⚠ Error rate too high (${activeErrorCount()}/${processed}) — aborting embedding","messagePattern":"⚠ Error rate too high \\((.+?)/(.+?)\\) — aborting embedding","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/store.ts","lineNumber":2146,"sourceCode":"\n      const totalBatchChunkBytes = batchChunks.reduce((sum, chunk) => sum + chunk.bytes, 0);\n      let batchChunkBytesProcessed = 0;\n\n      for (let batchStart = 0; batchStart < batchChunks.length; batchStart += BATCH_SIZE) {\n        // Abort early if session has been invalidated (e.g. max duration exceeded)\n        if (!session.isValid) {\n          const remainingChunks = batchChunks.slice(batchStart);\n          for (const chunk of remainingChunks) recordFailure(chunk, \"LLM session expired before embedding chunk\");\n          console.warn(`⚠ Session expired — skipping ${remainingChunks.length} remaining chunks`);\n          break;\n        }\n\n        // Abort early if active error rate is too high (>80% of attempted chunks failed)\n        const processed = chunksEmbedded + activeErrorCount();\n        if (processed >= BATCH_SIZE && activeErrorCount() > processed * 0.8) {\n          const remainingChunks = batchChunks.slice(batchStart);\n          for (const chunk of remainingChunks) recordFailure(chunk, \"embedding aborted because error rate was too high\");\n          console.warn(`⚠ Error rate too high (${activeErrorCount()}/${processed}) — aborting embedding`);\n          break;\n        }\n\n        const batchEnd = Math.min(batchStart + BATCH_SIZE, batchChunks.length);\n        const chunkBatch = batchChunks.slice(batchStart, batchEnd);\n        const texts = chunkBatch.map(chunk => formatDocForEmbedding(chunk.text, chunk.title, embedModelUri));\n\n        try {\n          const embeddings = await session.embedBatch(texts, { model });\n          for (let i = 0; i < chunkBatch.length; i++) {\n            const chunk = chunkBatch[i]!;\n            const embedding = embeddings[i];\n            if (embedding) {\n              insertEmbedding(db, chunk.hash, chunk.seq, chunk.pos, new Float32Array(embedding.embedding), model, now, chunk.expectedTotalChunks, fingerprint);\n              chunksEmbedded++;\n              successesSinceRetry++;\n              clearFailure(chunk);\n            } else {","sourceCodeStart":2128,"sourceCodeEnd":2164,"githubUrl":"https://github.com/tobi/qmd/blob/dbfd0b4736aeaf761d1a16ca8e424f071df8feb9/src/store.ts#L2128-L2164","documentation":"While embedding chunks into the vector store, the batch loop aborts early when the active error rate exceeds 80% of attempted chunks (after at least BATCH_SIZE processed). All remaining un-attempted chunks are recorded as failures ('embedding aborted because error rate was too high') and the warning '⚠ Error rate too high (x/y) — aborting embedding' is printed. This is a circuit breaker: it prevents burning time/API quota when embeddings are systematically failing.","triggerScenarios":"Running `qmd embed` (or embedding during indexing) when the embedding model (embeddinggemma via node-llama-cpp) fails for most batches — out of VRAM/RAM, model file missing or corrupt, native library crash per call, or context length overflows for every chunk. Once processed >= BATCH_SIZE and activeErrorCount() > 0.8 * processed, the loop records the remaining chunks as failed and breaks.","commonSituations":"Embedding a large collection on a machine with too little VRAM; the embedding model was never fully downloaded; a node-llama-cpp ABI mismatch makes every inference call throw; extremely long chunks that exceed the model's context window across the board.","solutions":["Run `qmd doctor` to diagnose model/device issues, then free memory (close GPU processes) or run on CPU before re-running `qmd embed`","Check the recorded failure reasons for the failed chunks (they include the per-chunk error) to confirm whether it is memory, missing model, or context length","If the model files are missing/corrupt, reinstall model assets (`bun install`, or clear the model cache to force re-download)","Re-run `qmd embed` after fixing the root cause — chunks already embedded are not redone, only the failed ones retry"],"exampleFix":"# before\nqmd embed   # aborts: Error rate too high (81/100)\n\n# after\nqmd doctor                       # confirm device/model issue\n# free VRAM or switch to CPU, then:\nqmd embed","handlingStrategy":"retry","validationCode":"// Pre-flight before qmd embed: cheap model sanity check\nimport { spawnSync } from 'node:child_process';\n\nfunction embeddingModelOk(): boolean {\n  const r = spawnSync('qmd', ['doctor'], { encoding: 'utf8' });\n  return r.status === 0 && !/model|device|vram/i.test(r.stdout.split('\\n').filter(l => /fail|error/i.test(l)).join(''));\n}\nif (!embeddingModelOk()) throw new Error('Fix model/device issues before embedding (qmd doctor)');","typeGuard":"function canEmbed(freeVramMb: number | null): boolean {\n  return freeVramMb === null || freeVramMb >= 1000; // embeddinggemma needs headroom\n}","tryCatchPattern":"// Re-run pattern: failed chunks are recorded; after fixing the root cause,\n// re-running qmd embed retries only failed chunks.\ndo {\n  run('qmd embed');\n} while (lastRunAbortedHighErrorRate() && ++attempt < 2);","preventionTips":["Run qmd doctor before qmd embed on new machines","Free VRAM/RAM or force CPU inference before embedding large collections","Embed in smaller batches (add collections incrementally) to isolate failures","Inspect recorded chunk failure reasons after an abort to distinguish memory vs model-file issues"],"tags":["embedding","circuit-breaker","vram","node-llama-cpp","batch-processing"],"backgroundTag":"embedding-batch-failure-abort","analyzedSha":"dbfd0b4736aeaf761d1a16ca8e424f071df8feb9","analyzedAt":"2026-08-28T18:07:46.628Z","schemaVersion":2},"datasetVersion":"2026-08-28T21:17:43.275Z"}