{"record":{"id":"07c1bb416eb4a9b3","repo":"lfnovo/open-notebook","slug":"embedding-count-mismatch-got-len-embeddings-em","errorCode":null,"errorMessage":"Embedding count mismatch: got {len(embeddings)} embeddings for {len(chunks)} chunks","messagePattern":"Embedding count mismatch: got (.+?) embeddings for (.+?) chunks","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"commands/embedding_commands.py","lineNumber":369,"sourceCode":"        chunk_sizes = [len(c) for c in chunks]\n        logger.info(\n            f\"Created {total_chunks} chunks for source {input_data.source_id} \"\n            f\"(sizes: min={min(chunk_sizes) if chunk_sizes else 0}, \"\n            f\"max={max(chunk_sizes) if chunk_sizes else 0}, \"\n            f\"avg={sum(chunk_sizes) // len(chunk_sizes) if chunk_sizes else 0} chars)\"\n        )\n\n        if total_chunks == 0:\n            raise ValueError(\"No chunks created after splitting text\")\n\n        # 5. Generate embeddings for all chunks in batches\n        cmd_id = get_command_id(input_data)\n        logger.debug(f\"Generating embeddings for {total_chunks} chunks\")\n        embeddings = await generate_embeddings(chunks, command_id=cmd_id)\n\n        # Verify we got embeddings for all chunks\n        if len(embeddings) != len(chunks):\n            raise ValueError(\n                f\"Embedding count mismatch: got {len(embeddings)} embeddings \"\n                f\"for {len(chunks)} chunks\"\n            )\n\n        # 6. Bulk INSERT source_embedding records\n        records = [\n            {\n                \"source\": ensure_record_id(input_data.source_id),\n                \"order\": idx,\n                \"content\": chunk,\n                \"embedding\": embedding,\n            }\n            for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))\n        ]\n\n        logger.debug(f\"Inserting {len(records)} source_embedding records\")\n        await repo_insert(\"source_embedding\", records)\n","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/lfnovo/open-notebook/blob/a7de90d38aaf18ee85fd661854d35c11e44613e2/commands/embedding_commands.py#L351-L387","documentation":"ValueError raised by the source embed command when generate_embeddings returns fewer or more embeddings than the number of input chunks. The pipeline requires a 1:1 correspondence so it can bulk-insert source_embedding rows aligned to chunks.","triggerScenarios":"An embedding provider/batch API that drops items, merges requests, or returns an inconsistent array length (e.g. a custom provider returning one pooled embedding instead of per-chunk embeddings, or a batch endpoint truncating results).","commonSituations":"Swapping in a custom or misbehaving embedding model, provider partial failures under load, or a provider SDK change in batch response shape.","solutions":["Check the worker/API logs for the provider response shape; the mismatch counts are in the error message","Retry the embedding job — transient provider batch failures often self-heal","If using a custom embedding model, verify it returns exactly one embedding per input string","Reduce batch size in generate_embeddings to avoid provider truncation"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"for attempt in range(3):\n    result = await run_command('embed_source', {'source_id': sid})\n    if result.output.get('success'):\n        break\n    if 'mismatch' not in result.output.get('error', ''):\n        raise RuntimeError(result.output['error'])\n    await asyncio.sleep(2 ** attempt)","preventionTips":["Use embedding providers that return one vector per input string","Retry on transient batch mismatches before escalating","Keep embedding batch sizes conservative for your provider"],"tags":["python","valueerror","embedding","batch-mismatch","provider"],"backgroundTag":"embedding-batch-count-mismatch","analyzedSha":"a7de90d38aaf18ee85fd661854d35c11e44613e2","analyzedAt":"2026-08-27T02:39:58.166Z","schemaVersion":2},"datasetVersion":"2026-08-27T03:17:27.898Z"}