lfnovo/open-notebook · error · ValueError

Embedding count mismatch: got {len(embeddings)} embeddings f

Error message

Embedding count mismatch: got {len(embeddings)} embeddings for {len(chunks)} chunks

What it means

ValueError raised by the source embed command when generate_embeddings returns fewer or more embeddings than the number of input chunks. The pipeline requires a 1:1 correspondence so it can bulk-insert source_embedding rows aligned to chunks.

Source

Thrown at commands/embedding_commands.py:369

        chunk_sizes = [len(c) for c in chunks]
        logger.info(
            f"Created {total_chunks} chunks for source {input_data.source_id} "
            f"(sizes: min={min(chunk_sizes) if chunk_sizes else 0}, "
            f"max={max(chunk_sizes) if chunk_sizes else 0}, "
            f"avg={sum(chunk_sizes) // len(chunk_sizes) if chunk_sizes else 0} chars)"
        )

        if total_chunks == 0:
            raise ValueError("No chunks created after splitting text")

        # 5. Generate embeddings for all chunks in batches
        cmd_id = get_command_id(input_data)
        logger.debug(f"Generating embeddings for {total_chunks} chunks")
        embeddings = await generate_embeddings(chunks, command_id=cmd_id)

        # Verify we got embeddings for all chunks
        if len(embeddings) != len(chunks):
            raise ValueError(
                f"Embedding count mismatch: got {len(embeddings)} embeddings "
                f"for {len(chunks)} chunks"
            )

        # 6. Bulk INSERT source_embedding records
        records = [
            {
                "source": ensure_record_id(input_data.source_id),
                "order": idx,
                "content": chunk,
                "embedding": embedding,
            }
            for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
        ]

        logger.debug(f"Inserting {len(records)} source_embedding records")
        await repo_insert("source_embedding", records)

View on GitHub (pinned to a7de90d38a)

Solutions

Check the worker/API logs for the provider response shape; the mismatch counts are in the error message
Retry the embedding job — transient provider batch failures often self-heal
If using a custom embedding model, verify it returns exactly one embedding per input string
Reduce batch size in generate_embeddings to avoid provider truncation

Defensive patterns

Strategy: retry

Try / catch

for attempt in range(3):
    result = await run_command('embed_source', {'source_id': sid})
    if result.output.get('success'):
        break
    if 'mismatch' not in result.output.get('error', ''):
        raise RuntimeError(result.output['error'])
    await asyncio.sleep(2 ** attempt)

Prevention

Use embedding providers that return one vector per input string
Retry on transient batch mismatches before escalating
Keep embedding batch sizes conservative for your provider

When it happens

Trigger: An embedding provider/batch API that drops items, merges requests, or returns an inconsistent array length (e.g. a custom provider returning one pooled embedding instead of per-chunk embeddings, or a batch endpoint truncating results).

Common situations: Swapping in a custom or misbehaving embedding model, provider partial failures under load, or a provider SDK change in batch response shape.

Related errors

AI-assisted analysis of lfnovo/open-notebook@a7de90d38a (2026-08-27). Data as JSON: /api/errors/07c1bb416eb4a9b3. Report an issue: GitHub.