MemPalace/mempalace · error · ValueError

documents length {len(documents)} does not match ids length

Error message

documents length {len(documents)} does not match ids length {n}

What it means

Raised by sqlite_exact's `_validate_write_batch` before any SQL runs: the `documents` list has a different length than the `ids` list. The backend requires all parallel arrays in a batch write (upsert/add) to align element-for-element, so a length mismatch is rejected atomically instead of producing partially-paired rows.

Source

Thrown at mempalace/backends/sqlite_exact.py:244

            continue
        if key == "$or":
            if not any(_matches_where_document(document, clause) for clause in value or []):
                return False
            continue
        raise UnsupportedFilterError(f"where_document operator {key!r} not supported")
    return True


def _validate_write_batch(
    *,
    documents: list[str],
    ids: list[str],
    metadatas: Optional[list[dict]],
    embeddings: Optional[list[list[float]]],
) -> None:
    n = len(ids)
    if len(documents) != n:
        raise ValueError(f"documents length {len(documents)} does not match ids length {n}")
    if metadatas is not None and len(metadatas) != n:
        raise ValueError(f"metadatas length {len(metadatas)} does not match ids length {n}")
    if embeddings is not None and len(embeddings) != n:
        raise ValueError(f"embeddings length {len(embeddings)} does not match ids length {n}")


class _SQLiteExactHandle:
    def __init__(
        self,
        conn: sqlite3.Connection,
        lock: threading.RLock,
        palace_path: str,
        *,
        read_only: bool = False,
        immutable: bool = False,
    ):
        self.conn = conn
        self.lock = lock

View on GitHub (pinned to 06cb6987f0)

Solutions

Assert `len(documents) == len(ids)` right before the write call and log both lengths.
Build documents and ids in the same loop iteration so they cannot diverge.
Zip your source records once (`for id, doc in zip(ids, documents)`) and derive both lists from the zipped pairs.
Fix the construction bug: usually a filter/`if` applied to only one of the two lists.

Example fix

# before
ids = [r["id"] for r in records]
documents = [r["text"] for r in records if r["text"].strip()]  # shorter!
col.upsert(ids=ids, documents=documents)

# after
pairs = [(r["id"], r["text"]) for r in records if r["text"].strip()]
col.upsert(ids=[p[0] for p in pairs], documents=[p[1] for p in pairs])

Defensive patterns

Strategy: validation

Validate before calling

def validate_batch(ids, documents, metadatas=None, embeddings=None):
    n = len(ids)
    assert len(documents) == n, f"documents {len(documents)} != ids {n}"
    if metadatas is not None:
        assert len(metadatas) == n
    if embeddings is not None:
        assert len(embeddings) == n
    return True

Try / catch

try:
    col.upsert(ids=ids, documents=documents, embeddings=embeddings)
except ValueError as e:
    if "does not match ids length" in str(e):
        logger.error("batch misaligned: ids=%d docs=%d emb=%d", len(ids), len(documents), len(embeddings or []))
        raise

Prevention

Build ids, documents, metadatas, embeddings in one loop over the same source records.
Add a pre-upsert assert of all list lengths in dev/test builds.
Never filter one list independently of its siblings.

When it happens

Trigger: Calling `upsert(ids=[...], documents=[...], ...)` (or add) where `len(documents) != len(ids)` — e.g. building documents with a comprehension that filters items while ids come from the unfiltered source, or appending to one list inside a conditional branch.

Common situations: Batch construction loops that skip empty documents but keep the id; off-by-one slicing (`ids[1:]` but `documents`); refactoring a single-item write to a batch and forgetting to wrap one argument in a list.

Related errors

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/8a2ce5bfc09c8bd0. Report an issue: GitHub.