langchain-ai/langchain · error · ValueError

Source IDs are required when cleanup mode is incremental or

Error message

Source IDs are required when cleanup mode is incremental or scoped_full. Document that starts with content: {hashed_doc.page_content[:100]} was not assigned as source id.

What it means

Raised inside `index()` while processing a batch when cleanup is 'incremental' or 'scoped_full' and one or more documents resolve to a None source ID. The `source_id_key` lookup (metadata key or callable) returned None for a document, which would break the bookkeeping that deletes stale entries per source. The message includes the first 100 characters of the offending document's content to help identify it.

Source

Thrown at libs/core/langchain_core/indexing/api.py:500

        # Count documents removed by within-batch deduplication
        num_skipped += original_batch_size - len(hashed_docs)

        source_ids: Sequence[str | None] = [
            source_id_assigner(hashed_doc) for hashed_doc in hashed_docs
        ]

        if cleanup in {"incremental", "scoped_full"}:
            # Source IDs are required.
            for source_id, hashed_doc in zip(source_ids, hashed_docs, strict=False):
                if source_id is None:
                    msg = (
                        f"Source IDs are required when cleanup mode is "
                        f"incremental or scoped_full. "
                        f"Document that starts with "
                        f"content: {hashed_doc.page_content[:100]} "
                        f"was not assigned as source id."
                    )
                    raise ValueError(msg)
                if cleanup == "scoped_full":
                    scoped_full_cleanup_source_ids.add(source_id)
            # Source IDs cannot be None after for loop above.
            source_ids = cast("Sequence[str]", source_ids)

        exists_batch = record_manager.exists(
            cast("Sequence[str]", [doc.id for doc in hashed_docs])
        )

        # Filter out documents that already exist in the record store.
        uids = []
        docs_to_index = []
        uids_to_refresh = []
        seen_docs: set[str] = set()
        for hashed_doc, doc_exists in zip(hashed_docs, exists_batch, strict=False):
            hashed_id = cast("str", hashed_doc.id)
            if doc_exists:
                if force_update:

View on GitHub (pinned to e32fa9a52e)

Solutions

1. Ensure every document has the source metadata key before indexing; fix the loader to always set it.
2. Make your callable never return None: `lambda doc: doc.metadata.get("source") or doc.metadata["file"]`.
3. Filter out documents lacking a source if they should not participate: `[d for d in docs if d.metadata.get("source")]`.
4. If None-source documents are legitimate, reconsider whether incremental cleanup is the right mode.

Example fix

# before
def src(doc):
    return doc.metadata.get("url")  # None for local files

index(vs, docs, rm, cleanup="incremental", source_id_key=src)

# after
def src(doc):
    return doc.metadata.get("url") or f"local:{doc.metadata['file']}"

index(vs, docs, rm, cleanup="incremental", source_id_key=src)

missing = [d.metadata.get("source") for d in docs].count(None)
if missing:
    raise ValueError(f"{missing} docs lack 'source'; incremental cleanup requires it")
index(vs, docs, rm, cleanup="incremental", source_id_key="source")

def all_docs_have_source(docs, key="source") -> bool:
    return all(d.metadata.get(key) not in (None, "") for d in docs)

try:
    index(vs, docs, rm, cleanup="incremental", source_id_key="source")
except ValueError as e:
    if "Source IDs are required" in str(e):
        docs = [d for d in docs if d.metadata.get("source")]
        index(vs, docs, rm, cleanup="incremental", source_id_key="source")
    else:
        raise

When it happens

Trigger: A document missing the metadata key used as `source_id_key` (e.g. `metadata` has no "source"); a callable `source_id_key` returning None for some documents; inconsistent loaders where only some docs carry the key. Note the metadata-key path uses `doc.metadata[source_id_key]`, so a missing key raises KeyError earlier — None source IDs typically come from a callable returning None, or `source_id_key=None` combined with these cleanup modes (guarded earlier) — in practice, custom callables and mixed-source batches.

Common situations: Merging documents from multiple loaders where one loader omits source metadata; callable extractors with fallback `return None` branches; empty-string vs None source confusion.

Related errors

• Source id key is required when cleanup mode is incremental o
• source_id_key should be either None, a string or a callable.
• The delete operation to VectorStore failed.
• The delete operation to DocumentIndex failed.
• cleanup should be one of 'incremental', 'full', 'scoped_full

AI-assisted analysis of langchain-ai/langchain@e32fa9a52e (2026-08-14). Data as JSON: /api/errors/9f99b35e8bb32cdd. Report an issue: GitHub.