langchain-ai/langchain · error · ValueError

Source IDs are required when cleanup mode is incremental or…

Error message

Source IDs are required when cleanup mode is incremental or scoped_full. Document that starts with content: {hashed_doc.page_content[:100]} was not assigned as source id.

What it means

Raised inside `index()` while processing a batch when cleanup is 'incremental' or 'scoped_full' and one or more documents resolve to a None source ID. The `source_id_key` lookup (metadata key or callable) returned None for a document, which would break the bookkeeping that deletes stale entries per source. The message includes the first 100 characters of the offending document's content to help identify it.

Solutions

Ensure every document has the source metadata key before indexing; fix the loader to always set it.
Make your callable never return None: `lambda doc: doc.metadata.get("source") or doc.metadata["file"]`.
Filter out documents lacking a source if they should not participate: `[d for d in docs if d.metadata.get("source")]`.
If None-source documents are legitimate, reconsider whether incremental cleanup is the right mode.

Example fix

# before
def src(doc):
    return doc.metadata.get("url")  # None for local files

index(vs, docs, rm, cleanup="incremental", source_id_key=src)

# after
def src(doc):
    return doc.metadata.get("url") or f"local:{doc.metadata['file']}"

index(vs, docs, rm, cleanup="incremental", source_id_key=src)

Defensive patterns

Strategy: validation

Validate before calling

missing = [d.metadata.get("source") for d in docs].count(None)
if missing:
    raise ValueError(f"{missing} docs lack 'source'; incremental cleanup requires it")
index(vs, docs, rm, cleanup="incremental", source_id_key="source")

Type guard

def all_docs_have_source(docs, key="source") -> bool:
    return all(d.metadata.get(key) not in (None, "") for d in docs)

Try / catch

try:
    index(vs, docs, rm, cleanup="incremental", source_id_key="source")
except ValueError as e:
    if "Source IDs are required" in str(e):
        docs = [d for d in docs if d.metadata.get("source")]
        index(vs, docs, rm, cleanup="incremental", source_id_key="source")
    else:
        raise

Prevention

Make loaders always set the source metadata key — treat it as a required field.
Ensure custom source_id_key callables never return None (use a fallback like a content hash).
Preflight-scan the batch for None sources before starting a long incremental run.

When it happens

Trigger: A document missing the metadata key used as `source_id_key` (e.g. `metadata` has no "source"); a callable `source_id_key` returning None for some documents; inconsistent loaders where only some docs carry the key. Note the metadata-key path uses `doc.metadata[source_id_key]`, so a missing key raises KeyError earlier — None source IDs typically come from a callable returning None, or `source_id_key=None` combined with these cleanup modes (guarded earlier) — in practice, custom callables and mixed-source batches.

Common situations: Merging documents from multiple loaders where one loader omits source metadata; callable extractors with fallback `return None` branches; empty-string vs None source confusion.

Related errors

AI-assisted analysis of langchain-ai/langchain@e32fa9a52e (2026-08-14). Data as JSON: /api/errors/9f99b35e8bb32cdd. Report an issue: GitHub.

Appendix: source

Thrown at libs/core/langchain_core/indexing/api.py:500

        # Count documents removed by within-batch deduplication
        num_skipped += original_batch_size - len(hashed_docs)

        source_ids: Sequence[str | None] = [
            source_id_assigner(hashed_doc) for hashed_doc in hashed_docs
        ]

        if cleanup in {"incremental", "scoped_full"}:
            # Source IDs are required.
            for source_id, hashed_doc in zip(source_ids, hashed_docs, strict=False):
                if source_id is None:
                    msg = (
                        f"Source IDs are required when cleanup mode is "
                        f"incremental or scoped_full. "
                        f"Document that starts with "
                        f"content: {hashed_doc.page_content[:100]} "
                        f"was not assigned as source id."
                    )
                    raise ValueError(msg)
                if cleanup == "scoped_full":
                    scoped_full_cleanup_source_ids.add(source_id)
            # Source IDs cannot be None after for loop above.
            source_ids = cast("Sequence[str]", source_ids)

        exists_batch = record_manager.exists(
            cast("Sequence[str]", [doc.id for doc in hashed_docs])
        )

        # Filter out documents that already exist in the record store.
        uids = []
        docs_to_index = []
        uids_to_refresh = []
        seen_docs: set[str] = set()
        for hashed_doc, doc_exists in zip(hashed_docs, exists_batch, strict=False):
            hashed_id = cast("str", hashed_doc.id)
            if doc_exists:
                if force_update:

View on GitHub (pinned to e32fa9a52e)