{"record":{"id":"9f99b35e8bb32cdd","repo":"langchain-ai/langchain","slug":"source-ids-are-required-when-cleanup-mode-is-incre","errorCode":null,"errorMessage":"Source IDs are required when cleanup mode is incremental or scoped_full. Document that starts with content: {hashed_doc.page_content[:100]} was not assigned as source id.","messagePattern":"Source IDs are required when cleanup mode is incremental or scoped_full\\. Document that starts with content: (.+?) was not assigned as source id\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"libs/core/langchain_core/indexing/api.py","lineNumber":500,"sourceCode":"        # Count documents removed by within-batch deduplication\n        num_skipped += original_batch_size - len(hashed_docs)\n\n        source_ids: Sequence[str | None] = [\n            source_id_assigner(hashed_doc) for hashed_doc in hashed_docs\n        ]\n\n        if cleanup in {\"incremental\", \"scoped_full\"}:\n            # Source IDs are required.\n            for source_id, hashed_doc in zip(source_ids, hashed_docs, strict=False):\n                if source_id is None:\n                    msg = (\n                        f\"Source IDs are required when cleanup mode is \"\n                        f\"incremental or scoped_full. \"\n                        f\"Document that starts with \"\n                        f\"content: {hashed_doc.page_content[:100]} \"\n                        f\"was not assigned as source id.\"\n                    )\n                    raise ValueError(msg)\n                if cleanup == \"scoped_full\":\n                    scoped_full_cleanup_source_ids.add(source_id)\n            # Source IDs cannot be None after for loop above.\n            source_ids = cast(\"Sequence[str]\", source_ids)\n\n        exists_batch = record_manager.exists(\n            cast(\"Sequence[str]\", [doc.id for doc in hashed_docs])\n        )\n\n        # Filter out documents that already exist in the record store.\n        uids = []\n        docs_to_index = []\n        uids_to_refresh = []\n        seen_docs: set[str] = set()\n        for hashed_doc, doc_exists in zip(hashed_docs, exists_batch, strict=False):\n            hashed_id = cast(\"str\", hashed_doc.id)\n            if doc_exists:\n                if force_update:","sourceCodeStart":482,"sourceCodeEnd":518,"githubUrl":"https://github.com/langchain-ai/langchain/blob/e32fa9a52eab3b61ad7a45399bfde59b3e580fc4/libs/core/langchain_core/indexing/api.py#L482-L518","documentation":"Raised inside `index()` while processing a batch when cleanup is 'incremental' or 'scoped_full' and one or more documents resolve to a None source ID. The `source_id_key` lookup (metadata key or callable) returned None for a document, which would break the bookkeeping that deletes stale entries per source. The message includes the first 100 characters of the offending document's content to help identify it.","triggerScenarios":"A document missing the metadata key used as `source_id_key` (e.g. `metadata` has no \"source\"); a callable `source_id_key` returning None for some documents; inconsistent loaders where only some docs carry the key. Note the metadata-key path uses `doc.metadata[source_id_key]`, so a missing key raises KeyError earlier — None source IDs typically come from a callable returning None, or `source_id_key=None` combined with these cleanup modes (guarded earlier) — in practice, custom callables and mixed-source batches.","commonSituations":"Merging documents from multiple loaders where one loader omits source metadata; callable extractors with fallback `return None` branches; empty-string vs None source confusion.","solutions":["Ensure every document has the source metadata key before indexing; fix the loader to always set it.","Make your callable never return None: `lambda doc: doc.metadata.get(\"source\") or doc.metadata[\"file\"]`.","Filter out documents lacking a source if they should not participate: `[d for d in docs if d.metadata.get(\"source\")]`.","If None-source documents are legitimate, reconsider whether incremental cleanup is the right mode."],"exampleFix":"# before\ndef src(doc):\n    return doc.metadata.get(\"url\")  # None for local files\n\nindex(vs, docs, rm, cleanup=\"incremental\", source_id_key=src)\n\n# after\ndef src(doc):\n    return doc.metadata.get(\"url\") or f\"local:{doc.metadata['file']}\"\n\nindex(vs, docs, rm, cleanup=\"incremental\", source_id_key=src)","handlingStrategy":"validation","validationCode":"missing = [d.metadata.get(\"source\") for d in docs].count(None)\nif missing:\n    raise ValueError(f\"{missing} docs lack 'source'; incremental cleanup requires it\")\nindex(vs, docs, rm, cleanup=\"incremental\", source_id_key=\"source\")","typeGuard":"def all_docs_have_source(docs, key=\"source\") -> bool:\n    return all(d.metadata.get(key) not in (None, \"\") for d in docs)","tryCatchPattern":"try:\n    index(vs, docs, rm, cleanup=\"incremental\", source_id_key=\"source\")\nexcept ValueError as e:\n    if \"Source IDs are required\" in str(e):\n        docs = [d for d in docs if d.metadata.get(\"source\")]\n        index(vs, docs, rm, cleanup=\"incremental\", source_id_key=\"source\")\n    else:\n        raise","preventionTips":["Make loaders always set the source metadata key — treat it as a required field.","Ensure custom source_id_key callables never return None (use a fallback like a content hash).","Preflight-scan the batch for None sources before starting a long incremental run."],"tags":["indexing","cleanup","source-id","incremental"],"backgroundTag":null,"analyzedSha":"e32fa9a52eab3b61ad7a45399bfde59b3e580fc4","analyzedAt":"2026-08-14T18:42:09.092Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}