{"record":{"id":"df7b9352703277b5","repo":"FlowiseAI/Flowise","slug":"sourceidkey-must-be-provided-when-cleanup-is-incre","errorCode":null,"errorMessage":"sourceIdKey must be provided when cleanup is incremental","messagePattern":"sourceIdKey must be provided when cleanup is incremental","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/src/indexing.ts","lineNumber":292,"sourceCode":"    let numAdded = 0\n    let addedDocs: Document[] = []\n    let numDeleted = 0\n    let numUpdated = 0\n    let numSkipped = 0\n    let totalKeys = 0\n\n    const batches = _batch<DocumentInterface>(batchSize ?? 100, docs)\n\n    for (const batch of batches) {\n        const hashedDocs = _deduplicateInOrder(batch.map((doc) => _HashedDocument.fromDocument(doc)))\n\n        const sourceIds = hashedDocs.map((doc) => sourceIdAssigner(doc))\n\n        if (cleanup === 'incremental') {\n            hashedDocs.forEach((_hashedDoc, index) => {\n                const source = sourceIds[index]\n                if (source === null) {\n                    throw new Error('sourceIdKey must be provided when cleanup is incremental')\n                }\n            })\n        }\n\n        const batchExists = await recordManager.exists(hashedDocs.map((doc) => doc.uid))\n\n        const uids: string[] = []\n        const docsToIndex: DocumentInterface[] = []\n        const docsToUpdate: Array<{ uid: string; docId: string }> = []\n        const seenDocs = new Set<string>()\n        hashedDocs.forEach((hashedDoc, i) => {\n            const docExists = batchExists[i]\n            if (docExists) {\n                if (forceUpdate) {\n                    seenDocs.add(hashedDoc.uid)\n                } else {\n                    docsToUpdate.push({ uid: hashedDoc.uid, docId: hashedDoc.metadata.docId as string })\n                    return","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/src/indexing.ts#L274-L310","documentation":"Thrown by index() during per-batch processing when cleanup is 'incremental' and the sourceIdAssigner returns null for at least one document. This is a deeper guard than error 594: sourceIdKey was provided, but for a specific document the extractor yielded null — meaning the metadata key is missing or the function returned null. Incremental cleanup cannot proceed without a source ID for every document.","triggerScenarios":"sourceIdKey is a string metadata key, but some documents lack that key (doc.metadata[sourceIdKey] is undefined → null via the assigner). Or sourceIdKey is a function that returns null for some inputs. The per-document check at lines 289-294 fires inside the batch loop.","commonSituations":"Heterogeneous document sources where only some documents carry the source field. A metadata key typo (sourceIdKey: 'sorce' vs 'source'). Documents loaded from a source that strips metadata. A sourceIdKey function with an edge case returning null.","solutions":["Ensure every document has a non-null value at the sourceIdKey metadata key before indexing.","If sourceIdKey is a function, guarantee it returns a non-null string for all inputs.","Default missing values to a stable sentinel (e.g. doc.metadata.source ?? 'unknown') in a pre-indexing transform.","Fix the metadata key typo or the loader that drops the field."],"exampleFix":"// before\nconst docs = [\n  { pageContent: 'a', metadata: { source: 'file1' } },\n  { pageContent: 'b', metadata: {} } // missing source\n]\nawait index({ docsSource: docs, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })\n\n// after\nconst docs = rawDocs.map(d => ({ ...d, metadata: { ...d.metadata, source: d.metadata.source ?? 'unknown' } }))\nawait index({ docsSource: docs, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })","handlingStrategy":"validation","validationCode":"// Ensure every document carries a non-null source id before indexing\nfunction ensureSourceIds(docs: DocumentInterface[], key: string): DocumentInterface[] {\n  return docs.map((d) => {\n    if (d.metadata[key] == null) {\n      throw new Error(`Doc missing metadata.${key}: ${d.pageContent.slice(0, 60)}`)\n    }\n    return d\n  })\n}\n\nconst checked = ensureSourceIds(docs, 'source')\nawait index({ docsSource: checked, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })","typeGuard":"function allHaveSourceId(docs: DocumentInterface[], key: string): boolean {\n  return docs.every((d) => d.metadata[key] != null && d.metadata[key] !== '')\n}","tryCatchPattern":"try {\n  await index(args)\n} catch (e) {\n  if (String(e).includes('sourceIdKey must be provided when cleanup is incremental')) {\n    const patched = docs.map((d) => ({ ...d, metadata: { ...d.metadata, source: d.metadata.source ?? 'unknown' } }))\n    await index({ ...args, docsSource: patched })\n  } else throw e\n}","preventionTips":["Validate that every document has the sourceIdKey field before indexing.","Default missing source IDs to a stable sentinel at the loader stage.","Use a sourceIdKey function with a guaranteed non-null return."],"tags":["indexing","validation","langchain","data-quality"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}