{"record":{"id":"28d6e37307f9fba8","repo":"FlowiseAI/Flowise","slug":"source-id-cannot-be-null","errorCode":null,"errorMessage":"Source id cannot be null","messagePattern":"Source id cannot be null","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/components/src/indexing.ts","lineNumber":340,"sourceCode":"        if (docsToIndex.length > 0) {\n            await vectorStore.addDocuments(docsToIndex, { ids: uids })\n            const newDocs = docsToIndex.map((docs) => ({\n                pageContent: docs.pageContent,\n                metadata: docs.metadata\n            }))\n            addedDocs.push(...newDocs)\n            numAdded += docsToIndex.length - seenDocs.size\n            numUpdated += seenDocs.size\n        }\n\n        await recordManager.update(\n            hashedDocs.map((doc) => ({ uid: doc.uid, docId: doc.metadata.docId as string })),\n            { timeAtLeast: indexStartDt, groupIds: sourceIds }\n        )\n\n        if (cleanup === 'incremental') {\n            sourceIds.forEach((sourceId) => {\n                if (!sourceId) throw new Error('Source id cannot be null')\n            })\n            const uidsToDelete = await recordManager.listKeys({\n                before: indexStartDt,\n                groupIds: sourceIds\n            })\n            await vectorStore.delete({ ids: uidsToDelete })\n            await recordManager.deleteKeys(uidsToDelete)\n            numDeleted += uidsToDelete.length\n        }\n    }\n\n    if (cleanup === 'full') {\n        let uidsToDelete = await recordManager.listKeys({\n            before: indexStartDt,\n            limit: cleanupBatchSize\n        })\n        while (uidsToDelete.length > 0) {\n            await vectorStore.delete({ ids: uidsToDelete })","sourceCodeStart":322,"sourceCodeEnd":358,"githubUrl":"https://github.com/FlowiseAI/Flowise/blob/abe4a8601a058047b350c260676826e21dd14101/packages/components/src/indexing.ts#L322-L358","documentation":"Thrown by index() during the incremental-cleanup phase, after documents have been added, when iterating sourceIds and encountering a falsy one. This is a final defensive re-check (line 340) before calling recordManager.listKeys with the sourceIds — it ensures no null/undefined leaks into the groupIds query, which would corrupt cleanup. In a correct run, errors 594 and 595 already guarantee non-null sourceIds, so reaching here indicates a logic gap.","triggerScenarios":"cleanup is 'incremental' and at least one sourceId in the batch is falsy (null/undefined/empty string) at the post-add cleanup stage. This should not happen if the earlier guards (lines 261, 289) fired correctly, so observing it suggests the sourceIdAssigner returned a value that was truthy earlier but is now falsy, or that sourceIds was mutated.","commonSituations":"A sourceIdKey function whose return value depends on mutable state. A document whose metadata key was deleted between the line-289 check and line 340. Effectively a secondary safety net for the same condition as error 595.","solutions":["Apply the same fix as error 595: guarantee every document yields a non-null, non-empty source ID.","Audit the sourceIdKey function for non-determinism or external-state dependence.","Add a unit test that runs incremental cleanup end-to-end with the production sourceIdAssigner.","If the error persists, log the offending document's metadata to identify which input produces the null."],"exampleFix":"// before\nconst sourceIdKey = (doc) => doc.metadata.source // returns undefined for some docs\n\n// after\nconst sourceIdKey = (doc) => {\n  const id = doc.metadata.source\n  if (!id) throw new Error(`Document missing source id: ${doc.pageContent.slice(0, 50)}`)\n  return id\n}","handlingStrategy":"validation","validationCode":"// Strong sourceIdKey function that throws early with context\nconst sourceIdKey = (doc: DocumentInterface): string => {\n  const id = doc.metadata['source']\n  if (typeof id !== 'string' || id.length === 0) {\n    throw new Error(`Missing source id for doc: ${doc.pageContent.slice(0, 60)}`)\n  }\n  return id\n}\n\nawait index({ docsSource, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey } })","typeGuard":"function sourceIdsAllValid(ids: (string | null)[]): ids is string[] {\n  return ids.every((id): id is string => typeof id === 'string' && id.length > 0)\n}","tryCatchPattern":"try {\n  await index(args)\n} catch (e) {\n  if (String(e) === 'Source id cannot be null') {\n    throw new Error('Detected null source id post-add; audit sourceIdKey function for determinism')\n  }\n  throw e\n}","preventionTips":["Make sourceIdKey functions deterministic and side-effect-free.","Validate source IDs both before and after indexing in tests.","Treat this defensive check firing as a signal of a non-deterministic extractor."],"tags":["indexing","validation","langchain","defensive-code"],"backgroundTag":null,"analyzedSha":"abe4a8601a058047b350c260676826e21dd14101","analyzedAt":"2026-08-12T16:04:40.823Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}