FlowiseAI/Flowise · error · Error

Source id cannot be null

Error message

Source id cannot be null

What it means

Thrown by index() during the incremental-cleanup phase, after documents have been added, when iterating sourceIds and encountering a falsy one. This is a final defensive re-check (line 340) before calling recordManager.listKeys with the sourceIds — it ensures no null/undefined leaks into the groupIds query, which would corrupt cleanup. In a correct run, errors 594 and 595 already guarantee non-null sourceIds, so reaching here indicates a logic gap.

Source

Thrown at packages/components/src/indexing.ts:340

        if (docsToIndex.length > 0) {
            await vectorStore.addDocuments(docsToIndex, { ids: uids })
            const newDocs = docsToIndex.map((docs) => ({
                pageContent: docs.pageContent,
                metadata: docs.metadata
            }))
            addedDocs.push(...newDocs)
            numAdded += docsToIndex.length - seenDocs.size
            numUpdated += seenDocs.size
        }

        await recordManager.update(
            hashedDocs.map((doc) => ({ uid: doc.uid, docId: doc.metadata.docId as string })),
            { timeAtLeast: indexStartDt, groupIds: sourceIds }
        )

        if (cleanup === 'incremental') {
            sourceIds.forEach((sourceId) => {
                if (!sourceId) throw new Error('Source id cannot be null')
            })
            const uidsToDelete = await recordManager.listKeys({
                before: indexStartDt,
                groupIds: sourceIds
            })
            await vectorStore.delete({ ids: uidsToDelete })
            await recordManager.deleteKeys(uidsToDelete)
            numDeleted += uidsToDelete.length
        }
    }

    if (cleanup === 'full') {
        let uidsToDelete = await recordManager.listKeys({
            before: indexStartDt,
            limit: cleanupBatchSize
        })
        while (uidsToDelete.length > 0) {
            await vectorStore.delete({ ids: uidsToDelete })

View on GitHub (pinned to abe4a8601a)

Solutions

1. Apply the same fix as error 595: guarantee every document yields a non-null, non-empty source ID.
2. Audit the sourceIdKey function for non-determinism or external-state dependence.
3. Add a unit test that runs incremental cleanup end-to-end with the production sourceIdAssigner.
4. If the error persists, log the offending document's metadata to identify which input produces the null.

Example fix

// before
const sourceIdKey = (doc) => doc.metadata.source // returns undefined for some docs

// after
const sourceIdKey = (doc) => {
  const id = doc.metadata.source
  if (!id) throw new Error(`Document missing source id: ${doc.pageContent.slice(0, 50)}`)
  return id
}

// Strong sourceIdKey function that throws early with context
const sourceIdKey = (doc: DocumentInterface): string => {
  const id = doc.metadata['source']
  if (typeof id !== 'string' || id.length === 0) {
    throw new Error(`Missing source id for doc: ${doc.pageContent.slice(0, 60)}`)
  }
  return id
}

await index({ docsSource, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey } })

function sourceIdsAllValid(ids: (string | null)[]): ids is string[] {
  return ids.every((id): id is string => typeof id === 'string' && id.length > 0)
}

try {
  await index(args)
} catch (e) {
  if (String(e) === 'Source id cannot be null') {
    throw new Error('Detected null source id post-add; audit sourceIdKey function for determinism')
  }
  throw e
}

When it happens

Trigger: cleanup is 'incremental' and at least one sourceId in the batch is falsy (null/undefined/empty string) at the post-add cleanup stage. This should not happen if the earlier guards (lines 261, 289) fired correctly, so observing it suggests the sourceIdAssigner returned a value that was truthy earlier but is now falsy, or that sourceIds was mutated.

Common situations: A sourceIdKey function whose return value depends on mutable state. A document whose metadata key was deleted between the line-289 check and line 340. Effectively a secondary safety net for the same condition as error 595.

Related errors

• Metadata cannot contain key ${key} as it is reserved for int
• sourceIdKey should be null, a string or a function, got ${ty
• sourceIdKey is required when cleanup mode is incremental. Pl
• sourceIdKey must be provided when cleanup is incremental
• Received tool input did not match expected schema: ${JSON.st

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/28d6e37307f9fba8. Report an issue: GitHub.