FlowiseAI/Flowise · error · Error

sourceIdKey must be provided when cleanup is incremental

Error message

sourceIdKey must be provided when cleanup is incremental

What it means

Thrown by index() during per-batch processing when cleanup is 'incremental' and the sourceIdAssigner returns null for at least one document. This is a deeper guard than error 594: sourceIdKey was provided, but for a specific document the extractor yielded null — meaning the metadata key is missing or the function returned null. Incremental cleanup cannot proceed without a source ID for every document.

Source

Thrown at packages/components/src/indexing.ts:292

    let numAdded = 0
    let addedDocs: Document[] = []
    let numDeleted = 0
    let numUpdated = 0
    let numSkipped = 0
    let totalKeys = 0

    const batches = _batch<DocumentInterface>(batchSize ?? 100, docs)

    for (const batch of batches) {
        const hashedDocs = _deduplicateInOrder(batch.map((doc) => _HashedDocument.fromDocument(doc)))

        const sourceIds = hashedDocs.map((doc) => sourceIdAssigner(doc))

        if (cleanup === 'incremental') {
            hashedDocs.forEach((_hashedDoc, index) => {
                const source = sourceIds[index]
                if (source === null) {
                    throw new Error('sourceIdKey must be provided when cleanup is incremental')
                }
            })
        }

        const batchExists = await recordManager.exists(hashedDocs.map((doc) => doc.uid))

        const uids: string[] = []
        const docsToIndex: DocumentInterface[] = []
        const docsToUpdate: Array<{ uid: string; docId: string }> = []
        const seenDocs = new Set<string>()
        hashedDocs.forEach((hashedDoc, i) => {
            const docExists = batchExists[i]
            if (docExists) {
                if (forceUpdate) {
                    seenDocs.add(hashedDoc.uid)
                } else {
                    docsToUpdate.push({ uid: hashedDoc.uid, docId: hashedDoc.metadata.docId as string })
                    return

View on GitHub (pinned to abe4a8601a)

Solutions

1. Ensure every document has a non-null value at the sourceIdKey metadata key before indexing.
2. If sourceIdKey is a function, guarantee it returns a non-null string for all inputs.
3. Default missing values to a stable sentinel (e.g. doc.metadata.source ?? 'unknown') in a pre-indexing transform.
4. Fix the metadata key typo or the loader that drops the field.

Example fix

// before
const docs = [
  { pageContent: 'a', metadata: { source: 'file1' } },
  { pageContent: 'b', metadata: {} } // missing source
]
await index({ docsSource: docs, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })

// after
const docs = rawDocs.map(d => ({ ...d, metadata: { ...d.metadata, source: d.metadata.source ?? 'unknown' } }))
await index({ docsSource: docs, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })

// Ensure every document carries a non-null source id before indexing
function ensureSourceIds(docs: DocumentInterface[], key: string): DocumentInterface[] {
  return docs.map((d) => {
    if (d.metadata[key] == null) {
      throw new Error(`Doc missing metadata.${key}: ${d.pageContent.slice(0, 60)}`)
    }
    return d
  })
}

const checked = ensureSourceIds(docs, 'source')
await index({ docsSource: checked, recordManager, vectorStore, options: { cleanup: 'incremental', sourceIdKey: 'source' } })

function allHaveSourceId(docs: DocumentInterface[], key: string): boolean {
  return docs.every((d) => d.metadata[key] != null && d.metadata[key] !== '')
}

try {
  await index(args)
} catch (e) {
  if (String(e).includes('sourceIdKey must be provided when cleanup is incremental')) {
    const patched = docs.map((d) => ({ ...d, metadata: { ...d.metadata, source: d.metadata.source ?? 'unknown' } }))
    await index({ ...args, docsSource: patched })
  } else throw e
}

When it happens

Trigger: sourceIdKey is a string metadata key, but some documents lack that key (doc.metadata[sourceIdKey] is undefined → null via the assigner). Or sourceIdKey is a function that returns null for some inputs. The per-document check at lines 289-294 fires inside the batch loop.

Common situations: Heterogeneous document sources where only some documents carry the source field. A metadata key typo (sourceIdKey: 'sorce' vs 'source'). Documents loaded from a source that strips metadata. A sourceIdKey function with an edge case returning null.

Related errors

• Metadata cannot contain key ${key} as it is reserved for int
• sourceIdKey should be null, a string or a function, got ${ty
• sourceIdKey is required when cleanup mode is incremental. Pl
• Source id cannot be null
• Received tool input did not match expected schema: ${JSON.st

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/df7b9352703277b5. Report an issue: GitHub.