FlowiseAI/Flowise · error · Error

sourceIdKey is required when cleanup mode is incremental. Pl

Error message

sourceIdKey is required when cleanup mode is incremental. Please provide through 'options.sourceIdKey'.

What it means

Thrown by index() at the top of the function when cleanup is set to 'incremental' but no sourceIdKey option is provided. Incremental cleanup tracks documents by source ID so it can delete stale entries from the same source; without a sourceIdKey the indexer cannot map documents to sources, so it fails fast before doing any work.

Source

Thrown at packages/components/src/indexing.ts:262

 * This allows us to keep track of which documents were updated, and which
 * documents were deleted, which documents should be skipped.
 *
 * For the time being, documents are indexed using their hashes, and users
 *  are not able to specify the uid of the document.
 *
 * @param {IndexArgs} args
 * @param {BaseDocumentLoader | DocumentInterface[]} args.docsSource The source of documents to index. Can be a DocumentLoader or a list of Documents.
 * @param {RecordManagerInterface} args.recordManager The record manager to use for keeping track of indexed documents.
 * @param {VectorStore} args.vectorStore The vector store to use for storing the documents.
 * @param {IndexOptions | undefined} args.options Options for indexing.
 * @returns {Promise<IndexingResult>}
 */
export async function index(args: IndexArgs): Promise<IndexingResult> {
    const { docsSource, recordManager, vectorStore, options } = args
    const { batchSize = 100, cleanup, sourceIdKey, cleanupBatchSize = 1000, forceUpdate = false, vectorStoreName } = options ?? {}

    if (cleanup === 'incremental' && !sourceIdKey) {
        throw new Error("sourceIdKey is required when cleanup mode is incremental. Please provide through 'options.sourceIdKey'.")
    }

    if (vectorStoreName) {
        ;(recordManager as any).namespace = (recordManager as any).namespace + '_' + vectorStoreName
    }

    const docs = _isBaseDocumentLoader(docsSource) ? await docsSource.load() : docsSource

    const sourceIdAssigner = _getSourceIdAssigner(sourceIdKey ?? null)

    const indexStartDt = await recordManager.getTime()
    let numAdded = 0
    let addedDocs: Document[] = []
    let numDeleted = 0
    let numUpdated = 0
    let numSkipped = 0
    let totalKeys = 0

View on GitHub (pinned to abe4a8601a)

Solutions

1. Provide options.sourceIdKey as a metadata key string or a function returning the source ID.
2. If you do not have a stable source ID, switch to cleanup: 'full' instead (deletes all documents not seen in this run).
3. Ensure every document in the source has a non-null value at the chosen metadata key.

Example fix

// before
await index({ docsSource, recordManager, vectorStore, options: { cleanup: 'incremental' } })

// after
await index({
  docsSource, recordManager, vectorStore,
  options: { cleanup: 'incremental', sourceIdKey: 'source' }
})

function validateIndexOptions(opts?: IndexOptions) {
  if (opts?.cleanup === 'incremental' && !opts?.sourceIdKey) {
    throw new Error('cleanup=incremental requires options.sourceIdKey')
  }
}

validateIndexOptions(options)
await index({ docsSource, recordManager, vectorStore, options })

function hasSourceIdKeyForIncremental(opts?: IndexOptions): boolean {
  if (opts?.cleanup !== 'incremental') return true
  return Boolean(opts?.sourceIdKey)
}

When it happens

Trigger: Calling index({ ..., options: { cleanup: 'incremental' } }) with no sourceIdKey. The guard at line 261 (cleanup === 'incremental' && !sourceIdKey) fires immediately, before any documents are loaded.

Common situations: Switching cleanup mode from undefined/full to incremental and forgetting to add sourceIdKey. Copying example code that omitted the option. Misunderstanding that incremental cleanup requires a per-document source identifier.

Related errors

• Metadata cannot contain key ${key} as it is reserved for int
• sourceIdKey should be null, a string or a function, got ${ty
• sourceIdKey must be provided when cleanup is incremental
• Source id cannot be null
• Model is required

AI-assisted analysis of FlowiseAI/Flowise@abe4a8601a (2026-08-12). Data as JSON: /api/errors/41ae9c37527b79ce. Report an issue: GitHub.