What causes "fail to create iterator" in pingcap/tidb?

Retry the IMPORT INTO after the previous one fully terminated (check import jobs; IMPORT INTO cannot run concurrently on the same table).

pingcap/tidb · critical

fail to create iterator

Error message

fail to create iterator

What it means

newDupDetectIter in pkg/ingestor/ingestctrl (the engine behind IMPORT INTO / TiDB Lightning duplicate detection) creates a pebble iterator over the target engine to scan existing keys; if db.NewIter returns an error the constructor panics with 'fail to create iterator'. The panic aborts the import worker instead of continuing with a broken dedup scan.

Source

Thrown at pkg/ingestor/ingestctrl/iterator.go:170

	keyAdapter common.KeyAdapter,
	opts *pebble.IterOptions,
	dupDB *pebble.DB,
	logger log.Logger,
	dupOpt common.DupDetectOpt,
	buf *membuf.Buffer,
) *dupDetectIter {
	newOpts := &pebble.IterOptions{TableFilter: opts.TableFilter}
	if len(opts.LowerBound) > 0 {
		newOpts.LowerBound = keyAdapter.Encode(nil, opts.LowerBound, common.MinRowID)
	}
	if len(opts.UpperBound) > 0 {
		newOpts.UpperBound = keyAdapter.Encode(nil, opts.UpperBound, common.MinRowID)
	}

	detector := common.NewDupDetector(keyAdapter, dupDB.NewBatch(), logger, dupOpt)
	iter, err := db.NewIter(newOpts)
	if err != nil {
		panic("fail to create iterator")
	}
	return &dupDetectIter{
		keyAdapter:  keyAdapter,
		iter:        iter,
		buf:         buf,
		dupDetector: detector,
		logger:      logger,
	}
}

type dupDBIter struct {
	iter       *pebble.Iterator
	keyAdapter common.KeyAdapter
	curKey     []byte
	err        error
}

func (d *dupDBIter) Error() error {

View on GitHub (pinned to d01f9615c1)

Solutions

Retry the IMPORT INTO after the previous one fully terminated (check import jobs; IMPORT INTO cannot run concurrently on the same table).
Clean the import working area: stop the import, clear leftover engine data under tmp paths (or run with a fresh import dir), then rerun.
If duplicate detection is not needed, import with DUPLICATE_DETECTION=OFF (or 'replace'/'ignore' error handling per your statement options) to bypass the dup iterator.
Check disk health and space under the import temp dir; corrupt SSTs there cause NewIter failures.
If it reproduces on a healthy setup, report it at github.com/pingcap/tidb with the stack.

Example fix

-- before
IMPORT INTO t FROM '/data/t.csv'; -- panics when a prior import left a dirty engine

-- after
-- 1) confirm no import is running
SHOW IMPORT JOBS;
-- 2) rerun cleanly; skip dup scan if duplicates are impossible
IMPORT INTO t FROM '/data/t.csv' WITH DUPLICATE_DETECTION=OFF;

Defensive patterns

Strategy: retry

Validate before calling

-- before importing, confirm no other import is active and the engine is clean
SHOW IMPORT JOBS;
-- expect no 'running'/'pending' job for the target table
SHOW TABLE t REGIONS; -- sanity: table is writable, no import lock

Try / catch

-- panics surface as a failed IMPORT job; inspect then retry on clean state
SHOW IMPORT JOBS; SELECT * FROM information_schema.import_jobs WHERE status = 'failed';
-- after cleanup: rerun IMPORT INTO t FROM '/data/t.csv';

Prevention

Never run two IMPORT INTO / Lightning tasks against the same table concurrently.
After a failed or cancelled import, clean leftover engine/dup dirs before retrying.
Use DUPLICATE_DETECTION=OFF when the source is already deduplicated.
Monitor disk health and capacity under the import temp paths.

When it happens

Trigger: IMPORT INTO / Lightning with duplicate detection (DUP_DETECT) reaching the scan phase while the underlying pebble engine cannot create an iterator: engine already closed/closing (import cancelled concurrently), corrupted local engine files, or invalid encoded bounds produced by KeyAdapter.Encode.

Common situations: Cancelling an IMPORT INTO mid-flight so a shared engine closes while the dup iterator is still being built; disk corruption under the import temp dir; running duplicate detection on an engine left dirty by a previously failed import; racing duplicate-check engines across duplicate jobs sharing disk.

Related errors

AI-assisted analysis of pingcap/tidb@d01f9615c1 (2026-08-15). Data as JSON: /api/errors/c227136fd469efcc. Report an issue: GitHub.