apache/seatunnel · error · IndexOutOfBoundsException

current cache size = ${currentCache.size()}, larger than ${s

Error message

current cache size = ${currentCache.size()}, larger than ${scanLimit}

What it means

ScanIterator.cacheLoadFails detects that after loading a scan batch the local cache holds more entries than scanLimit (min of requested limit and the configured scan batch size), meaning the iterator would return more rows than allowed. It is an internal invariant check on TiKV scan batching; IndexOutOfBoundsException signals a bug or a misconfigured scan batch size rather than user data problems.

Source

Thrown at seatunnel-connectors-v2/connector-cdc/connector-cdc-tidb/src/main/java/org/tikv/common/iterator/ScanIterator.java:103

            // currentCache is null means no keys found, whereas currentCache is empty means no
            // values
            // found. The difference lies in whether to continue scanning, because chances are that
            // an empty region exists due to deletion, region split, e.t.c.
            // See https://github.com/pingcap/tispark/issues/393 for details
            if (currentCache == null) {
                return true;
            }
            index = 0;
            Key lastKey = Key.EMPTY;
            // Session should be single-threaded itself
            // so that we don't worry about conf change in the middle
            // of a transaction. Otherwise, below code might lose data
            int scanLimit = Math.min(limit, conf.getScanBatchSize());
            if (currentCache.size() < scanLimit) {
                startKey = curRegionEndKey;
                lastKey = Key.toRawKey(curRegionEndKey);
            } else if (currentCache.size() > scanLimit) {
                throw new IndexOutOfBoundsException(
                        "current cache size = "
                                + currentCache.size()
                                + ", larger than "
                                + scanLimit);
            } else {
                // Start new scan from exact next key in current region
                lastKey = Key.toRawKey(currentCache.get(currentCache.size() - 1).getKey());
                startKey = lastKey.next().toByteString();
            }
            // notify last batch if lastKey is greater than or equal to endKey
            // if startKey is empty, it indicates +∞
            if (hasEndKey && lastKey.compareTo(endKey) >= 0 || startKey.isEmpty()) {
                processingLastBatch = true;
                startKey = null;
            }
        } catch (Exception e) {
            throw new TiClientInternalException("Error scanning data from region.", e);
        }

View on GitHub (pinned to cf67b549a7)

Solutions

Increase conf.getScanBatchSize() so it comfortably covers rows returned per region scan
Ensure the iterator is fully consumed/drained before starting a new scan
Check that limit passed to the ScanIterator is not smaller than the region scan response size in an inconsistent way
If reproducible with default settings, report a TiKV client iterator bug with region/key details

Example fix

// before
conf.setScanBatchSize(1); // cache from one region scan exceeds limit
// after
conf.setScanBatchSize(1024); // default-compatible batch size

Defensive patterns

Strategy: validation

Validate before calling

// Ensure scan batch size is sane before building the iterator
int batchSize = conf.getScanBatchSize();
if (batchSize <= 0 || batchSize < limit / 2) {
    throw new IllegalArgumentException("scanBatchSize too small (" + batchSize + ") for limit " + limit);
}

Try / catch

try {
    while (iterator.isValid()) { iterator.next(); }
} catch (IndexOutOfBoundsException e) {
    if (e.getMessage() != null && e.getMessage().contains("current cache size")) {
        LOG.error("Scan batch invariant broken; raise scanBatchSize or recreate iterator");
    }
    throw e;
}

Prevention

Keep conf.getScanBatchSize() at defaults or comfortably above the per-request limit
Fully drain one ScanIterator before creating another
Avoid shrinking limit/batch values below region-scan response sizes

When it happens

Trigger: A region scan returned more rows than min(limit, conf.getScanBatchSize()) — e.g. conf.getScanBatchSize() misconfigured relative to server-side batch limits, or the iterator's limit accounting diverges across region boundaries.

Common situations: Very small conf.getScanBatchSize() settings combined with dense regions; custom TiKV client config in the snapshot/incremental scan path; iterator reuse across multiple regions where the cache is not drained.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/3e749473b92e2a77. Report an issue: GitHub.