{"record":{"id":"3e749473b92e2a77","repo":"apache/seatunnel","slug":"current-cache-size-currentcache-size-large","errorCode":null,"errorMessage":"current cache size = ${currentCache.size()}, larger than ${scanLimit}","messagePattern":"current cache size = (.+?), larger than (.+?)","errorType":"exception","errorClass":"IndexOutOfBoundsException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-cdc/connector-cdc-tidb/src/main/java/org/tikv/common/iterator/ScanIterator.java","lineNumber":103,"sourceCode":"            // currentCache is null means no keys found, whereas currentCache is empty means no\n            // values\n            // found. The difference lies in whether to continue scanning, because chances are that\n            // an empty region exists due to deletion, region split, e.t.c.\n            // See https://github.com/pingcap/tispark/issues/393 for details\n            if (currentCache == null) {\n                return true;\n            }\n            index = 0;\n            Key lastKey = Key.EMPTY;\n            // Session should be single-threaded itself\n            // so that we don't worry about conf change in the middle\n            // of a transaction. Otherwise, below code might lose data\n            int scanLimit = Math.min(limit, conf.getScanBatchSize());\n            if (currentCache.size() < scanLimit) {\n                startKey = curRegionEndKey;\n                lastKey = Key.toRawKey(curRegionEndKey);\n            } else if (currentCache.size() > scanLimit) {\n                throw new IndexOutOfBoundsException(\n                        \"current cache size = \"\n                                + currentCache.size()\n                                + \", larger than \"\n                                + scanLimit);\n            } else {\n                // Start new scan from exact next key in current region\n                lastKey = Key.toRawKey(currentCache.get(currentCache.size() - 1).getKey());\n                startKey = lastKey.next().toByteString();\n            }\n            // notify last batch if lastKey is greater than or equal to endKey\n            // if startKey is empty, it indicates +∞\n            if (hasEndKey && lastKey.compareTo(endKey) >= 0 || startKey.isEmpty()) {\n                processingLastBatch = true;\n                startKey = null;\n            }\n        } catch (Exception e) {\n            throw new TiClientInternalException(\"Error scanning data from region.\", e);\n        }","sourceCodeStart":85,"sourceCodeEnd":121,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-cdc/connector-cdc-tidb/src/main/java/org/tikv/common/iterator/ScanIterator.java#L85-L121","documentation":"ScanIterator.cacheLoadFails detects that after loading a scan batch the local cache holds more entries than scanLimit (min of requested limit and the configured scan batch size), meaning the iterator would return more rows than allowed. It is an internal invariant check on TiKV scan batching; IndexOutOfBoundsException signals a bug or a misconfigured scan batch size rather than user data problems.","triggerScenarios":"A region scan returned more rows than min(limit, conf.getScanBatchSize()) — e.g. conf.getScanBatchSize() misconfigured relative to server-side batch limits, or the iterator's limit accounting diverges across region boundaries.","commonSituations":"Very small conf.getScanBatchSize() settings combined with dense regions; custom TiKV client config in the snapshot/incremental scan path; iterator reuse across multiple regions where the cache is not drained.","solutions":["Increase conf.getScanBatchSize() so it comfortably covers rows returned per region scan","Ensure the iterator is fully consumed/drained before starting a new scan","Check that limit passed to the ScanIterator is not smaller than the region scan response size in an inconsistent way","If reproducible with default settings, report a TiKV client iterator bug with region/key details"],"exampleFix":"// before\nconf.setScanBatchSize(1); // cache from one region scan exceeds limit\n// after\nconf.setScanBatchSize(1024); // default-compatible batch size","handlingStrategy":"validation","validationCode":"// Ensure scan batch size is sane before building the iterator\nint batchSize = conf.getScanBatchSize();\nif (batchSize <= 0 || batchSize < limit / 2) {\n    throw new IllegalArgumentException(\"scanBatchSize too small (\" + batchSize + \") for limit \" + limit);\n}","typeGuard":null,"tryCatchPattern":"try {\n    while (iterator.isValid()) { iterator.next(); }\n} catch (IndexOutOfBoundsException e) {\n    if (e.getMessage() != null && e.getMessage().contains(\"current cache size\")) {\n        LOG.error(\"Scan batch invariant broken; raise scanBatchSize or recreate iterator\");\n    }\n    throw e;\n}","preventionTips":["Keep conf.getScanBatchSize() at defaults or comfortably above the per-request limit","Fully drain one ScanIterator before creating another","Avoid shrinking limit/batch values below region-scan response sizes"],"tags":["tikv","scan-iterator","batching","internal-invariant"],"backgroundTag":"index-out-of-bounds","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}