{"record":{"id":"bdedb0cbd5700fad","repo":"apache/seatunnel","slug":"produce-too-many-splits","errorCode":null,"errorMessage":"Produce too many splits.","messagePattern":"Produce too many splits\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/PaimonSourceSplitGenerator.java","lineNumber":48,"sourceCode":"    private final char[] currentId = \"0000000000\".toCharArray();\n\n    public List<PaimonSourceSplit> createSplits(String tableId, TableScan.Plan plan) {\n        return plan.splits().stream()\n                .map(s -> new PaimonSourceSplit(getNextId(), tableId, s))\n                .collect(Collectors.toList());\n    }\n\n    protected final String getNextId() {\n        // because we just increment numbers, we increment the char representation directly,\n        // rather than incrementing an integer and converting it to a string representation\n        // every time again (requires quite some expensive conversion logic).\n        incrementCharArrayByOne(currentId, currentId.length - 1);\n        return new String(currentId);\n    }\n\n    private static void incrementCharArrayByOne(char[] array, int pos) {\n        if (pos < 0) {\n            throw new RuntimeException(\"Produce too many splits.\");\n        }\n\n        char c = array[pos];\n        c++;\n\n        if (c > '9') {\n            c = '0';\n            incrementCharArrayByOne(array, pos - 1);\n        }\n        array[pos] = c;\n    }\n}\n","sourceCodeStart":30,"sourceCodeEnd":61,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/PaimonSourceSplitGenerator.java#L30-L61","documentation":"PaimonSourceSplitGenerator assigns split IDs as digit strings by incrementing the last character in place; when the numeric string overflows (carry past position 0) it throws RuntimeException 'Produce too many splits.' It is an internal safeguard against generating more splits than the ID space allows.","triggerScenarios":"Generating more Paimon source splits than the fixed-length numeric ID space supports — e.g. a very large number of snapshot/manifest splits causing getNextId to keep incrementing until carry propagates beyond the first character.","commonSituations":"Huge Paimon tables with millions of small files/snapshots producing enormous split counts; misconfigured split size options too small, exploding the number of splits.","solutions":["Increase split size (e.g. split.size / snapshot split size options) to reduce the number of splits","Reduce the table's small-file count via compaction before reading","Reduce source parallelism/split count requirements for the job","Upgrade the connector if a newer version widens the ID space"],"exampleFix":"// before\nPaimonSourceOptions: read batch size default, tiny split size\n// after\nsink/source config:\n  read.batch-size = 4096\n  split.size = 128MB   # larger splits, fewer IDs","handlingStrategy":"try-catch","validationCode":"long expectedSplits = estimateSplitCount(tableFiles, splitSize); if (expectedSplits > maxSupportedSplits) throw new IllegalArgumentException(\"reduce split size or compact table first\");","typeGuard":null,"tryCatchPattern":"try { splits = generator.generate(...); } catch (RuntimeException e) { if (\"Produce too many splits.\".equals(e.getMessage())) { /* increase split size / compact table, then retry */ } throw e; }","preventionTips":["Tune split size options so split counts stay well below the ID space limit","Run Paimon compaction to reduce small files before large reads","Monitor split counts for large tables during job planning"],"tags":["paimon","source","splits","overflow"],"backgroundTag":"internal-invariant-violation","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}