What causes "Produce too many splits." in apache/seatunnel?

Increase split size (e.g. split.size / snapshot split size options) to reduce the number of splits

apache/seatunnel · error · RuntimeException

Produce too many splits.

Error message

Produce too many splits.

What it means

PaimonSourceSplitGenerator assigns split IDs as digit strings by incrementing the last character in place; when the numeric string overflows (carry past position 0) it throws RuntimeException 'Produce too many splits.' It is an internal safeguard against generating more splits than the ID space allows.

Solutions

Increase split size (e.g. split.size / snapshot split size options) to reduce the number of splits
Reduce the table's small-file count via compaction before reading
Reduce source parallelism/split count requirements for the job
Upgrade the connector if a newer version widens the ID space

Example fix

// before
PaimonSourceOptions: read batch size default, tiny split size
// after
sink/source config:
  read.batch-size = 4096
  split.size = 128MB   # larger splits, fewer IDs

Defensive patterns

Strategy: try-catch

Validate before calling

long expectedSplits = estimateSplitCount(tableFiles, splitSize); if (expectedSplits > maxSupportedSplits) throw new IllegalArgumentException("reduce split size or compact table first");

Try / catch

try { splits = generator.generate(...); } catch (RuntimeException e) { if ("Produce too many splits.".equals(e.getMessage())) { /* increase split size / compact table, then retry */ } throw e; }

Prevention

Tune split size options so split counts stay well below the ID space limit
Run Paimon compaction to reduce small files before large reads
Monitor split counts for large tables during job planning

When it happens

Trigger: Generating more Paimon source splits than the fixed-length numeric ID space supports — e.g. a very large number of snapshot/manifest splits causing getNextId to keep incrementing until carry propagates beyond the first character.

Common situations: Huge Paimon tables with millions of small files/snapshots producing enormous split counts; misconfigured split size options too small, exploding the number of splits.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/bdedb0cbd5700fad. Report an issue: GitHub.

Appendix: source

Thrown at seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/PaimonSourceSplitGenerator.java:48

    private final char[] currentId = "0000000000".toCharArray();

    public List<PaimonSourceSplit> createSplits(String tableId, TableScan.Plan plan) {
        return plan.splits().stream()
                .map(s -> new PaimonSourceSplit(getNextId(), tableId, s))
                .collect(Collectors.toList());
    }

    protected final String getNextId() {
        // because we just increment numbers, we increment the char representation directly,
        // rather than incrementing an integer and converting it to a string representation
        // every time again (requires quite some expensive conversion logic).
        incrementCharArrayByOne(currentId, currentId.length - 1);
        return new String(currentId);
    }

    private static void incrementCharArrayByOne(char[] array, int pos) {
        if (pos < 0) {
            throw new RuntimeException("Produce too many splits.");
        }

        char c = array[pos];
        c++;

        if (c > '9') {
            c = '0';
            incrementCharArrayByOne(array, pos - 1);
        }
        array[pos] = c;
    }
}

View on GitHub (pinned to cf67b549a7)