apache/flink · error · IOException

Could not get Splits.

Error message

Could not get Splits.

What it means

Wraps an InterruptedException thrown by mapreduceInputFormat.getSplits(jobContext) during createInputSplits(), which runs on the job client / JobManager when computing input splits before scheduling. Flink re-throws the Hadoop InterruptedException as an IOException so it surfaces through the InputFormat API. It indicates split computation was interrupted, not that the input is malformed.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopInputFormatBase.java:166

    }

    @Override
    public HadoopInputSplit[] createInputSplits(int minNumSplits) throws IOException {
        configuration.setInt("mapreduce.input.fileinputformat.split.minsize", minNumSplits);

        JobContext jobContext = new JobContextImpl(configuration, new JobID());

        jobContext.getCredentials().addAll(this.credentials);
        Credentials currentUserCreds = getCredentialsFromUGI(UserGroupInformation.getCurrentUser());
        if (currentUserCreds != null) {
            jobContext.getCredentials().addAll(currentUserCreds);
        }

        List<org.apache.hadoop.mapreduce.InputSplit> splits;
        try {
            splits = this.mapreduceInputFormat.getSplits(jobContext);
        } catch (InterruptedException e) {
            throw new IOException("Could not get Splits.", e);
        }
        HadoopInputSplit[] hadoopInputSplits = new HadoopInputSplit[splits.size()];

        for (int i = 0; i < hadoopInputSplits.length; i++) {
            hadoopInputSplits[i] = new HadoopInputSplit(i, splits.get(i), jobContext);
        }
        return hadoopInputSplits;
    }

    @Override
    public InputSplitAssigner getInputSplitAssigner(HadoopInputSplit[] inputSplits) {
        return new LocatableInputSplitAssigner(inputSplits);
    }

    @Override
    public void open(HadoopInputSplit split) throws IOException {

        // enforce sequential open() calls

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Inspect the causing InterruptedException in the logs to see what interrupted split computation (job cancellation, client shutdown, or an I/O timeout on the listing).
2. If split listing against S3/GCS is slow/flaky, retry the submission and verify filesystem connectivity and credentials.
3. For huge inputs, consider pre-computing splits or raising client-side timeouts rather than relying on interruptible listing.
4. If the job was intentionally cancelled, this is expected and can be ignored.

try {
    hadoopInputSplits = hadoopInputFormat.createInputSplits(minSplits);
} catch (IOException e) {
    if (e.getCause() instanceof InterruptedException && isTransient(jobClient)) {
        // split listing interrupted (often remote FS hiccup) — back off and retry submission
        retrySubmissionWithBackoff();
    } else {
        throw e;
    }
}

When it happens

Trigger: Produced when HadoopInputFormatBase.createInputSplits(minNumSplits) calls the underlying Hadoop InputFormat.getSplits(jobContext) and that call throws InterruptedException — e.g. the client was cancelled during split listing, a listing of a remote filesystem (HDFS/S3) was interrupted, or the underlying InputFormat performs blocking work that got interrupted.

Common situations: Job submission cancelled while listing splits against a slow or remote filesystem; S3/HDFS connectivity drops during split computation; a very large directory listing interrupted by a timeout; concurrent job client shutdown.

Related errors

• Could not get KeyValue pair.
• Could not create RecordReader.
• Could not fetch next KeyValue pair.
• Unable to instantiate the hadoop input format
• Could not write Record.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/737fc5f8342cacc9. Report an issue: GitHub.