apache/flink · error · IOException

Task id too large.

Error message

Task id too large.

What it means

Thrown in open() when the (zero-based taskNumber + 1) renders to more than 6 digits, because the legacy Hadoop task-attempt ID format HadoopOutputFormatBase constructs only reserves 6 digits for the task number. With task parallelism at or above 1,000,000 the synthetic attempt ID would overflow and the open refuses to proceed.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopOutputFormatBase.java:109

        synchronized (CONFIGURE_MUTEX) {
            if (this.mapreduceOutputFormat instanceof Configurable) {
                ((Configurable) this.mapreduceOutputFormat).setConf(this.configuration);
            }
        }
    }

    /**
     * create the temporary output file for hadoop RecordWriter.
     *
     * @throws java.io.IOException
     */
    @Override
    public void open(InitializationContext context) throws IOException {
        int taskNumber = context.getTaskNumber();
        // enforce sequential open() calls
        synchronized (OPEN_MUTEX) {
            if (Integer.toString(taskNumber + 1).length() > 6) {
                throw new IOException("Task id too large.");
            }

            this.taskNumber = taskNumber + 1;

            // for hadoop 2.2
            this.configuration.set("mapreduce.output.basename", "tmp");

            TaskAttemptID taskAttemptID =
                    TaskAttemptID.forName(
                            "attempt__0000_r_"
                                    + String.format(
                                                    "%"
                                                            + (6
                                                                    - Integer.toString(
                                                                                    taskNumber + 1)
                                                                            .length())
                                                            + "s",
                                                    " ")

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Lower the operator/job parallelism to a realistic value well under 1,000,000 subtasks.
2. If the task number comes from a custom InitializationContext/partitioning scheme, ensure it stays within the 6-digit legacy limit.
3. For genuinely extreme fan-out, partition output through a different sink rather than one HadoopOutputFormat subtask per partition.
4. Verify the InitializationContext.getTaskNumber() passed to open() is the actual subtask index and not an inflated counter.

// Before running, ensure parallelism stays within the 6-digit legacy task-id limit
int parallelism = env.getParallelism(); // or the operator parallelism
if (String.valueOf(parallelism).length() > 6) {
    throw new IllegalStateException("Parallelism " + parallelism
        + " exceeds the 6-digit HadoopOutputFormat task-id limit (999999). Lower it.");
}

When it happens

Trigger: Produced in HadoopOutputFormatBase.open(context) when context.getTaskNumber()+1 yields a decimal string longer than 6 characters — i.e. the logical parallelism / task index reaches 1,000,000. The code builds a Hadoop TaskAttemptID like 'attempt__0000_r_%06d_0' that cannot represent a 7-digit task number.

Common situations: An extremely high parallelism or a subtask index numbering scheme that produces task numbers in the millions; misconfigured parallelism set far higher than intended; testing harnesses that pass artificially large task numbers; running with a partitioning scheme that inflates the logical task count.

Related errors

• Could not write Record.
• Could not create RecordWriter.
• Could not close RecordReader.
• Unable to instantiate the hadoop output format
• Task id too large.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/1cf9211ddcb889eb. Report an issue: GitHub.