apache/flink · error · IOException

Task id too large.

Error message

Task id too large.

What it means

Thrown by HadoopOutputFormatBase.open when (taskNumber + 1) formatted as a string exceeds 6 characters. Hadoop's TaskAttemptID format reserves exactly 6 digits for the task number (attempt__0000_r_NNNNNN_0), so parallelism above 999999 subtasks overflows the field. This is a hard limit imposed by the Hadoop ID format, not by Flink.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java:120

            } else if (this.mapredOutputFormat instanceof JobConfigurable) {
                ((JobConfigurable) this.mapredOutputFormat).configure(this.jobConf);
            }
        }
    }

    /**
     * create the temporary output file for hadoop RecordWriter.
     *
     * @param context The initialization context.
     * @throws java.io.IOException
     */
    @Override
    public void open(InitializationContext context) throws IOException {
        int taskNumber = context.getTaskNumber();
        // enforce sequential open() calls
        synchronized (OPEN_MUTEX) {
            if (Integer.toString(taskNumber + 1).length() > 6) {
                throw new IOException("Task id too large.");
            }

            TaskAttemptID taskAttemptID =
                    TaskAttemptID.forName(
                            "attempt__0000_r_"
                                    + String.format(
                                                    "%"
                                                            + (6
                                                                    - Integer.toString(
                                                                                    taskNumber + 1)
                                                                            .length())
                                                            + "s",
                                                    " ")
                                            .replace(" ", "0")
                                    + Integer.toString(taskNumber + 1)
                                    + "_0");

            this.jobConf.set("mapred.task.id", taskAttemptID.toString());

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Reduce the operator parallelism to below 999999 — values this high are almost always a misconfiguration.
2. If genuinely high parallelism is needed, use a Flink-native sink instead of HadoopOutputFormat which does not have this ID format constraint.
3. Review the job graph to ensure parallelism is set intentionally and not inherited from an inflated default.

// Validate parallelism does not exceed Hadoop TaskAttemptID limit before deploying
int parallelism = env.getParallelism();
if (parallelism > 999999) {
    throw new IllegalArgumentException(
        "HadoopOutputFormat does not support parallelism > 999999 (Hadoop TaskAttemptID limit). "
        + "Current: " + parallelism);
}

When it happens

Trigger: Deploying a HadoopOutputFormat job with source/sink parallelism greater than 999999, causing the task number to exceed the 6-digit Hadoop TaskAttemptID field.

Common situations: Extremely high parallelism configurations (which are rare and usually a misconfiguration); test environments with accidental parallelism values; jobs with very high parallelism on large clusters.

Related errors

• Unable to instantiate the hadoop output format
• Task id too large.
• Number of input splits has to be at least 1.
• Number of input splits has to be at least 1.
• No cluster id was specified. Please specify a cluster to whi

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/a5ed1c381e216043. Report an issue: GitHub.