{"record":{"id":"1cf9211ddcb889eb","repo":"apache/flink","slug":"task-id-too-large-1cf921","errorCode":null,"errorMessage":"Task id too large.","messagePattern":"Task id too large\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopOutputFormatBase.java","lineNumber":109,"sourceCode":"        synchronized (CONFIGURE_MUTEX) {\n            if (this.mapreduceOutputFormat instanceof Configurable) {\n                ((Configurable) this.mapreduceOutputFormat).setConf(this.configuration);\n            }\n        }\n    }\n\n    /**\n     * create the temporary output file for hadoop RecordWriter.\n     *\n     * @throws java.io.IOException\n     */\n    @Override\n    public void open(InitializationContext context) throws IOException {\n        int taskNumber = context.getTaskNumber();\n        // enforce sequential open() calls\n        synchronized (OPEN_MUTEX) {\n            if (Integer.toString(taskNumber + 1).length() > 6) {\n                throw new IOException(\"Task id too large.\");\n            }\n\n            this.taskNumber = taskNumber + 1;\n\n            // for hadoop 2.2\n            this.configuration.set(\"mapreduce.output.basename\", \"tmp\");\n\n            TaskAttemptID taskAttemptID =\n                    TaskAttemptID.forName(\n                            \"attempt__0000_r_\"\n                                    + String.format(\n                                                    \"%\"\n                                                            + (6\n                                                                    - Integer.toString(\n                                                                                    taskNumber + 1)\n                                                                            .length())\n                                                            + \"s\",\n                                                    \" \")","sourceCodeStart":91,"sourceCodeEnd":127,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopOutputFormatBase.java#L91-L127","documentation":"Thrown in open() when the (zero-based taskNumber + 1) renders to more than 6 digits, because the legacy Hadoop task-attempt ID format HadoopOutputFormatBase constructs only reserves 6 digits for the task number. With task parallelism at or above 1,000,000 the synthetic attempt ID would overflow and the open refuses to proceed.","triggerScenarios":"Produced in HadoopOutputFormatBase.open(context) when context.getTaskNumber()+1 yields a decimal string longer than 6 characters — i.e. the logical parallelism / task index reaches 1,000,000. The code builds a Hadoop TaskAttemptID like 'attempt__0000_r_%06d_0' that cannot represent a 7-digit task number.","commonSituations":"An extremely high parallelism or a subtask index numbering scheme that produces task numbers in the millions; misconfigured parallelism set far higher than intended; testing harnesses that pass artificially large task numbers; running with a partitioning scheme that inflates the logical task count.","solutions":["Lower the operator/job parallelism to a realistic value well under 1,000,000 subtasks.","If the task number comes from a custom InitializationContext/partitioning scheme, ensure it stays within the 6-digit legacy limit.","For genuinely extreme fan-out, partition output through a different sink rather than one HadoopOutputFormat subtask per partition.","Verify the InitializationContext.getTaskNumber() passed to open() is the actual subtask index and not an inflated counter."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before running, ensure parallelism stays within the 6-digit legacy task-id limit\nint parallelism = env.getParallelism(); // or the operator parallelism\nif (String.valueOf(parallelism).length() > 6) {\n    throw new IllegalStateException(\"Parallelism \" + parallelism\n        + \" exceeds the 6-digit HadoopOutputFormat task-id limit (999999). Lower it.\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep operator/job parallelism well under 1,000,000.","If very high fan-out is required, partition output through a different sink.","Validate the InitializationContext task number is a real subtask index, not an inflated counter.","Surface this limit in deployment configs/alerts."],"tags":["hadoop","mapreduce","output-format","parallelism","task-id"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}