{"record":{"id":"a5ed1c381e216043","repo":"apache/flink","slug":"task-id-too-large","errorCode":null,"errorMessage":"Task id too large.","messagePattern":"Task id too large\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java","lineNumber":120,"sourceCode":"            } else if (this.mapredOutputFormat instanceof JobConfigurable) {\n                ((JobConfigurable) this.mapredOutputFormat).configure(this.jobConf);\n            }\n        }\n    }\n\n    /**\n     * create the temporary output file for hadoop RecordWriter.\n     *\n     * @param context The initialization context.\n     * @throws java.io.IOException\n     */\n    @Override\n    public void open(InitializationContext context) throws IOException {\n        int taskNumber = context.getTaskNumber();\n        // enforce sequential open() calls\n        synchronized (OPEN_MUTEX) {\n            if (Integer.toString(taskNumber + 1).length() > 6) {\n                throw new IOException(\"Task id too large.\");\n            }\n\n            TaskAttemptID taskAttemptID =\n                    TaskAttemptID.forName(\n                            \"attempt__0000_r_\"\n                                    + String.format(\n                                                    \"%\"\n                                                            + (6\n                                                                    - Integer.toString(\n                                                                                    taskNumber + 1)\n                                                                            .length())\n                                                            + \"s\",\n                                                    \" \")\n                                            .replace(\" \", \"0\")\n                                    + Integer.toString(taskNumber + 1)\n                                    + \"_0\");\n\n            this.jobConf.set(\"mapred.task.id\", taskAttemptID.toString());","sourceCodeStart":102,"sourceCodeEnd":138,"githubUrl":"https://github.com/apache/flink/blob/2f3c205e9266cb30240eb7f4fdab15cad629a70f/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java#L102-L138","documentation":"Thrown by HadoopOutputFormatBase.open when (taskNumber + 1) formatted as a string exceeds 6 characters. Hadoop's TaskAttemptID format reserves exactly 6 digits for the task number (attempt__0000_r_NNNNNN_0), so parallelism above 999999 subtasks overflows the field. This is a hard limit imposed by the Hadoop ID format, not by Flink.","triggerScenarios":"Deploying a HadoopOutputFormat job with source/sink parallelism greater than 999999, causing the task number to exceed the 6-digit Hadoop TaskAttemptID field.","commonSituations":"Extremely high parallelism configurations (which are rare and usually a misconfiguration); test environments with accidental parallelism values; jobs with very high parallelism on large clusters.","solutions":["Reduce the operator parallelism to below 999999 — values this high are almost always a misconfiguration.","If genuinely high parallelism is needed, use a Flink-native sink instead of HadoopOutputFormat which does not have this ID format constraint.","Review the job graph to ensure parallelism is set intentionally and not inherited from an inflated default."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Validate parallelism does not exceed Hadoop TaskAttemptID limit before deploying\nint parallelism = env.getParallelism();\nif (parallelism > 999999) {\n    throw new IllegalArgumentException(\n        \"HadoopOutputFormat does not support parallelism > 999999 (Hadoop TaskAttemptID limit). \"\n        + \"Current: \" + parallelism);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep HadoopOutputFormat parallelism below 999999 — such high values are virtually always a misconfiguration.","If extreme parallelism is needed, use a Flink-native sink instead.","Validate parallelism in job submission scripts."],"tags":["hadoop-compatibility","output-format","parallelism","configuration"],"backgroundTag":null,"analyzedSha":"2f3c205e9266cb30240eb7f4fdab15cad629a70f","analyzedAt":"2026-08-14T08:48:24.518Z","schemaVersion":2},"datasetVersion":"2026-08-14T10:17:34.591Z"}