{"record":{"id":"cffb7cd78929d1d6","repo":"apache/hadoop","slug":"illegal-partition-for-key-partition","errorCode":null,"errorMessage":"Illegal partition for {key} ({partition})","messagePattern":"Illegal partition for (.+?) \\((.+?)\\)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java","lineNumber":1107,"sourceCode":"     * Serialize the key, value to intermediate storage.\n     * When this method returns, kvindex must refer to sufficient unused\n     * storage to store one METADATA.\n     */\n    public synchronized void collect(K key, V value, final int partition\n                                     ) throws IOException {\n      reporter.progress();\n      if (key.getClass() != keyClass) {\n        throw new IOException(\"Type mismatch in key from map: expected \"\n                              + keyClass.getName() + \", received \"\n                              + key.getClass().getName());\n      }\n      if (value.getClass() != valClass) {\n        throw new IOException(\"Type mismatch in value from map: expected \"\n                              + valClass.getName() + \", received \"\n                              + value.getClass().getName());\n      }\n      if (partition < 0 || partition >= partitions) {\n        throw new IOException(\"Illegal partition for \" + key + \" (\" +\n            partition + \")\");\n      }\n      checkSpillException();\n      bufferRemaining -= METASIZE;\n      if (bufferRemaining <= 0) {\n        // start spill if the thread is not running and the soft limit has been\n        // reached\n        spillLock.lock();\n        try {\n          do {\n            if (!spillInProgress) {\n              final int kvbidx = 4 * kvindex;\n              final int kvbend = 4 * kvend;\n              // serialized, unspilled bytes always lie between kvindex and\n              // bufindex, crossing the equator. Note that any void space\n              // created by a reset must be included in \"used\" bytes\n              final int bUsed = distanceTo(kvbidx, bufindex);\n              final boolean bufsoftlimit = bUsed >= softLimit;","sourceCodeStart":1089,"sourceCodeEnd":1125,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java#L1089-L1125","documentation":"After the Partitioner runs, MapTask validates the returned index against the reducer count (0 <= partition < job.getNumReduceTasks()). An out-of-range index fails the map task on the first bad record with this IOException, which names the key and the illegal partition.","triggerScenarios":"Custom partitioner computing key.hashCode() % numPartitions (negative when hashCode() < 0); returning a constant equal to numPartitions; partitioner caching a partition count valid only for a different job.setNumReduceTasks value.","commonSituations":"Hand-rolled partitioners that skip the sign mask; partitioners tested only with non-negative keys (strings) then fed numeric keys; jobs where numReduceTasks was changed after the partitioner was written; off-by-one bugs near the boundary.","solutions":["Mask before mod in the partitioner: return (key.hashCode() & Integer.MAX_VALUE) % numPartitions (the HashPartitioner idiom)","Audit every return path of the custom partitioner for 0 <= p < numPartitions using the runtime numPartitions argument","Add a property test over the key domain (including negative hashCodes) for the partitioner"],"exampleFix":"// before\npublic int getPartition(IntWritable key, Text value, int numPartitions) {\n  return key.get() % numPartitions;          // negative keys -> negative partition\n}\n\n// after\npublic int getPartition(IntWritable key, Text value, int numPartitions) {\n  return (key.get() & Integer.MAX_VALUE) % numPartitions;\n}","handlingStrategy":"validation","validationCode":"// property test before shipping: partitioner stays in range for the whole key domain\nRandom r = new Random();\nfor (int i = 0; i < 100_000; i++) {\n  IntWritable k = new IntWritable(r.nextInt()); // includes negative hashes\n  int p = new MyPartitioner().getPartition(k, new Text(\"x\"), numReduces);\n  if (p < 0 || p >= numReduces) throw new AssertionError(\"partition \" + p + \" out of range\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Copy HashPartitioner's (hash & Integer.MAX_VALUE) % numPartitions idiom","Never cache numPartitions inside a partitioner - read the method argument each call","Boundary-test with the exact numReduceTasks the job will run with"],"tags":["hadoop","mapreduce","partitioner","index-out-of-bounds","custom-partitioner"],"backgroundTag":"partition-index-out-of-range","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}