apache/hadoop · error · IOException

Illegal partition for {key} ({partition})

Error message

Illegal partition for {key} ({partition})

What it means

After the Partitioner runs, MapTask validates the returned index against the reducer count (0 <= partition < job.getNumReduceTasks()). An out-of-range index fails the map task on the first bad record with this IOException, which names the key and the illegal partition.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java:1107

     * Serialize the key, value to intermediate storage.
     * When this method returns, kvindex must refer to sufficient unused
     * storage to store one METADATA.
     */
    public synchronized void collect(K key, V value, final int partition
                                     ) throws IOException {
      reporter.progress();
      if (key.getClass() != keyClass) {
        throw new IOException("Type mismatch in key from map: expected "
                              + keyClass.getName() + ", received "
                              + key.getClass().getName());
      }
      if (value.getClass() != valClass) {
        throw new IOException("Type mismatch in value from map: expected "
                              + valClass.getName() + ", received "
                              + value.getClass().getName());
      }
      if (partition < 0 || partition >= partitions) {
        throw new IOException("Illegal partition for " + key + " (" +
            partition + ")");
      }
      checkSpillException();
      bufferRemaining -= METASIZE;
      if (bufferRemaining <= 0) {
        // start spill if the thread is not running and the soft limit has been
        // reached
        spillLock.lock();
        try {
          do {
            if (!spillInProgress) {
              final int kvbidx = 4 * kvindex;
              final int kvbend = 4 * kvend;
              // serialized, unspilled bytes always lie between kvindex and
              // bufindex, crossing the equator. Note that any void space
              // created by a reset must be included in "used" bytes
              final int bUsed = distanceTo(kvbidx, bufindex);
              final boolean bufsoftlimit = bUsed >= softLimit;

View on GitHub (pinned to 2add963021)

Solutions

1. Mask before mod in the partitioner: return (key.hashCode() & Integer.MAX_VALUE) % numPartitions (the HashPartitioner idiom)
2. Audit every return path of the custom partitioner for 0 <= p < numPartitions using the runtime numPartitions argument
3. Add a property test over the key domain (including negative hashCodes) for the partitioner

Example fix

// before
public int getPartition(IntWritable key, Text value, int numPartitions) {
  return key.get() % numPartitions;          // negative keys -> negative partition
}

// after
public int getPartition(IntWritable key, Text value, int numPartitions) {
  return (key.get() & Integer.MAX_VALUE) % numPartitions;
}

// property test before shipping: partitioner stays in range for the whole key domain
Random r = new Random();
for (int i = 0; i < 100_000; i++) {
  IntWritable k = new IntWritable(r.nextInt()); // includes negative hashes
  int p = new MyPartitioner().getPartition(k, new Text("x"), numReduces);
  if (p < 0 || p >= numReduces) throw new AssertionError("partition " + p + " out of range");
}

When it happens

Trigger: Custom partitioner computing key.hashCode() % numPartitions (negative when hashCode() < 0); returning a constant equal to numPartitions; partitioner caching a partition count valid only for a different job.setNumReduceTasks value.

Common situations: Hand-rolled partitioners that skip the sign mask; partitioners tested only with non-negative keys (strings) then fed numeric keys; jobs where numReduceTasks was changed after the partitioner was written; off-by-one bugs near the boundary.

Related errors

• Invalid read parameters: buf.length=%d, off=%d, len=%d
• Unable to recover task %s, output: %s
• Requested more bytes than destination buffer size: request l
• write (b[{b.length}], {off}, {len})
• Bad key buffer offset-length combination.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cffb7cd78929d1d6. Report an issue: GitHub.