apache/hadoop · error · IOException

Wrong number of partitions in keyset

Error message

Wrong number of partitions in keyset

What it means

TotalOrderPartitioner assigns keys to R reducers using R-1 sorted split points read from a partition file (mapreduce.totalorderpartitioner.path, default _partition.lst, normally produced by InputSampler.writePartitionFile). setConf throws this IOException when the number of split points read does not equal getNumReduceTasks() - 1.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java:89

   * the partition keyset using the {@link org.apache.hadoop.io.RawComparator}
   * defined for this job. The input file must be sorted with the same
   * comparator and contain {@link Job#getNumReduceTasks()} - 1 keys.
   */
  @SuppressWarnings("unchecked") // keytype from conf not static
  public void setConf(Configuration conf) {
    try {
      this.conf = conf;
      String parts = getPartitionFile(conf);
      final Path partFile = new Path(parts);
      final FileSystem fs = (DEFAULT_PATH.equals(parts))
        ? FileSystem.getLocal(conf)     // assume in DistributedCache
        : partFile.getFileSystem(conf);

      Job job = Job.getInstance(conf);
      Class<K> keyClass = (Class<K>)job.getMapOutputKeyClass();
      K[] splitPoints = readPartitions(fs, partFile, keyClass, conf);
      if (splitPoints.length != job.getNumReduceTasks() - 1) {
        throw new IOException("Wrong number of partitions in keyset");
      }
      RawComparator<K> comparator =
        (RawComparator<K>) job.getSortComparator();
      for (int i = 0; i < splitPoints.length - 1; ++i) {
        if (comparator.compare(splitPoints[i], splitPoints[i+1]) >= 0) {
          throw new IOException("Split points are out of order");
        }
      }
      boolean natOrder =
        conf.getBoolean(NATURAL_ORDER, true);
      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {
        partitions = buildTrie((BinaryComparable[])splitPoints, 0,
            splitPoints.length, new byte[0],
            // Now that blocks of identical splitless trie nodes are 
            // represented reentrantly, and we develop a leaf for any trie
            // node with only one split point, the only reason for a depth
            // limit is to refute stack overflow or bloat in the pathological
            // case where the split points are long and mostly look like bytes 

View on GitHub (pinned to 2add963021)

Solutions

1. Regenerate the partition file with InputSampler.writePartitionFile(job, sampler) immediately before job.submit() on the same Job instance
2. Keep numReduceTasks and the sample count consistent: the file must hold exactly numReduceTasks-1 points
3. Point mapreduce.totalorderpartitioner.path (or TotalOrderPartitioner.setPartitionFile) at the freshly written file for this job

Example fix

// before: partition file from a previous run with 9 points
Job job = Job.getInstance(conf);
job.setNumReduceTasks(12);
// after: sample for THIS job before submit
job.setNumReduceTasks(12);
InputSampler.writePartitionFile(job, new InputSampler.RandomSampler<Text>(0.1, 10000));

Path partFile = new Path(conf.get("mapreduce.totalorderpartitioner.path", "_partition.lst"));
long expected = job.getNumReduceTasks() - 1;
try (FileSystem fs = partFile.getFileSystem(conf);
     BufferedReader r = new BufferedReader(new InputStreamReader(fs.open(partFile), StandardCharsets.UTF_8))) {
  long lines = r.lines().count();
  if (lines != expected) {
    throw new IOException("Partition file has " + lines + " points but job needs " + expected);
  }
}

try { // smoke-test partitioner setup before submit
  TotalOrderPartitioner<Text, Text> p = new TotalOrderPartitioner<>();
  p.setConf(job.getConfiguration());
} catch (IllegalArgumentException | IOException e) {
  throw new RuntimeException("TotalOrderPartitioner setup invalid: " + e.getMessage(), e);
}

When it happens

Trigger: Calling setNumReduceTasks(R) with a partition file that contains a different count of points: file sampled in a previous run or a separate job, sampler settings changed after the file was written, or the file hand-edited/generated with an arbitrary number of lines.

Common situations: Bumping reduce tasks between runs while reusing the old _partition.lst; sampling in a shell/driver step that runs before the number of reduces is finalized; writing the partition file from a different input than the job actually processes.

Related errors

• Split points are out of order
• Can't read partitions file
• "Not creating intermediate history logDir: [" + doneDirPath
• Missing Log Directory for History
• Only one of the following keys can be specified for a single

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d8b57c7a8c2473e2. Report an issue: GitHub.