apache/hadoop · error · IOException

Wrong number of partitions in keyset

Error message

Wrong number of partitions in keyset

What it means

TotalOrderPartitioner assigns keys to R reducers using R-1 sorted split points read from a partition file (mapreduce.totalorderpartitioner.path, default _partition.lst, normally produced by InputSampler.writePartitionFile). setConf throws this IOException when the number of split points read does not equal getNumReduceTasks() - 1.

Solutions

1. Regenerate the partition file with InputSampler.writePartitionFile(job, sampler) immediately before job.submit() on the same Job instance
2. Keep numReduceTasks and the sample count consistent: the file must hold exactly numReduceTasks-1 points
3. Point mapreduce.totalorderpartitioner.path (or TotalOrderPartitioner.setPartitionFile) at the freshly written file for this job

Example fix

// before: partition file from a previous run with 9 points
Job job = Job.getInstance(conf);
job.setNumReduceTasks(12);
// after: sample for THIS job before submit
job.setNumReduceTasks(12);
InputSampler.writePartitionFile(job, new InputSampler.RandomSampler<Text>(0.1, 10000));

Path partFile = new Path(conf.get("mapreduce.totalorderpartitioner.path", "_partition.lst"));
long expected = job.getNumReduceTasks() - 1;
try (FileSystem fs = partFile.getFileSystem(conf);
     BufferedReader r = new BufferedReader(new InputStreamReader(fs.open(partFile), StandardCharsets.UTF_8))) {
  long lines = r.lines().count();
  if (lines != expected) {
    throw new IOException("Partition file has " + lines + " points but job needs " + expected);
  }
}

try { // smoke-test partitioner setup before submit
  TotalOrderPartitioner<Text, Text> p = new TotalOrderPartitioner<>();
  p.setConf(job.getConfiguration());
} catch (IllegalArgumentException | IOException e) {
  throw new RuntimeException("TotalOrderPartitioner setup invalid: " + e.getMessage(), e);
}

When it happens

Trigger: Calling setNumReduceTasks(R) with a partition file that contains a different count of points: file sampled in a previous run or a separate job, sampler settings changed after the file was written, or the file hand-edited/generated with an arbitrary number of lines.

Common situations: Bumping reduce tasks between runs while reusing the old _partition.lst; sampling in a shell/driver step that runs before the number of reduces is finalized; writing the partition file from a different input than the job actually processes.

Related errors

• Can't read partitions file
• Split points are out of order
• Can't get Master Kerberos principal for use as renewer
• Can't have recursive MultithreadedMapper instances.
• Commit directory not configured

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d8b57c7a8c2473e2. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java:89

   * the partition keyset using the {@link org.apache.hadoop.io.RawComparator}
   * defined for this job. The input file must be sorted with the same
   * comparator and contain {@link Job#getNumReduceTasks()} - 1 keys.
   */
  @SuppressWarnings("unchecked") // keytype from conf not static
  public void setConf(Configuration conf) {
    try {
      this.conf = conf;
      String parts = getPartitionFile(conf);
      final Path partFile = new Path(parts);
      final FileSystem fs = (DEFAULT_PATH.equals(parts))
        ? FileSystem.getLocal(conf)     // assume in DistributedCache
        : partFile.getFileSystem(conf);

      Job job = Job.getInstance(conf);
      Class<K> keyClass = (Class<K>)job.getMapOutputKeyClass();
      K[] splitPoints = readPartitions(fs, partFile, keyClass, conf);
      if (splitPoints.length != job.getNumReduceTasks() - 1) {
        throw new IOException("Wrong number of partitions in keyset");
      }
      RawComparator<K> comparator =
        (RawComparator<K>) job.getSortComparator();
      for (int i = 0; i < splitPoints.length - 1; ++i) {
        if (comparator.compare(splitPoints[i], splitPoints[i+1]) >= 0) {
          throw new IOException("Split points are out of order");
        }
      }
      boolean natOrder =
        conf.getBoolean(NATURAL_ORDER, true);
      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {
        partitions = buildTrie((BinaryComparable[])splitPoints, 0,
            splitPoints.length, new byte[0],
            // Now that blocks of identical splitless trie nodes are 
            // represented reentrantly, and we develop a leaf for any trie
            // node with only one split point, the only reason for a depth
            // limit is to refute stack overflow or bloat in the pathological
            // case where the split points are long and mostly look like bytes 

View on GitHub (pinned to 2add963021)