What causes "Split points are out of order" in apache/hadoop?

Dedupe the split points, then re-check the count equals numReduceTasks-1

apache/hadoop · error · IOException

Split points are out of order

Error message

Split points are out of order

What it means

After the count check, TotalOrderPartitioner.setConf validates that split points strictly increase under the job's sort comparator: compare(splitPoints[i], splitPoints[i+1]) >= 0 for any adjacent pair throws this IOException. Equal or descending neighbours mean the file is not a strictly sorted sequence of cut points.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java:95

    try {
      this.conf = conf;
      String parts = getPartitionFile(conf);
      final Path partFile = new Path(parts);
      final FileSystem fs = (DEFAULT_PATH.equals(parts))
        ? FileSystem.getLocal(conf)     // assume in DistributedCache
        : partFile.getFileSystem(conf);

      Job job = Job.getInstance(conf);
      Class<K> keyClass = (Class<K>)job.getMapOutputKeyClass();
      K[] splitPoints = readPartitions(fs, partFile, keyClass, conf);
      if (splitPoints.length != job.getNumReduceTasks() - 1) {
        throw new IOException("Wrong number of partitions in keyset");
      }
      RawComparator<K> comparator =
        (RawComparator<K>) job.getSortComparator();
      for (int i = 0; i < splitPoints.length - 1; ++i) {
        if (comparator.compare(splitPoints[i], splitPoints[i+1]) >= 0) {
          throw new IOException("Split points are out of order");
        }
      }
      boolean natOrder =
        conf.getBoolean(NATURAL_ORDER, true);
      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {
        partitions = buildTrie((BinaryComparable[])splitPoints, 0,
            splitPoints.length, new byte[0],
            // Now that blocks of identical splitless trie nodes are 
            // represented reentrantly, and we develop a leaf for any trie
            // node with only one split point, the only reason for a depth
            // limit is to refute stack overflow or bloat in the pathological
            // case where the split points are long and mostly look like bytes 
            // iii...iixii...iii   .  Therefore, we make the default depth
            // limit large but not huge.
            conf.getInt(MAX_TRIE_DEPTH, 200));
      } else {
        partitions = new BinarySearchNode(splitPoints, comparator);
      }

View on GitHub (pinned to 2add963021)

Solutions

Dedupe the split points, then re-check the count equals numReduceTasks-1
Increase sample size (more samples / higher frequency in InputSampler) so distinct cut points exist
Ensure the job's map output key class and sort comparator are identical to those used when the partition file was written

Example fix

// before: too few samples on a low-cardinality key -> duplicate points
new InputSampler.RandomSampler<>(0.01, 10);
// after: larger sample, then dedupe and match count to R-1
new InputSampler.RandomSampler<>(0.5, 10000);

Defensive patterns

Strategy: validation

Validate before calling

RawComparator<Text> cmp = (RawComparator<Text>) job.getSortComparator();
List<Text> points = readSplitPoints(partFile); // same deserialization as readPartitions
for (int i = 0; i + 1 < points.size(); i++) {
  if (cmp.compare(points.get(i), points.get(i + 1)) >= 0) {
    throw new IOException("Split points not strictly increasing at index " + i);
  }
}

Try / catch

try {
  TotalOrderPartitioner<Text, Text> p = new TotalOrderPartitioner<>();
  p.setConf(job.getConfiguration());
} catch (IllegalArgumentException | IOException e) {
  throw new RuntimeException("Split points invalid: " + e.getMessage(), e);
}

Prevention

Dedupe sampled cut points and verify count equals R-1 before submit
Keep the key class and comparator identical between sampling and the job
Check sample cardinality: low-distinct keys need a different partitioning strategy

When it happens

Trigger: Duplicate keys among the sampled points (low-cardinality key space with RandomSampler/HashSampler); a partition file written with a different key class or comparator ordering than the running job uses (e.g. custom RawComparator vs natural Text ordering); hand-written file not sorted.

Common situations: Sampling a key column with very few distinct values so the same cut point appears twice; switching the map output key class or comparator after the file was generated; sorting the file with the wrong collation (locale-aware sort on the shell).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1352d658913d2cf4. Report an issue: GitHub.