apache/hadoop · error · IllegalArgumentException

Can't read partitions file

Error message

Can't read partitions file

What it means

This IllegalArgumentException is the catch-all wrapper in TotalOrderPartitioner.setConf around every IOException raised while locating, opening, or reading the partition file (file missing, permissions, EOF while reading points, key deserialization failures). The actual reason is always in the cause chain.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java:115

      }
      boolean natOrder =
        conf.getBoolean(NATURAL_ORDER, true);
      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {
        partitions = buildTrie((BinaryComparable[])splitPoints, 0,
            splitPoints.length, new byte[0],
            // Now that blocks of identical splitless trie nodes are 
            // represented reentrantly, and we develop a leaf for any trie
            // node with only one split point, the only reason for a depth
            // limit is to refute stack overflow or bloat in the pathological
            // case where the split points are long and mostly look like bytes 
            // iii...iixii...iii   .  Therefore, we make the default depth
            // limit large but not huge.
            conf.getInt(MAX_TRIE_DEPTH, 200));
      } else {
        partitions = new BinarySearchNode(splitPoints, comparator);
      }
    } catch (IOException e) {
      throw new IllegalArgumentException("Can't read partitions file", e);
    }
  }

  public Configuration getConf() {
    return conf;
  }
  
  // by construction, we know if our keytype
  @SuppressWarnings("unchecked") // is memcmp-able and uses the trie
  public int getPartition(K key, V value, int numPartitions) {
    return partitions.findPartition(key);
  }

  /**
   * Set the path to the SequenceFile storing the sorted partition keyset.
   * It must be the case that for <code>R</code> reduces, there are <code>R-1</code>
   * keys in the SequenceFile.
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Read getCause() first: FileNotFoundException means the file was never written or the path is wrong
2. Write the file with InputSampler.writePartitionFile(job, sampler) and register it via TotalOrderPartitioner.setPartitionFile(conf, path)
3. Verify the map output key class used at sampling time is identical to the running job's (readPartitions deserializes with job.getMapOutputKeyClass)

Example fix

// before: relying on a default partition file that was never written
// after
TotalOrderPartitioner.setPartitionFile(job.getConfiguration(), new Path(outDir, "_partition.lst"));
InputSampler.writePartitionFile(job, sampler);

Path partFile = new Path(conf.get("mapreduce.totalorderpartitioner.path", "_partition.lst"));
FileSystem fs = partFile.getFileSystem(conf);
if (!fs.exists(partFile)) {
  throw new IOException("Partition file missing: " + partFile);
}

try {
  Partitioner<K, V> p = new TotalOrderPartitioner<>();
  p.setConf(conf);
} catch (IllegalArgumentException e) {
  Throwable cause = e.getCause();
  log.error("Cannot read partitions file: {}", cause == null ? e : cause);
  throw e;
}

When it happens

Trigger: mapreduce.totalorderpartitioner.path unset with no _partition.lst in the job submit dir / DistributedCache; a typo'd path; partition file deleted by cleanup between sampling and task execution; readPartitions failing because the map output key class is not the WritableComparable the file was serialized with.

Common situations: Running a chain job where the sampling job's output path is not carried into the partitioner config; reusing examples (TotalOrderPartitioner/Sort example) without writing the partition file; secure clusters where the task cannot read the file's permissions.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Wrong number of partitions in keyset
• Couldn't open queue configuration at " + xmlInUrl
• Split points are out of order
• "Not creating intermediate history logDir: [" + doneDirPath
• Missing Log Directory for History

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3b9416b3fff7f5aa. Report an issue: GitHub.