{"record":{"id":"3b9416b3fff7f5aa","repo":"apache/hadoop","slug":"can-t-read-partitions-file","errorCode":null,"errorMessage":"Can't read partitions file","messagePattern":"Can't read partitions file","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java","lineNumber":115,"sourceCode":"      }\n      boolean natOrder =\n        conf.getBoolean(NATURAL_ORDER, true);\n      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {\n        partitions = buildTrie((BinaryComparable[])splitPoints, 0,\n            splitPoints.length, new byte[0],\n            // Now that blocks of identical splitless trie nodes are \n            // represented reentrantly, and we develop a leaf for any trie\n            // node with only one split point, the only reason for a depth\n            // limit is to refute stack overflow or bloat in the pathological\n            // case where the split points are long and mostly look like bytes \n            // iii...iixii...iii   .  Therefore, we make the default depth\n            // limit large but not huge.\n            conf.getInt(MAX_TRIE_DEPTH, 200));\n      } else {\n        partitions = new BinarySearchNode(splitPoints, comparator);\n      }\n    } catch (IOException e) {\n      throw new IllegalArgumentException(\"Can't read partitions file\", e);\n    }\n  }\n\n  public Configuration getConf() {\n    return conf;\n  }\n  \n  // by construction, we know if our keytype\n  @SuppressWarnings(\"unchecked\") // is memcmp-able and uses the trie\n  public int getPartition(K key, V value, int numPartitions) {\n    return partitions.findPartition(key);\n  }\n\n  /**\n   * Set the path to the SequenceFile storing the sorted partition keyset.\n   * It must be the case that for <code>R</code> reduces, there are <code>R-1</code>\n   * keys in the SequenceFile.\n   */","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java#L97-L133","documentation":"This IllegalArgumentException is the catch-all wrapper in TotalOrderPartitioner.setConf around every IOException raised while locating, opening, or reading the partition file (file missing, permissions, EOF while reading points, key deserialization failures). The actual reason is always in the cause chain.","triggerScenarios":"mapreduce.totalorderpartitioner.path unset with no _partition.lst in the job submit dir / DistributedCache; a typo'd path; partition file deleted by cleanup between sampling and task execution; readPartitions failing because the map output key class is not the WritableComparable the file was serialized with.","commonSituations":"Running a chain job where the sampling job's output path is not carried into the partitioner config; reusing examples (TotalOrderPartitioner/Sort example) without writing the partition file; secure clusters where the task cannot read the file's permissions.","solutions":["Read getCause() first: FileNotFoundException means the file was never written or the path is wrong","Write the file with InputSampler.writePartitionFile(job, sampler) and register it via TotalOrderPartitioner.setPartitionFile(conf, path)","Verify the map output key class used at sampling time is identical to the running job's (readPartitions deserializes with job.getMapOutputKeyClass)"],"exampleFix":"// before: relying on a default partition file that was never written\n// after\nTotalOrderPartitioner.setPartitionFile(job.getConfiguration(), new Path(outDir, \"_partition.lst\"));\nInputSampler.writePartitionFile(job, sampler);","handlingStrategy":"try-catch","validationCode":"Path partFile = new Path(conf.get(\"mapreduce.totalorderpartitioner.path\", \"_partition.lst\"));\nFileSystem fs = partFile.getFileSystem(conf);\nif (!fs.exists(partFile)) {\n  throw new IOException(\"Partition file missing: \" + partFile);\n}","typeGuard":null,"tryCatchPattern":"try {\n  Partitioner<K, V> p = new TotalOrderPartitioner<>();\n  p.setConf(conf);\n} catch (IllegalArgumentException e) {\n  Throwable cause = e.getCause();\n  log.error(\"Cannot read partitions file: {}\", cause == null ? e : cause);\n  throw e;\n}","preventionTips":["Always call InputSampler.writePartitionFile and set the path in the same driver","Smoke-test the partitioner's setConf in the submit path to fail fast on the client","Make sure the partition file outlives all task attempts (do not delete output dir mid-job)"],"tags":["mapreduce","partitioning","total-order-sort","file-io","configuration"],"backgroundTag":"file-not-found","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}