{"record":{"id":"d8b57c7a8c2473e2","repo":"apache/hadoop","slug":"wrong-number-of-partitions-in-keyset","errorCode":null,"errorMessage":"Wrong number of partitions in keyset","messagePattern":"Wrong number of partitions in keyset","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java","lineNumber":89,"sourceCode":"   * the partition keyset using the {@link org.apache.hadoop.io.RawComparator}\n   * defined for this job. The input file must be sorted with the same\n   * comparator and contain {@link Job#getNumReduceTasks()} - 1 keys.\n   */\n  @SuppressWarnings(\"unchecked\") // keytype from conf not static\n  public void setConf(Configuration conf) {\n    try {\n      this.conf = conf;\n      String parts = getPartitionFile(conf);\n      final Path partFile = new Path(parts);\n      final FileSystem fs = (DEFAULT_PATH.equals(parts))\n        ? FileSystem.getLocal(conf)     // assume in DistributedCache\n        : partFile.getFileSystem(conf);\n\n      Job job = Job.getInstance(conf);\n      Class<K> keyClass = (Class<K>)job.getMapOutputKeyClass();\n      K[] splitPoints = readPartitions(fs, partFile, keyClass, conf);\n      if (splitPoints.length != job.getNumReduceTasks() - 1) {\n        throw new IOException(\"Wrong number of partitions in keyset\");\n      }\n      RawComparator<K> comparator =\n        (RawComparator<K>) job.getSortComparator();\n      for (int i = 0; i < splitPoints.length - 1; ++i) {\n        if (comparator.compare(splitPoints[i], splitPoints[i+1]) >= 0) {\n          throw new IOException(\"Split points are out of order\");\n        }\n      }\n      boolean natOrder =\n        conf.getBoolean(NATURAL_ORDER, true);\n      if (natOrder && BinaryComparable.class.isAssignableFrom(keyClass)) {\n        partitions = buildTrie((BinaryComparable[])splitPoints, 0,\n            splitPoints.length, new byte[0],\n            // Now that blocks of identical splitless trie nodes are \n            // represented reentrantly, and we develop a leaf for any trie\n            // node with only one split point, the only reason for a depth\n            // limit is to refute stack overflow or bloat in the pathological\n            // case where the split points are long and mostly look like bytes ","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java#L71-L107","documentation":"TotalOrderPartitioner assigns keys to R reducers using R-1 sorted split points read from a partition file (mapreduce.totalorderpartitioner.path, default _partition.lst, normally produced by InputSampler.writePartitionFile). setConf throws this IOException when the number of split points read does not equal getNumReduceTasks() - 1.","triggerScenarios":"Calling setNumReduceTasks(R) with a partition file that contains a different count of points: file sampled in a previous run or a separate job, sampler settings changed after the file was written, or the file hand-edited/generated with an arbitrary number of lines.","commonSituations":"Bumping reduce tasks between runs while reusing the old _partition.lst; sampling in a shell/driver step that runs before the number of reduces is finalized; writing the partition file from a different input than the job actually processes.","solutions":["Regenerate the partition file with InputSampler.writePartitionFile(job, sampler) immediately before job.submit() on the same Job instance","Keep numReduceTasks and the sample count consistent: the file must hold exactly numReduceTasks-1 points","Point mapreduce.totalorderpartitioner.path (or TotalOrderPartitioner.setPartitionFile) at the freshly written file for this job"],"exampleFix":"// before: partition file from a previous run with 9 points\nJob job = Job.getInstance(conf);\njob.setNumReduceTasks(12);\n// after: sample for THIS job before submit\njob.setNumReduceTasks(12);\nInputSampler.writePartitionFile(job, new InputSampler.RandomSampler<Text>(0.1, 10000));","handlingStrategy":"validation","validationCode":"Path partFile = new Path(conf.get(\"mapreduce.totalorderpartitioner.path\", \"_partition.lst\"));\nlong expected = job.getNumReduceTasks() - 1;\ntry (FileSystem fs = partFile.getFileSystem(conf);\n     BufferedReader r = new BufferedReader(new InputStreamReader(fs.open(partFile), StandardCharsets.UTF_8))) {\n  long lines = r.lines().count();\n  if (lines != expected) {\n    throw new IOException(\"Partition file has \" + lines + \" points but job needs \" + expected);\n  }\n}","typeGuard":null,"tryCatchPattern":"try { // smoke-test partitioner setup before submit\n  TotalOrderPartitioner<Text, Text> p = new TotalOrderPartitioner<>();\n  p.setConf(job.getConfiguration());\n} catch (IllegalArgumentException | IOException e) {\n  throw new RuntimeException(\"TotalOrderPartitioner setup invalid: \" + e.getMessage(), e);\n}","preventionTips":["Sample and write the partition file in the same job right before submit","Treat numReduceTasks and the partition file as one atomic config decision","Never reuse _partition.lst across jobs with different reduce counts"],"tags":["mapreduce","partitioning","total-order-sort","sampling","configuration"],"backgroundTag":"partition-count-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}