apache/hadoop · error · IOException

Missing reduce spec

Error message

Missing reduce spec

What it means

SleepJob.SleepReducer.setup() expects the first key delivered to the reducer to be a GridmixKey of type REDUCE_SPEC, which carries the reduce-output byte count used to size the simulated sleep. If context.nextKey() returns false or the first key's getType() is not GridmixKey.REDUCE_SPEC, it throws IOException('Missing reduce spec'). This is a data-contract check on the intermediate map-output format, not a user-config knob.

Source

Thrown at hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java:176

          id += nReds;
          context.write(key, NullWritable.get());
        }
      }
    }

  }

  public static class SleepReducer
  extends Reducer<GridmixKey, NullWritable, NullWritable, NullWritable> {

    private long duration = 0L;

    @Override
    protected void setup(Context context)
    throws IOException, InterruptedException {
      if (!context.nextKey() ||
        context.getCurrentKey().getType() != GridmixKey.REDUCE_SPEC) {
        throw new IOException("Missing reduce spec");
      }
      for (NullWritable ignored : context.getValues()) {
        final GridmixKey spec = context.getCurrentKey();
        duration += spec.getReduceOutputBytes();
      }
      long sleepInterval = 
        context.getConfiguration().getLong(GRIDMIX_SLEEP_INTERVAL, 5);
      final long RINTERVAL = 
        TimeUnit.MILLISECONDS.convert(sleepInterval, TimeUnit.SECONDS);
      //This is to stop accumulating deviation from expected sleep time
      //over a period of time.
      long start = Time.monotonicNow();
      long slept = 0L;
      long sleep = 0L;
      while (slept < duration) {
        final long rem = duration - slept;
        sleep = Math.min(rem, RINTERVAL);
        context.setStatus("Sleeping... " + rem + " ms left");

View on GitHub (pinned to 2add963021)

Solutions

1. Replay a complete, well-formed job trace produced by a standard job-history parser so every reduce sees its REDUCE_SPEC GridmixKey
2. Revert customizations to SleepJob's input format or GridmixKey emission — the first record per reduce partition must be the REDUCE_SPEC key
3. If the trace genuinely has no reduce-output stats, run SleepJob in a mode/job with zero reduces or with -Dgridmix.sleep.fake-activities disabled rather than feeding reducers empty spec streams
4. Check for GridmixKey serialization version skew between the trace-generation code and the replaying gridmix build

// inside a reducer wrapper around SleepReducer semantics
try {
  super.setup(context);
} catch (IOException e) {
  if (e.getMessage().contains("Missing reduce spec")) {
    // input stream lacks the REDUCE_SPEC GridmixKey; regenerate/repair the job input
  } else throw e;
}

When it happens

Trigger: Running gridmix's SleepJob where the reduce input stream lacks the leading REDUCE_SPEC record: a job story/trace with reduce tasks but missing reduce-output-bytes data feeding malformed GridmixKeys, a customized InputFormat or gridmix job factory that emits raw data keys without the spec record, or 0 maps with reduces>0 producing an empty shuffle.

Common situations: Replaying a truncated or 3rd-party trace whose reduce statistics are absent; patching SleepJob or GridmixKey serialization; version mismatches between a saved GridmixKey stream and the current SleepJob; generating sleep-job input outside GridmixJob.

Related errors

• Invalid gridmix.sleep.interval: {}
• File {} does not exist in pseudo local file system
• Append is not supported in pseudo local file system.
• Mkdirs is not supported in pseudo local file system.
• Rename is not supported in pseudo local file system.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5772a38eeeafb59e. Report an issue: GitHub.