{"record":{"id":"5772a38eeeafb59e","repo":"apache/hadoop","slug":"missing-reduce-spec-5772a3","errorCode":null,"errorMessage":"Missing reduce spec","messagePattern":"Missing reduce spec","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java","lineNumber":176,"sourceCode":"          id += nReds;\n          context.write(key, NullWritable.get());\n        }\n      }\n    }\n\n  }\n\n  public static class SleepReducer\n  extends Reducer<GridmixKey, NullWritable, NullWritable, NullWritable> {\n\n    private long duration = 0L;\n\n    @Override\n    protected void setup(Context context)\n    throws IOException, InterruptedException {\n      if (!context.nextKey() ||\n        context.getCurrentKey().getType() != GridmixKey.REDUCE_SPEC) {\n        throw new IOException(\"Missing reduce spec\");\n      }\n      for (NullWritable ignored : context.getValues()) {\n        final GridmixKey spec = context.getCurrentKey();\n        duration += spec.getReduceOutputBytes();\n      }\n      long sleepInterval = \n        context.getConfiguration().getLong(GRIDMIX_SLEEP_INTERVAL, 5);\n      final long RINTERVAL = \n        TimeUnit.MILLISECONDS.convert(sleepInterval, TimeUnit.SECONDS);\n      //This is to stop accumulating deviation from expected sleep time\n      //over a period of time.\n      long start = Time.monotonicNow();\n      long slept = 0L;\n      long sleep = 0L;\n      while (slept < duration) {\n        final long rem = duration - slept;\n        sleep = Math.min(rem, RINTERVAL);\n        context.setStatus(\"Sleeping... \" + rem + \" ms left\");","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/SleepJob.java#L158-L194","documentation":"SleepJob.SleepReducer.setup() expects the first key delivered to the reducer to be a GridmixKey of type REDUCE_SPEC, which carries the reduce-output byte count used to size the simulated sleep. If context.nextKey() returns false or the first key's getType() is not GridmixKey.REDUCE_SPEC, it throws IOException('Missing reduce spec'). This is a data-contract check on the intermediate map-output format, not a user-config knob.","triggerScenarios":"Running gridmix's SleepJob where the reduce input stream lacks the leading REDUCE_SPEC record: a job story/trace with reduce tasks but missing reduce-output-bytes data feeding malformed GridmixKeys, a customized InputFormat or gridmix job factory that emits raw data keys without the spec record, or 0 maps with reduces>0 producing an empty shuffle.","commonSituations":"Replaying a truncated or 3rd-party trace whose reduce statistics are absent; patching SleepJob or GridmixKey serialization; version mismatches between a saved GridmixKey stream and the current SleepJob; generating sleep-job input outside GridmixJob.","solutions":["Replay a complete, well-formed job trace produced by a standard job-history parser so every reduce sees its REDUCE_SPEC GridmixKey","Revert customizations to SleepJob's input format or GridmixKey emission — the first record per reduce partition must be the REDUCE_SPEC key","If the trace genuinely has no reduce-output stats, run SleepJob in a mode/job with zero reduces or with -Dgridmix.sleep.fake-activities disabled rather than feeding reducers empty spec streams","Check for GridmixKey serialization version skew between the trace-generation code and the replaying gridmix build"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// inside a reducer wrapper around SleepReducer semantics\ntry {\n  super.setup(context);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"Missing reduce spec\")) {\n    // input stream lacks the REDUCE_SPEC GridmixKey; regenerate/repair the job input\n  } else throw e;\n}","preventionTips":["Replay only complete job traces with reduce statistics when using SleepJob","Do not replace GridmixKey emission when customizing SleepJob input formats","Keep trace-generation and replay on the same Hadoop/gridmix version"],"tags":["hadoop","gridmix","sleep-job","reducer","data-contract"],"backgroundTag":"invalid-input-data-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}