apache/hadoop · error · RuntimeException

problem advancing post rec#"+ctr

Error message

problem advancing post rec#"+ctr

What it means

Inside the reduce-side value iterator, next() advances the underlying merge stream via readNextValue()/readNextKey(); any IOException from that stream (corrupt spilled map output, local disk read failure, truncated intermediate data) is wrapped in this RuntimeException together with the record counter for context. The original IOException is attached as the cause. It means the task's intermediate data became unreadable mid-reduce.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:1631

      hasNext = more;
    }

    RawKeyValueIterator getRawIterator() { return in; }
    
    /// Iterator methods

    public boolean hasNext() { return hasNext; }

    private int ctr = 0;
    public VALUE next() {
      if (!hasNext) {
        throw new NoSuchElementException("iterate past last value");
      }
      try {
        readNextValue();
        readNextKey();
      } catch (IOException ie) {
        throw new RuntimeException("problem advancing post rec#"+ctr, ie);
      }
      reporter.progress();
      return value;
    }

    public void remove() { throw new RuntimeException("not implemented"); }

    /// Auxiliary methods

    /** Start processing next unique key. */
    public void nextKey() throws IOException {
      // read until we find a new key
      while (hasNext) { 
        readNextKey();
      }
      ++ctr;
      
      // move the next key to the current one

View on GitHub (pinned to 2add963021)

Solutions

1. Let the task attempt retry (default behavior) — transient IO hiccups usually clear on another attempt/node.
2. Pull the cause from the RuntimeException and the rec# counter to locate which segment/stream failed, then check that node's disks and mapreduce.cluster.local.dir space.
3. If one node repeats it, drain/blacklist the node or replace the failing disk.
4. Persistent failure on the same input points to corrupt map outputs — rerun the map phase attempts (job retry).

try {
  while (values.hasNext()) { consume(values.next()); }
} catch (RuntimeException e) {
  if (e.getCause() instanceof IOException) {
    throw (IOException) e.getCause(); // rethrow so the task attempt is retried
  }
  throw e;
}

When it happens

Trigger: A failing local disk while on-merge data is being read; corrupt or truncated map-output segments that passed fetch but fail during merge iteration; local dir cleanup races removing files still in use.

Common situations: Flaky disks on worker nodes; disk-full events during merge; tasks dying with this error and succeeding on retry elsewhere. The job-level effect is a failed task attempt and, if retries exhaust, a failed job.

Related errors

• getMapFinishTime() not supported for ReduceTask
• setMapFinishTime() not supported for ReduceTask
• length can't be negative
• Input only available on map
• iterate past last value

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5f0cd815f95ee3b4. Report an issue: GitHub.