apache/hadoop · error · RuntimeException

next value iterator failed

Error message

next value iterator failed

What it means

The same reset-mode path in ValueIterator.next(): while inReset, an IOException from backupStore.next(), nextValue(), or value deserialization while replaying marked values is wrapped as RuntimeException('next value iterator failed', e). It indicates the BackupStore of marked values failed mid-replay, not the normal shuffle input.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java:223

        try {
          if (backupStore.hasNext()) {
            backupStore.next();
            DataInputBuffer next = backupStore.nextValue();
            buffer.reset(next.getData(), next.getPosition(), next.getLength()
                - next.getPosition());
            value = valueDeserializer.deserialize(value);
            return value;
          } else {
            inReset = false;
            backupStore.exitResetMode();
            if (clearMarkFlag) {
              clearMarkFlag = false;
              isMarked = false;
            }
          }
        } catch (IOException e) {
          e.printStackTrace();
          throw new RuntimeException("next value iterator failed", e);
        }
      } 

      // if this is the first record, we don't need to advance
      if (firstValue) {
        firstValue = false;
        return value;
      }
      // if this isn't the first record and the next key is different, they
      // can't advance it here.
      if (!nextKeyIsSame) {
        throw new NoSuchElementException("iterate past last value");
      }
      // otherwise, go to the next key/value pair
      try {
        nextKeyValue();
        return value;
      } catch (IOException ie) {

View on GitHub (pinned to 2add963021)

Solutions

  1. Read the cause chain in the task log for the concrete IO error
  2. Verify NodeManager local dirs (space, device health); rely on task retry to move off a bad node
  3. Reduce per-key group size or drop the rewind pattern to avoid BackupStore entirely
Defensive patterns

Strategy: try-catch

Try / catch

try {
  while (values.hasNext()) { consume(values.next()); }
} catch (RuntimeException e) {
  if ("next value iterator failed".equals(e.getMessage()) && e.getCause() instanceof IOException) {
    log.error("BackupStore IO during reset-mode replay", e.getCause());
  }
  throw e;
}

Prevention

When it happens

Trigger: Reducer in reset mode after mark()/reset() and BackupStore throws IO reading its spill segments (local disk failure, spill file cleanup); deserialization of a stored value fails because the value Writable readFields raised IO.

Common situations: Same as other backup-store failures: two-pass reducers on nodes with unhealthy or full local disks; long-running reducers whose local dir was cleaned under them.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/504f5f75180204ee. Report an issue: GitHub.