apache/hadoop · error · RuntimeException

hasNext failed

Error message

hasNext failed

What it means

ReduceContextImpl.ValueIterator.hasNext(), when replaying marked values (inReset after mark()/reset()), delegates to BackupStore.hasNext(); any exception there is wrapped as RuntimeException('hasNext failed', e) after printStackTrace. The underlying failure — typically local-disk IO while reading the backup store's spill files — is in the cause chain.

Solutions

1. Inspect the wrapped cause in the task log to identify the real IO failure
2. Check NodeManager local directories for space and disk health; the framework retries the task on another node
3. Avoid mark()/reset() rewind for very large key groups — structure the algorithm as a single pass or use secondary sort

try {
  while (values.hasNext()) { consume(values.next()); }
} catch (RuntimeException e) {
  log.error("Value iteration failed (backup store): {}", e.getCause());
  throw e; // let the task fail and be retried on a healthy node
}

When it happens

Trigger: A reducer uses mark()/reset() to re-iterate a large value group, and during reset-mode iteration the BackupStore spill files under the task's local dirs cannot be read (disk full, failed disk, spill corruption).

Common situations: Reducers that rewind over huge value groups (per-key aggregation needing two passes) on nodes with tight NodeManager local-disk space; disks failing under mapreduce.cluster.local.dir.

Related errors

• next value iterator failed
• iterate past last value
• remove not implemented
• Reset called without a previous mark
• Attempt to remove non-existent val

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6df88115e93a25ba. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java:197

  
  BackupStore<KEYIN,VALUEIN> getBackupStore() {
    return backupStore;
  }
  
  protected class ValueIterator implements ReduceContext.ValueIterator<VALUEIN> {

    private boolean inReset = false;
    private boolean clearMarkFlag = false;

    @Override
    public boolean hasNext() {
      try {
        if (inReset && backupStore.hasNext()) {
          return true;
        } 
      } catch (Exception e) {
        e.printStackTrace();
        throw new RuntimeException("hasNext failed", e);
      }
      return firstValue || nextKeyIsSame;
    }

    @Override
    public VALUEIN next() {
      if (inReset) {
        try {
          if (backupStore.hasNext()) {
            backupStore.next();
            DataInputBuffer next = backupStore.nextValue();
            buffer.reset(next.getData(), next.getPosition(), next.getLength()
                - next.getPosition());
            value = valueDeserializer.deserialize(value);
            return value;
          } else {
            inReset = false;
            backupStore.exitResetMode();

View on GitHub (pinned to 2add963021)