What causes "iterate past last value" in apache/hadoop?

Guard every next() with hasNext() evaluated in the same iteration

apache/hadoop · error · NoSuchElementException

iterate past last value

Error message

iterate past last value

What it means

ValueIterator.next() throws NoSuchElementException when firstValue is false and nextKeyIsSame is false — every value of the current key has been consumed. This is the standard java.util.Iterator contract firing on user reducer code that calls next() without a preceding hasNext().

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/ReduceContextImpl.java:235

              clearMarkFlag = false;
              isMarked = false;
            }
          }
        } catch (IOException e) {
          e.printStackTrace();
          throw new RuntimeException("next value iterator failed", e);
        }
      } 

      // if this is the first record, we don't need to advance
      if (firstValue) {
        firstValue = false;
        return value;
      }
      // if this isn't the first record and the next key is different, they
      // can't advance it here.
      if (!nextKeyIsSame) {
        throw new NoSuchElementException("iterate past last value");
      }
      // otherwise, go to the next key/value pair
      try {
        nextKeyValue();
        return value;
      } catch (IOException ie) {
        throw new RuntimeException("next value iterator failed", ie);
      } catch (InterruptedException ie) {
        // this is bad, but we can't modify the exception list of java.util
        throw new RuntimeException("next value iterator interrupted", ie);        
      }
    }

    @Override
    public void remove() {
      throw new UnsupportedOperationException("remove not implemented");
    }

View on GitHub (pinned to 2add963021)

Solutions

Guard every next() with hasNext() evaluated in the same iteration
Use the enhanced for loop over context.values(), which enforces the contract
Never cache or re-enter the values iterator after the loop body exits

Example fix

// before
VALUEIN v = values.next(); // called once too many after the loop
// after
while (values.hasNext()) {
  v = values.next();
}

Defensive patterns

Strategy: validation

Validate before calling

while (values.hasNext()) {
  VALUEIN v = values.next(); // next() only ever guarded by hasNext()
}

Try / catch

try {
  v = values.next();
} catch (NoSuchElementException e) {
  throw new IllegalStateException("Values iterator exhausted for key " + key, e);
}

Prevention

Use for-each over context.values() instead of manual iterator handling
Never call next() twice per hasNext() check
Do not cache the values iterator past the loop

When it happens

Trigger: One extra values.next() after the loop; calling next() twice per a single hasNext(); caching the values iterator in the reducer and consuming it again after the group ended.

Common situations: Reducers that peek at the next value manually; adapting the iterator to a streaming API that calls next() eagerly; copy-paste loop constructs like do { next(); } while (hasNext()).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c4a1c805f375edde. Report an issue: GitHub.