apache/hadoop · error · NoSuchElementException

iterate past last value

Error message

iterate past last value

What it means

ReduceValuesIterator (Task.ValuesIterator) is the old-API iterator over all values of the current reduce key, backed by the sorted merge stream. next() first checks the cached hasNext flag and throws NoSuchElementException if the iterator is exhausted. Calling next() without a preceding true from hasNext() — or continuing after the loop — triggers it.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java:1625

      this.keyDeserializer.open(keyIn);
      this.valDeserializer = serializationFactory.getDeserializer(valClass);
      this.valDeserializer.open(this.valueIn);
      readNextKey();
      key = nextKey;
      nextKey = null; // force new instance creation
      hasNext = more;
    }

    RawKeyValueIterator getRawIterator() { return in; }
    
    /// Iterator methods

    public boolean hasNext() { return hasNext; }

    private int ctr = 0;
    public VALUE next() {
      if (!hasNext) {
        throw new NoSuchElementException("iterate past last value");
      }
      try {
        readNextValue();
        readNextKey();
      } catch (IOException ie) {
        throw new RuntimeException("problem advancing post rec#"+ctr, ie);
      }
      reporter.progress();
      return value;
    }

    public void remove() { throw new RuntimeException("not implemented"); }

    /// Auxiliary methods

    /** Start processing next unique key. */
    public void nextKey() throws IOException {
      // read until we find a new key

View on GitHub (pinned to 2add963021)

Solutions

  1. Call next() only immediately after hasNext() returned true.
  2. Prefer the for-each form over the values Iterable where possible.
  3. For peek-ahead, keep a 'pending' variable and refill it from hasNext()/next() pairs instead of calling next() speculatively.

Example fix

// before
while (values.hasNext()) {
  VALUE v = values.next();
  // ...
  VALUE peek = values.next(); // may throw past the last value
}

// after
while (values.hasNext()) {
  VALUE v = values.next();
  // ... no speculative next(); use hasNext() before any extra read
}
Defensive patterns

Strategy: validation

Validate before calling

if (values.hasNext()) {
  VALUE v = values.next();
} else {
  // no more values for this key
}

Prevention

When it happens

Trigger: Calling next() twice per iteration inside reduce(), caching the iterator and reusing it after exhaustion, or hand-rolled while(true) loops that skip the hasNext() check on an edge case (empty value list for a key).

Common situations: Custom reduce logic that peeks ahead ('look at the next value') and forgets the boundary; secondary-sort patterns where user code steps the iterator manually; converting for-each loops to manual iteration.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6df6bf4a0f736774. Report an issue: GitHub.