What causes "Attempt to remove non-existent val" in apache/hadoop?

Do not call remove() on TupleWritable's iterator — treat it as read-only; build a filtered copy of the tuple instead of mutating through the iterator

apache/hadoop · warning · IllegalStateException

Attempt to remove non-existent val

Error message

Attempt to remove non-existent val

What it means

TupleWritable.iterator() returns an Iterator<Writable> over set (written) positions backed by a shared BitSet. Its remove() (TupleWritable.java:132-138) clears the bit at the current bitIndex; when that bit is not set — concretely when bitIndex == -1 after iteration is exhausted — it throws IllegalStateException('Attempt to remove non-existent val'). This is an Iterator-contract violation: remove() must delete the element last returned by next(), but this implementation targets the *next* index, and after hasNext() becomes false it throws.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java:137

   * from this iterator.
   */
  public Iterator<Writable> iterator() {
    final TupleWritable t = this;
    return new Iterator<Writable>() {
      int bitIndex = written.nextSetBit(0);
      public boolean hasNext() {
        return bitIndex >= 0;
      }
      public Writable next() {
        int returnIndex = bitIndex;
        if (returnIndex < 0)
          throw new NoSuchElementException();
        bitIndex = written.nextSetBit(bitIndex+1);
        return t.get(returnIndex);
      }
      public void remove() {
        if (!written.get(bitIndex)) {
          throw new IllegalStateException(
            "Attempt to remove non-existent val");
        }
        written.clear(bitIndex);
      }
    };
  }

  /**
   * Convert Tuple to String as in the following.
   * <code>[&lt;child1&gt;,&lt;child2&gt;,...,&lt;childn&gt;]</code>
   */
  public String toString() {
    StringBuilder buf = new StringBuilder("[");
    for (int i = 0; i < values.length; ++i) {
      buf.append(has(i) ? values[i].toString() : "");
      buf.append(",");
    }
    if (values.length != 0)

View on GitHub (pinned to 2add963021)

Solutions

Do not call remove() on TupleWritable's iterator — treat it as read-only; build a filtered copy of the tuple instead of mutating through the iterator
If you must drop tuple positions, use the package API clearWritten(i) (same package) or rebuild a TupleWritable containing only the wanted positions
Call remove() at most once per next() and only while hasNext() was true before that next() — but prefer not calling it at all
Wrap consumption in a for-each loop (next()/hasNext() only), which cannot trigger remove()

Example fix

// before
Iterator<Writable> it = tuple.iterator();
while (it.hasNext()) { it.next(); it.remove(); } // exhausts then throws

// after
List<Writable> kept = new ArrayList<>();
for (Writable w : tuple) {           // read-only iteration
  if (shouldKeep(w)) kept.add(w);
}
// build a new tuple/collection from kept instead of mutating via iterator

Defensive patterns

Strategy: try-catch

Try / catch

Iterator<Writable> it = tuple.iterator();
while (it.hasNext()) {
  Writable w = it.next();
  // never call it.remove(); collect what to keep instead
}
// if third-party code may call remove():
try { it.remove(); } catch (IllegalStateException e) { /* iterator exhausted or already consumed — ignore/log */ }

Prevention

Treat TupleWritable's iterator as read-only
Build filtered copies instead of mutating via iterator.remove()
Never call remove() after hasNext() returns false or twice per next()

When it happens

Trigger: Calling iterator.remove() after the iterator is exhausted (bitIndex == -1 → written.get(-1) is false → throw); calling remove() twice in a row (first remove clears the next element's bit, second may throw); calling remove() before next() silently removes the *first* upcoming element rather than erroring — the same design flaw manifesting differently.

Common situations: User code reusing TupleWritable iterators in mappers/reducers that consume join output and try to filter elements; generic collection-processing utilities that call remove() in a loop; porting code from a List iterator where remove-before-next throws NoSuchElementException-like errors instead.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/49741dc6d35e2cd6. Report an issue: GitHub.