{"record":{"id":"49741dc6d35e2cd6","repo":"apache/hadoop","slug":"attempt-to-remove-non-existent-val","errorCode":null,"errorMessage":"Attempt to remove non-existent val","messagePattern":"Attempt to remove non-existent val","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"warning","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java","lineNumber":137,"sourceCode":"   * from this iterator.\n   */\n  public Iterator<Writable> iterator() {\n    final TupleWritable t = this;\n    return new Iterator<Writable>() {\n      int bitIndex = written.nextSetBit(0);\n      public boolean hasNext() {\n        return bitIndex >= 0;\n      }\n      public Writable next() {\n        int returnIndex = bitIndex;\n        if (returnIndex < 0)\n          throw new NoSuchElementException();\n        bitIndex = written.nextSetBit(bitIndex+1);\n        return t.get(returnIndex);\n      }\n      public void remove() {\n        if (!written.get(bitIndex)) {\n          throw new IllegalStateException(\n            \"Attempt to remove non-existent val\");\n        }\n        written.clear(bitIndex);\n      }\n    };\n  }\n\n  /**\n   * Convert Tuple to String as in the following.\n   * <code>[&lt;child1&gt;,&lt;child2&gt;,...,&lt;childn&gt;]</code>\n   */\n  public String toString() {\n    StringBuilder buf = new StringBuilder(\"[\");\n    for (int i = 0; i < values.length; ++i) {\n      buf.append(has(i) ? values[i].toString() : \"\");\n      buf.append(\",\");\n    }\n    if (values.length != 0)","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/TupleWritable.java#L119-L155","documentation":"TupleWritable.iterator() returns an Iterator<Writable> over set (written) positions backed by a shared BitSet. Its remove() (TupleWritable.java:132-138) clears the bit at the current bitIndex; when that bit is not set — concretely when bitIndex == -1 after iteration is exhausted — it throws IllegalStateException('Attempt to remove non-existent val'). This is an Iterator-contract violation: remove() must delete the element last returned by next(), but this implementation targets the *next* index, and after hasNext() becomes false it throws.","triggerScenarios":"Calling iterator.remove() after the iterator is exhausted (bitIndex == -1 → written.get(-1) is false → throw); calling remove() twice in a row (first remove clears the next element's bit, second may throw); calling remove() before next() silently removes the *first* upcoming element rather than erroring — the same design flaw manifesting differently.","commonSituations":"User code reusing TupleWritable iterators in mappers/reducers that consume join output and try to filter elements; generic collection-processing utilities that call remove() in a loop; porting code from a List iterator where remove-before-next throws NoSuchElementException-like errors instead.","solutions":["Do not call remove() on TupleWritable's iterator — treat it as read-only; build a filtered copy of the tuple instead of mutating through the iterator","If you must drop tuple positions, use the package API clearWritten(i) (same package) or rebuild a TupleWritable containing only the wanted positions","Call remove() at most once per next() and only while hasNext() was true before that next() — but prefer not calling it at all","Wrap consumption in a for-each loop (next()/hasNext() only), which cannot trigger remove()"],"exampleFix":"// before\nIterator<Writable> it = tuple.iterator();\nwhile (it.hasNext()) { it.next(); it.remove(); } // exhausts then throws\n\n// after\nList<Writable> kept = new ArrayList<>();\nfor (Writable w : tuple) {           // read-only iteration\n  if (shouldKeep(w)) kept.add(w);\n}\n// build a new tuple/collection from kept instead of mutating via iterator","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"Iterator<Writable> it = tuple.iterator();\nwhile (it.hasNext()) {\n  Writable w = it.next();\n  // never call it.remove(); collect what to keep instead\n}\n// if third-party code may call remove():\ntry { it.remove(); } catch (IllegalStateException e) { /* iterator exhausted or already consumed — ignore/log */ }","preventionTips":["Treat TupleWritable's iterator as read-only","Build filtered copies instead of mutating via iterator.remove()","Never call remove() after hasNext() returns false or twice per next()"],"tags":["hadoop","mapreduce","join","iterator","mutable-state"],"backgroundTag":"iterator-remove-misuse","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}