What causes "modification={} != startModification = {}" in apache/hadoop?

Collect then mutate: gather elements to remove/add during iteration, apply after the loop (required - iterator has no remove()).

apache/hadoop · error · ConcurrentModificationException

modification={} != startModification = {}

Error message

modification={} != startModification = {}

What it means

LightWeightLinkedSet's iterator snapshots 'modification' as startModification when created; next() throws ConcurrentModificationException whenever any structural add/remove/clear happened since. Identical fail-fast semantics to LightWeightHashSet (3247), but note this iterator offers no remove() at all (3254), so the usual 'iterate and remove' workaround is not available here.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/LightWeightLinkedSet.java:253

  public Iterator<T> iterator() {
    return new LinkedSetIterator();
  }

  private class LinkedSetIterator implements Iterator<T> {
    /** The starting modification for fail-fast. */
    private final int startModification = modification;
    /** The next element to return. */
    private DoubleLinkedElement<T> next = head;

    @Override
    public boolean hasNext() {
      return next != null;
    }

    @Override
    public T next() {
      if (modification != startModification) {
        throw new ConcurrentModificationException("modification="
            + modification + " != startModification = " + startModification);
      }
      if (next == null) {
        throw new NoSuchElementException();
      }
      final T e = next.element;
      // find the next element
      next = next.after;
      return e;
    }

    @Override
    public void remove() {
      throw new UnsupportedOperationException("Remove is not supported.");
    }
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

Collect then mutate: gather elements to remove/add during iteration, apply after the loop (required - iterator has no remove()).
Iterate a copy: new ArrayList<>(set) or set.stream()... when mutation during traversal is expected.
Externalize synchronization or use ConcurrentHashMap.newKeySet() when threads genuinely race.

Example fix

// before
for (T e : orderedSet) {
  if (expired(e)) orderedSet.remove(e); // next() -> ConcurrentModificationException
}

// after - snapshot iteration (iterator here has no remove())
for (T e : new ArrayList<>(orderedSet)) {
  if (expired(e)) orderedSet.remove(e);
}

Defensive patterns

Strategy: fallback

Validate before calling

// this iterator has NO remove() - always iterate a snapshot when mutating
List<T> snapshot = new ArrayList<>(orderedSet);
for (T e : snapshot) { if (expired(e)) orderedSet.remove(e); }

Prevention

Defer all structural changes until after iteration on LightWeightLinkedSet.
Do not port 'iterate and it.remove()' patterns from LightWeightHashSet - unsupported here.
Synchronize or use concurrent collections when multiple threads touch the set.

When it happens

Trigger: for-each over a LightWeightLinkedSet while the same thread or another thread calls add/remove/clear on it - e.g., evicting expired entries inline.

Common situations: Expiry sweeps over insertion-ordered trackers; concurrent RPC threads mutating a shared set; re-entrant code adding during iteration.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a9847990986c1128. Report an issue: GitHub.