apache/hadoop · critical · IllegalStateException

${name}: serial number map is full

Error message

${name}: serial number map is full

What it means

SerialNumberMap assigns each distinct value (user name, group name, xattr name) a serial starting at 1, bounded by max = 2^bitLength-1 where bitLength is the bit width of the field that stores the serial inside inodes/ACLs (USER/GROUP derive from PermissionStatusFormat and AclEntryStatusFormat.NAME; XATTR from XAttrFormat.NAME — roughly 65k entries; exact caps are logged at NN startup as '<NAME> serial map: bits=… maxEntries=…'). When a value not seen before would push the counter past max, the counter is rolled back and IllegalStateException '<name>: serial number map is full' is thrown. There is no eviction, so once full, any new distinct name of that kind fails.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java:70

  SerialNumberMap(String name, int bitLength) {
    this.name = name;
    this.max = (1 << bitLength) - 1;
  }

  public int get(T t) {
    if (t == null) {
      return 0;
    }
    Integer sn = t2i.get(t);
    if (sn == null) {
      synchronized (this) {
        sn = t2i.get(t);
        if (sn == null) {
          sn = current.getAndIncrement();
          if (sn > max) {
            current.getAndDecrement();
            throw new IllegalStateException(name + ": serial number map is full");
          }
          Integer old = t2i.putIfAbsent(t, sn);
          if (old != null) {
            current.getAndDecrement();
            return old;
          }
          i2t.put(sn, t);
        }
      }
    }
    return sn;
  }

  public T get(int i) {
    if (i == 0) {
      return null;
    }
    T t = i2t.get(i);

View on GitHub (pinned to 2add963021)

Solutions

1. Check the NameNode startup log lines 'USER/GROUP/XATTR serial map: bits=… maxEntries=…' to learn the actual caps
2. Stop introducing new distinct names: reuse existing users/groups (centralize ownership mapping) and cap the xattr-name vocabulary
3. If already full, the practical remedy is a fresh-format-and-reload of the namespace (oiv -> recreate) so serials are re-allocated densely, done in a maintenance window
4. Track distinct counts proactively (audit logs / oiv -p XML analysis) and alert before reaching maxEntries

// track distinct-name cardinality before you are near the cap
Set<String> distinctOwners = new HashSet<>();
// while generating ops:
if (distinctOwners.size() >= SAFE_LIMIT /* e.g. 60000 */) {
  owner = SHARED_SERVICE_USER; // reuse an existing identity
} else {
  distinctOwners.add(owner);
}
// then call hdfs.getClient().getProto().mkdir(..., owner, ...)

try {
  fs.setOwner(path, newUser, group);
} catch (RemoteException re) {
  if (re.getClassName().endsWith("IllegalStateException")
      && re.getMessage().contains("serial number map is full")) {
    // namespace hit the distinct user/group/xattr-name cap:
    // stop generating new names, reuse existing identities
  } else { throw re; }
}

When it happens

Trigger: A namespace operation that introduces one more distinct user/group/xattr name than the map can hold — e.g., mkdir/setOwner/setPermission with a 65,536th unique owner, setAcl/setXattr with a new name — calls SerialNumberManager.getSerialNumber -> SerialNumberMap.get(T) and hits sn > max.

Common situations: Multi-tenant clusters with unbounded per-user directories created programmatically; a job or script that generates unique group names per run; xattr names generated with timestamps/UUIDs. Existing names keep working; only new distinct names fail, and the error can recur on every subsequent new-name edit.

Related errors

• ${name}: serial number ${i} does not exist
• All negative block group IDs are used, growing into positive
• All positive block IDs are used, wrapping to negative IDs, w
• XAttrs are not supported on symlinks
• serial id ${id} > ${maxEntryNumber}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3703ead19c5a86d4. Report an issue: GitHub.