apache/hadoop · critical · IllegalStateException

: serial number map is full

Error message

${name}: serial number map is full

What it means

SerialNumberMap assigns each distinct value (user name, group name, xattr name) a serial starting at 1, bounded by max = 2^bitLength-1 where bitLength is the bit width of the field that stores the serial inside inodes/ACLs (USER/GROUP derive from PermissionStatusFormat and AclEntryStatusFormat.NAME; XATTR from XAttrFormat.NAME — roughly 65k entries; exact caps are logged at NN startup as '<NAME> serial map: bits=… maxEntries=…'). When a value not seen before would push the counter past max, the counter is rolled back and IllegalStateException '<name>: serial number map is full' is thrown. There is no eviction, so once full, any new distinct name of that kind fails.

Solutions

1. Check the NameNode startup log lines 'USER/GROUP/XATTR serial map: bits=… maxEntries=…' to learn the actual caps
2. Stop introducing new distinct names: reuse existing users/groups (centralize ownership mapping) and cap the xattr-name vocabulary
3. If already full, the practical remedy is a fresh-format-and-reload of the namespace (oiv -> recreate) so serials are re-allocated densely, done in a maintenance window
4. Track distinct counts proactively (audit logs / oiv -p XML analysis) and alert before reaching maxEntries

// track distinct-name cardinality before you are near the cap
Set<String> distinctOwners = new HashSet<>();
// while generating ops:
if (distinctOwners.size() >= SAFE_LIMIT /* e.g. 60000 */) {
  owner = SHARED_SERVICE_USER; // reuse an existing identity
} else {
  distinctOwners.add(owner);
}
// then call hdfs.getClient().getProto().mkdir(..., owner, ...)

try {
  fs.setOwner(path, newUser, group);
} catch (RemoteException re) {
  if (re.getClassName().endsWith("IllegalStateException")
      && re.getMessage().contains("serial number map is full")) {
    // namespace hit the distinct user/group/xattr-name cap:
    // stop generating new names, reuse existing identities
  } else { throw re; }
}

When it happens

Trigger: A namespace operation that introduces one more distinct user/group/xattr name than the map can hold — e.g., mkdir/setOwner/setPermission with a 65,536th unique owner, setAcl/setXattr with a new name — calls SerialNumberManager.getSerialNumber -> SerialNumberMap.get(T) and hits sn > max.

Common situations: Multi-tenant clusters with unbounded per-user directories created programmatically; a job or script that generates unique group names per run; xattr names generated with timestamps/UUIDs. Existing names keep working; only new distinct names fail, and the error can recur on every subsequent new-name edit.

Related errors

• : serial number does not exist
• All negative block group IDs are used, growing into…
• All positive block IDs are used, wrapping to negative IDs…
• serial id >
• XAttrs are not supported on symlinks

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3703ead19c5a86d4. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java:70

  SerialNumberMap(String name, int bitLength) {
    this.name = name;
    this.max = (1 << bitLength) - 1;
  }

  public int get(T t) {
    if (t == null) {
      return 0;
    }
    Integer sn = t2i.get(t);
    if (sn == null) {
      synchronized (this) {
        sn = t2i.get(t);
        if (sn == null) {
          sn = current.getAndIncrement();
          if (sn > max) {
            current.getAndDecrement();
            throw new IllegalStateException(name + ": serial number map is full");
          }
          Integer old = t2i.putIfAbsent(t, sn);
          if (old != null) {
            current.getAndDecrement();
            return old;
          }
          i2t.put(sn, t);
        }
      }
    }
    return sn;
  }

  public T get(int i) {
    if (i == 0) {
      return null;
    }
    T t = i2t.get(i);

View on GitHub (pinned to 2add963021)