apache/hadoop · critical · IllegalStateException

All negative block group IDs are used, growing into positive

Error message

All negative block group IDs are used, growing into positive IDs, which might conflict with non-erasure coded blocks.

What it means

SequentialBlockGroupIdGenerator allocates erasure-coded block group IDs as negative longs in strides of MAX_BLOCKS_IN_GROUP, skipping ranges that collide with stored random block IDs (hasValidBlockInRange). If the generator's current value reaches >= 0, the entire negative ID space is consumed, and it throws IllegalStateException because positive IDs are reserved for replicated blocks. The current value is persisted with the namespace (NumberGenerator journaling), so the exhausted state survives restarts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/SequentialBlockGroupIdGenerator.java:63

  private final BlockManager blockManager;

  SequentialBlockGroupIdGenerator(BlockManager blockManagerRef) {
    super(Long.MIN_VALUE);
    this.blockManager = blockManagerRef;
  }

  @Override // NumberGenerator
  public long nextValue() {
    skipTo((getCurrentValue() & ~BLOCK_GROUP_INDEX_MASK) + MAX_BLOCKS_IN_GROUP);
    // Make sure there's no conflict with existing random block IDs
    final Block b = new Block(getCurrentValue());
    while (hasValidBlockInRange(b)) {
      skipTo(getCurrentValue() + MAX_BLOCKS_IN_GROUP);
      b.setBlockId(getCurrentValue());
    }
    if (b.getBlockId() >= 0) {
      throw new IllegalStateException("All negative block group IDs are used, "
          + "growing into positive IDs, "
          + "which might conflict with non-erasure coded blocks.");
    }
    return getCurrentValue();
  }

  /**
   * @param b A block object whose id is set to the starting point for check
   * @return true if any ID in the range
   *      {id, id+HdfsConstants.MAX_BLOCKS_IN_GROUP} is pointed-to by a stored
   *      block.
   */
  private boolean hasValidBlockInRange(Block b) {
    final long id = b.getBlockId();
    for (int i = 0; i < MAX_BLOCKS_IN_GROUP; i++) {
      b.setBlockId(id + i);
      if (blockManager.getStoredBlock(b) != null) {
        return true;

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect the persisted block group generation value in the fsimage/edits for corruption and restore the namespace from the last clean checkpoint
2. If the ID space is legitimately exhausted, rebuild the namespace (distcp data out, reformat, distcp back) — there is no runtime flag to extend the negative range
3. Report the case to the Hadoop community, since no first-class recovery path exists for this state

try (FSDataOutputStream out = fs.create(new Path("/ec/bigfile"), EcPolicy...)) {
  write(out);
} catch (RemoteException re) {
  if (re.getClassName().endsWith("IllegalStateException")
      && String.valueOf(re.getMessage()).contains("block group IDs")) {
    alertOps("EC block-group ID space exhausted on NameNode");
    writeToReplicatedPathInstead(); // fallback policy
  } else throw re;
}

When it happens

Trigger: Creating erasure-coded files until the sequential group counter walks from negative territory up to 0 — on the order of 2^63 / MAX_BLOCKS_IN_GROUP group allocations — or a corrupted/manually edited persisted 'blockGroupGeneration' counter that starts near zero. The throw happens inside nextValue() during EC block allocation on the NameNode.

Common situations: Practically unreachable through normal workloads; realistically seen only with tampered fsimage/edits generation counters or synthetic ID-exhaustion tests on long-lived test clusters.

Related errors

• All positive block IDs are used, wrapping to negative IDs, w
• ${name}: serial number map is full
• ${name}: serial number ${i} does not exist
• Block IDs mismatched: block${i}=${blocki}, expected block gr
• FileSystem ${item.fs.getUri()} does not support Erasure Codi

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ba529a5d3319b56b. Report an issue: GitHub.