apache/hadoop · error · RuntimeException

{this}: no more slots are available.

Error message

{this}: no more slots are available.

What it means

Each short-circuit replica occupies one 64-byte slot in the client's shm segment; allocAndRegisterSlot() picks the first clear bit in the allocatedSlots BitSet and throws RuntimeException '<shm>: no more slots are available.' when the segment is full. DfsClientShmManager is supposed to open a new segment before exhaustion, so hitting this throw usually indicates leaked replicas (slots never unref'd) or a sizing/race bug.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java:538

  private long calculateSlotAddress(int slotIdx) {
    long offset = slotIdx;
    offset *= BYTES_PER_SLOT;
    return this.baseAddress + offset;
  }

  /**
   * Allocate a new slot and register it.
   *
   * This function chooses an empty slot, initializes it, and then returns
   * the relevant Slot object.
   *
   * @return    The new slot.
   */
  synchronized public final Slot allocAndRegisterSlot(
      ExtendedBlockId blockId) {
    int idx = allocatedSlots.nextClearBit(0);
    if (idx >= slots.length) {
      throw new RuntimeException(this + ": no more slots are available.");
    }
    allocatedSlots.set(idx, true);
    Slot slot = new Slot(calculateSlotAddress(idx), blockId);
    slot.clear();
    slot.makeValid();
    slots[idx] = slot;
    if (LOG.isTraceEnabled()) {
      LOG.trace(this + ": allocAndRegisterSlot " + idx + ": allocatedSlots=" + allocatedSlots +
                  StringUtils.getStackTrace(Thread.currentThread()));
    }
    return slot;
  }

  synchronized public final Slot getSlot(int slotIdx)
      throws InvalidRequestException {
    if (!allocatedSlots.get(slotIdx)) {
      throw new InvalidRequestException(this + ": slot " + slotIdx +
          " does not exist.");

View on GitHub (pinned to 2add963021)

Solutions

1. Restart the client process - it drops the exhausted segment and gets a fresh shm from the datanode
2. Audit the application for unclosed HdfsDataInputStream / DFSInputStream (use try-with-resources); close returns the replica to ShortCircuitCache and frees its slot
3. Tune dfs.client.read.shortcircuit.streams.cache.size to bound how many replicas the cache pins
4. If reproducible, enable trace logging on org.apache.hadoop.hdfs.shortcircuit, capture jstack, and file a Hadoop JIRA; meanwhile set dfs.client.read.shortcircuit=false

Example fix

// before - stream never closed, slot leaks
HdfsDataInputStream in = (HdfsDataInputStream) fs.open(path);
// ... work ... (no close)

// after - try-with-resources releases the replica and its slot
try (HdfsDataInputStream in = (HdfsDataInputStream) fs.open(path)) {
  // ... work ...
}

try {
  // read path with dfs.client.read.shortcircuit=true
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().endsWith("no more slots are available.")) {
    // degrade: reopen the file with short-circuit reads disabled
  } else { throw e; }
}

When it happens

Trigger: ShortCircuitCache allocating a slot via allocAndRegisterSlot on a DfsClientShm whose allocatedSlots BitSet is already full - very high concurrency of pinned short-circuit replicas, replicas never released because streams were not closed, or the manager failing to notice the segment is full.

Common situations: Long-running client applications leaking HdfsDataInputStream handles with short-circuit reads enabled; readers holding mmap anchors well beyond cache capacity; heavy concurrent scans pinning more replicas than the segment has slots.

Related errors

• {this}: slot {slotIdx} does not exist.
• {this}: invalid negative slot index {slotIdx}
• {this}: invalid slot index {slotIdx}
• {this}: slot {slotIdx} is already in use.
• {this}: slot {slotIdx} is not marked as valid.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/7e54273ef7a59bd2. Report an issue: GitHub.