{"record":{"id":"7e54273ef7a59bd2","repo":"apache/hadoop","slug":"this-no-more-slots-are-available","errorCode":null,"errorMessage":"{this}: no more slots are available.","messagePattern":"(.+?): no more slots are available\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java","lineNumber":538,"sourceCode":"  private long calculateSlotAddress(int slotIdx) {\n    long offset = slotIdx;\n    offset *= BYTES_PER_SLOT;\n    return this.baseAddress + offset;\n  }\n\n  /**\n   * Allocate a new slot and register it.\n   *\n   * This function chooses an empty slot, initializes it, and then returns\n   * the relevant Slot object.\n   *\n   * @return    The new slot.\n   */\n  synchronized public final Slot allocAndRegisterSlot(\n      ExtendedBlockId blockId) {\n    int idx = allocatedSlots.nextClearBit(0);\n    if (idx >= slots.length) {\n      throw new RuntimeException(this + \": no more slots are available.\");\n    }\n    allocatedSlots.set(idx, true);\n    Slot slot = new Slot(calculateSlotAddress(idx), blockId);\n    slot.clear();\n    slot.makeValid();\n    slots[idx] = slot;\n    if (LOG.isTraceEnabled()) {\n      LOG.trace(this + \": allocAndRegisterSlot \" + idx + \": allocatedSlots=\" + allocatedSlots +\n                  StringUtils.getStackTrace(Thread.currentThread()));\n    }\n    return slot;\n  }\n\n  synchronized public final Slot getSlot(int slotIdx)\n      throws InvalidRequestException {\n    if (!allocatedSlots.get(slotIdx)) {\n      throw new InvalidRequestException(this + \": slot \" + slotIdx +\n          \" does not exist.\");","sourceCodeStart":520,"sourceCodeEnd":556,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java#L520-L556","documentation":"Each short-circuit replica occupies one 64-byte slot in the client's shm segment; allocAndRegisterSlot() picks the first clear bit in the allocatedSlots BitSet and throws RuntimeException '<shm>: no more slots are available.' when the segment is full. DfsClientShmManager is supposed to open a new segment before exhaustion, so hitting this throw usually indicates leaked replicas (slots never unref'd) or a sizing/race bug.","triggerScenarios":"ShortCircuitCache allocating a slot via allocAndRegisterSlot on a DfsClientShm whose allocatedSlots BitSet is already full - very high concurrency of pinned short-circuit replicas, replicas never released because streams were not closed, or the manager failing to notice the segment is full.","commonSituations":"Long-running client applications leaking HdfsDataInputStream handles with short-circuit reads enabled; readers holding mmap anchors well beyond cache capacity; heavy concurrent scans pinning more replicas than the segment has slots.","solutions":["Restart the client process - it drops the exhausted segment and gets a fresh shm from the datanode","Audit the application for unclosed HdfsDataInputStream / DFSInputStream (use try-with-resources); close returns the replica to ShortCircuitCache and frees its slot","Tune dfs.client.read.shortcircuit.streams.cache.size to bound how many replicas the cache pins","If reproducible, enable trace logging on org.apache.hadoop.hdfs.shortcircuit, capture jstack, and file a Hadoop JIRA; meanwhile set dfs.client.read.shortcircuit=false"],"exampleFix":"// before - stream never closed, slot leaks\nHdfsDataInputStream in = (HdfsDataInputStream) fs.open(path);\n// ... work ... (no close)\n\n// after - try-with-resources releases the replica and its slot\ntry (HdfsDataInputStream in = (HdfsDataInputStream) fs.open(path)) {\n  // ... work ...\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  // read path with dfs.client.read.shortcircuit=true\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().endsWith(\"no more slots are available.\")) {\n    // degrade: reopen the file with short-circuit reads disabled\n  } else { throw e; }\n}","preventionTips":["Always close HdfsDataInputStream promptly (try-with-resources) on short-circuit-enabled clusters","Size dfs.client.read.shortcircuit.streams.cache.size in line with expected concurrent reads","Treat this RuntimeException in load tests as an early signal of a stream leak"],"tags":["hdfs","short-circuit-read","shared-memory","slots","resource-exhaustion","stream-leak"],"backgroundTag":"resource-pool-exhausted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}