{"record":{"id":"a642cf9610e5cf8f","repo":"apache/hadoop","slug":"size-of-shared-memory-segment-was-intsize-but-t","errorCode":null,"errorMessage":"size of shared memory segment was {intSize}, but that is not enough to hold even one slot.","messagePattern":"size of shared memory segment was (.+?), but that is not enough to hold even one slot\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java","lineNumber":82,"sourceCode":"    } catch (Throwable e) {\n      LOG.error(\"failed to load misc.Unsafe\", e);\n    }\n    return null;\n  }\n\n  /**\n   * Calculate the usable size of a shared memory segment.\n   * We round down to a multiple of the slot size and do some validation.\n   *\n   * @param stream The stream we're using.\n   * @return       The usable size of the shared memory segment.\n   */\n  private static int getUsableLength(FileInputStream stream)\n      throws IOException {\n    int intSize = Ints.checkedCast(stream.getChannel().size());\n    int slots = intSize / BYTES_PER_SLOT;\n    if (slots == 0) {\n      throw new IOException(\"size of shared memory segment was \" +\n          intSize + \", but that is not enough to hold even one slot.\");\n    }\n    return slots * BYTES_PER_SLOT;\n  }\n\n  /**\n   * Identifies a DfsClientShm.\n   */\n  public static class ShmId implements Comparable<ShmId> {\n    private static final Random random = new Random();\n    private final long hi;\n    private final long lo;\n\n    /**\n     * Generate a random ShmId.\n     *\n     * We generate ShmIds randomly to prevent a malicious client from\n     * successfully guessing one and using that to interfere with another","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java#L64-L100","documentation":"ShortCircuitShm mmaps the shared-memory segment that the datanode created to track short-circuit replica validity. getUsableLength() divides the segment file's byte size by BYTES_PER_SLOT (64 bytes) and rounds down; a file smaller than one slot is unusable and construction fails with an IOException. The segment lives under the datanode's dfs.datanode.shared.file.descriptor.paths (default /dev/shm and /tmp), so a sub-slot file means that directory is broken or the segment was truncated.","triggerScenarios":"new ShortCircuitShm(shmId, stream) - client-side inside DfsClientShmManager after the datanode hands over the shared file descriptor - where stream.getChannel().size() < 64: a zero-byte or truncated file in /dev/shm, an exhausted tmpfs, or a version mismatch in shm creation.","commonSituations":"Containers with tiny or missing /dev/shm limits on datanode hosts; datanode interrupted mid-creation of the segment; mixing Hadoop versions between client and datanode on the short-circuit path.","solutions":["On the datanode, verify the dirs in dfs.datanode.shared.file.descriptor.paths exist and are writable (df -h /dev/shm); raise the container shm-size limit if needed","Restart the datanode so ShortCircuitRegistry recreates its shared-memory segments, then retry short-circuit reads","Verify client and datanode run the same Hadoop release; the shm handshake is version-coupled","Workaround: set dfs.client.read.shortcircuit=false until the shm setup is healthy"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Before constructing a shm from a datanode-provided fd\nlong size = stream.getChannel().size();\nif (size < 64 /* BYTES_PER_SLOT */) {\n  // reject this segment and request a new one from the datanode\n}","typeGuard":null,"tryCatchPattern":"try {\n  // shm construction path\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"not enough to hold even one slot\")) {\n    // discard the fd; ask the datanode for a fresh segment\n  } else { throw e; }\n}","preventionTips":["Size /dev/shm adequately on datanode containers (e.g. docker --shm-size)","Monitor tmpfs usage on hosts listed in dfs.datanode.shared.file.descriptor.paths","Keep client and datanode versions matched when short-circuit reads are on"],"tags":["hdfs","short-circuit-read","shared-memory","dev-shm","datanode-config","containers"],"backgroundTag":"shared-memory-segment-too-small","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}