apache/hadoop · error · IOException

size of shared memory segment was

Error message

size of shared memory segment was {intSize}, but that is not enough to hold even one slot.

What it means

ShortCircuitShm mmaps the shared-memory segment that the datanode created to track short-circuit replica validity. getUsableLength() divides the segment file's byte size by BYTES_PER_SLOT (64 bytes) and rounds down; a file smaller than one slot is unusable and construction fails with an IOException. The segment lives under the datanode's dfs.datanode.shared.file.descriptor.paths (default /dev/shm and /tmp), so a sub-slot file means that directory is broken or the segment was truncated.

Solutions

On the datanode, verify the dirs in dfs.datanode.shared.file.descriptor.paths exist and are writable (df -h /dev/shm); raise the container shm-size limit if needed
Restart the datanode so ShortCircuitRegistry recreates its shared-memory segments, then retry short-circuit reads
Verify client and datanode run the same Hadoop release; the shm handshake is version-coupled
Workaround: set dfs.client.read.shortcircuit=false until the shm setup is healthy

Defensive patterns

Strategy: validation

Validate before calling

// Before constructing a shm from a datanode-provided fd
long size = stream.getChannel().size();
if (size < 64 /* BYTES_PER_SLOT */) {
  // reject this segment and request a new one from the datanode
}

Try / catch

try {
  // shm construction path
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("not enough to hold even one slot")) {
    // discard the fd; ask the datanode for a fresh segment
  } else { throw e; }
}

Prevention

Size /dev/shm adequately on datanode containers (e.g. docker --shm-size)
Monitor tmpfs usage on hosts listed in dfs.datanode.shared.file.descriptor.paths
Keep client and datanode versions matched when short-circuit reads are on

When it happens

Trigger: new ShortCircuitShm(shmId, stream) - client-side inside DfsClientShmManager after the datanode hands over the shared file descriptor - where stream.getChannel().size() < 64: a zero-byte or truncated file in /dev/shm, an exhausted tmpfs, or a version mismatch in shm creation.

Common situations: Containers with tiny or missing /dev/shm limits on datanode hosts; datanode interrupted mid-creation of the segment; mixing Hadoop versions between client and datanode on the short-circuit path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a642cf9610e5cf8f. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/ShortCircuitShm.java:82

    } catch (Throwable e) {
      LOG.error("failed to load misc.Unsafe", e);
    }
    return null;
  }

  /**
   * Calculate the usable size of a shared memory segment.
   * We round down to a multiple of the slot size and do some validation.
   *
   * @param stream The stream we're using.
   * @return       The usable size of the shared memory segment.
   */
  private static int getUsableLength(FileInputStream stream)
      throws IOException {
    int intSize = Ints.checkedCast(stream.getChannel().size());
    int slots = intSize / BYTES_PER_SLOT;
    if (slots == 0) {
      throw new IOException("size of shared memory segment was " +
          intSize + ", but that is not enough to hold even one slot.");
    }
    return slots * BYTES_PER_SLOT;
  }

  /**
   * Identifies a DfsClientShm.
   */
  public static class ShmId implements Comparable<ShmId> {
    private static final Random random = new Random();
    private final long hi;
    private final long lo;

    /**
     * Generate a random ShmId.
     *
     * We generate ShmIds randomly to prevent a malicious client from
     * successfully guessing one and using that to interfere with another

View on GitHub (pinned to 2add963021)