apache/hadoop · warning · IOException

the datanode {} failed to pass a file descriptor for the sha

Error message

the datanode {} failed to pass a file descriptor for the shared memory segment.

What it means

recvFileInputStreams returned successfully but the received ancillary data contained no file descriptor (fis[0] == null): the DataNode replied SUCCESS for the shared-memory segment yet never actually passed the fd over the domain socket. The client cannot map the segment and throws IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java:176

      final DataOutputStream out =
          new DataOutputStream(
              new BufferedOutputStream(peer.getOutputStream()));
      new Sender(out).requestShortCircuitShm(clientName);
      ShortCircuitShmResponseProto resp =
          ShortCircuitShmResponseProto.parseFrom(
            PBHelperClient.vintPrefixed(peer.getInputStream()));
      String error = resp.hasError() ? resp.getError() : "(unknown)";
      switch (resp.getStatus()) {
      case SUCCESS:
        DomainSocket sock = peer.getDomainSocket();
        byte buf[] = new byte[1];
        FileInputStream[] fis = new FileInputStream[1];
        if (sock.recvFileInputStreams(fis, buf, 0, buf.length) < 0) {
          throw new EOFException("got EOF while trying to transfer the " +
              "file descriptor for the shared memory segment.");
        }
        if (fis[0] == null) {
          throw new IOException("the datanode " + datanode + " failed to " +
              "pass a file descriptor for the shared memory segment.");
        }
        try {
          DfsClientShm shm =
              new DfsClientShm(PBHelperClient.convert(resp.getId()),
                  fis[0], this, peer);
          LOG.trace("{}: createNewShm: created {}", this, shm);
          return shm;
        } finally {
          try {
            fis[0].close();
          } catch (Throwable e) {
            LOG.debug("Exception in closing " + fis[0], e);
          }
        }
      case ERROR_UNSUPPORTED:
        // The DataNode just does not support short-circuit shared memory
        // access, and we should stop asking.

View on GitHub (pinned to 2add963021)

Solutions

1. Raise DataNode file-descriptor limits (ulimit -n) and check /dev/shm usage
2. Align client and DataNode Hadoop versions
3. Until fixed, set dfs.client.read.shortcircuit=false so reads use TCP

catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("failed to pass a file descriptor")) {
    // DN could not send the shm fd: disable short-circuit for this client
    conf.setBoolean("dfs.client.read.shortcircuit", false);
    recreateFileSystem();
  } else {
    throw e;
  }
}

When it happens

Trigger: DataNode-side failure to create or send the shm fd - descriptor limit exhausted, /dev/shm full, or a native library/protocol issue - occurring after the SUCCESS response was already sent.

Common situations: DataNode hitting ulimit -n ceilings; /dev/shm exhausted; client and DN Hadoop versions disagreeing on the short-circuit fd-passing protocol.

Related errors

• no SharedFileDescriptorFactory paths were configured.
• Unexpected blockChecksumType '%s', expecting COMPOSITE_CRC
• Although a UNIX domain socket path is configured as {domainS
• You cannot pass file descriptors over anything but a UNIX do
• Unknown status: ${status}, message: ${message}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8854cf8897bc6384. Report an issue: GitHub.