apache/hadoop · warning · EOFException

got EOF while trying to transfer the file descriptor for the

Error message

got EOF while trying to transfer the file descriptor for the shared memory segment.

What it means

During short-circuit shared-memory setup the client already received a SUCCESS ShortCircuitShmResponseProto, then called DomainSocket.recvFileInputStreams to receive the segment's file descriptor over the UNIX domain socket. A negative return is EOF: the DataNode closed the socket before or while passing the fd, so the shared-memory segment cannot be created and an EOFException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java:172

     *                        itself (or the network) is the problem.
     */
    private DfsClientShm requestNewShm(String clientName, DomainPeer peer)
        throws IOException {
      final DataOutputStream out =
          new DataOutputStream(
              new BufferedOutputStream(peer.getOutputStream()));
      new Sender(out).requestShortCircuitShm(clientName);
      ShortCircuitShmResponseProto resp =
          ShortCircuitShmResponseProto.parseFrom(
            PBHelperClient.vintPrefixed(peer.getInputStream()));
      String error = resp.hasError() ? resp.getError() : "(unknown)";
      switch (resp.getStatus()) {
      case SUCCESS:
        DomainSocket sock = peer.getDomainSocket();
        byte buf[] = new byte[1];
        FileInputStream[] fis = new FileInputStream[1];
        if (sock.recvFileInputStreams(fis, buf, 0, buf.length) < 0) {
          throw new EOFException("got EOF while trying to transfer the " +
              "file descriptor for the shared memory segment.");
        }
        if (fis[0] == null) {
          throw new IOException("the datanode " + datanode + " failed to " +
              "pass a file descriptor for the shared memory segment.");
        }
        try {
          DfsClientShm shm =
              new DfsClientShm(PBHelperClient.convert(resp.getId()),
                  fis[0], this, peer);
          LOG.trace("{}: createNewShm: created {}", this, shm);
          return shm;
        } finally {
          try {
            fis[0].close();
          } catch (Throwable e) {
            LOG.debug("Exception in closing " + fis[0], e);
          }

View on GitHub (pinned to 2add963021)

Solutions

1. Check DataNode liveness and its logs around the shm request
2. Ensure dfs.domain.socket.path is identical on DataNode and client and the socket file belongs to the running DN
3. As a workaround set dfs.client.read.shortcircuit=false - reads fall back to normal TCP and remain correct, only slower

catch (EOFException e) {
  // shm fd never arrived over the domain socket: fall back to TCP reads
  conf.setBoolean("dfs.client.read.shortcircuit", false);
  // reconnect; correctness is preserved, only local-read performance is lost
}

When it happens

Trigger: DataNode shuts down or restarts right after approving the requestShortCircuitShm request; the domain socket file is stale (DN recreated it); the DN's native short-circuit path aborts mid-handshake.

Common situations: Short-circuit local reads enabled with dfs.domain.socket.path while DataNodes restart under load; resource limits tearing down the socket; version skew between client and DN short-circuit code.

Related errors

• no SharedFileDescriptorFactory paths were configured.
• You cannot pass file descriptors over anything but a UNIX do
• {feature} is enabled but dfs.domain.socket.path is not set.
• Error creating file descriptor in ${path}: ${e.getMessage()}
• Although a UNIX domain socket path is configured as {domainS

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/202742076d80d51a. Report an issue: GitHub.