{"record":{"id":"202742076d80d51a","repo":"apache/hadoop","slug":"got-eof-while-trying-to-transfer-the-file-descript","errorCode":null,"errorMessage":"got EOF while trying to transfer the file descriptor for the shared memory segment.","messagePattern":"got EOF while trying to transfer the file descriptor for the shared memory segment\\.","errorType":"exception","errorClass":"EOFException","httpStatus":null,"severity":"warning","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java","lineNumber":172,"sourceCode":"     *                        itself (or the network) is the problem.\n     */\n    private DfsClientShm requestNewShm(String clientName, DomainPeer peer)\n        throws IOException {\n      final DataOutputStream out =\n          new DataOutputStream(\n              new BufferedOutputStream(peer.getOutputStream()));\n      new Sender(out).requestShortCircuitShm(clientName);\n      ShortCircuitShmResponseProto resp =\n          ShortCircuitShmResponseProto.parseFrom(\n            PBHelperClient.vintPrefixed(peer.getInputStream()));\n      String error = resp.hasError() ? resp.getError() : \"(unknown)\";\n      switch (resp.getStatus()) {\n      case SUCCESS:\n        DomainSocket sock = peer.getDomainSocket();\n        byte buf[] = new byte[1];\n        FileInputStream[] fis = new FileInputStream[1];\n        if (sock.recvFileInputStreams(fis, buf, 0, buf.length) < 0) {\n          throw new EOFException(\"got EOF while trying to transfer the \" +\n              \"file descriptor for the shared memory segment.\");\n        }\n        if (fis[0] == null) {\n          throw new IOException(\"the datanode \" + datanode + \" failed to \" +\n              \"pass a file descriptor for the shared memory segment.\");\n        }\n        try {\n          DfsClientShm shm =\n              new DfsClientShm(PBHelperClient.convert(resp.getId()),\n                  fis[0], this, peer);\n          LOG.trace(\"{}: createNewShm: created {}\", this, shm);\n          return shm;\n        } finally {\n          try {\n            fis[0].close();\n          } catch (Throwable e) {\n            LOG.debug(\"Exception in closing \" + fis[0], e);\n          }","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DfsClientShmManager.java#L154-L190","documentation":"During short-circuit shared-memory setup the client already received a SUCCESS ShortCircuitShmResponseProto, then called DomainSocket.recvFileInputStreams to receive the segment's file descriptor over the UNIX domain socket. A negative return is EOF: the DataNode closed the socket before or while passing the fd, so the shared-memory segment cannot be created and an EOFException is thrown.","triggerScenarios":"DataNode shuts down or restarts right after approving the requestShortCircuitShm request; the domain socket file is stale (DN recreated it); the DN's native short-circuit path aborts mid-handshake.","commonSituations":"Short-circuit local reads enabled with dfs.domain.socket.path while DataNodes restart under load; resource limits tearing down the socket; version skew between client and DN short-circuit code.","solutions":["Check DataNode liveness and its logs around the shm request","Ensure dfs.domain.socket.path is identical on DataNode and client and the socket file belongs to the running DN","As a workaround set dfs.client.read.shortcircuit=false - reads fall back to normal TCP and remain correct, only slower"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (EOFException e) {\n  // shm fd never arrived over the domain socket: fall back to TCP reads\n  conf.setBoolean(\"dfs.client.read.shortcircuit\", false);\n  // reconnect; correctness is preserved, only local-read performance is lost\n}","preventionTips":["Keep dfs.domain.socket.path identical on DataNode and clients","Treat short-circuit reads as an optimization - verify jobs run correctly with the feature disabled","Recreate the FileSystem after disabling short-circuit so settings take effect"],"tags":["hdfs","short-circuit-reads","unix-domain-socket","shared-memory"],"backgroundTag":"unix-domain-socket-eof","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}