{"record":{"id":"d5b176f90a1d6398","repo":"apache/hadoop","slug":"the-datanode-failed-to-pass-a-file-descriptor","errorCode":null,"errorMessage":"the datanode {} failed to pass a file descriptor (might have reached open file limit).","messagePattern":"the datanode (.+?) failed to pass a file descriptor \\(might have reached open file limit\\)\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java","lineNumber":607,"sourceCode":"    final DataOutputStream out =\n        new DataOutputStream(new BufferedOutputStream(peer.getOutputStream(), SMALL_BUFFER_SIZE));\n    SlotId slotId = slot == null ? null : slot.getSlotId();\n    new Sender(out).requestShortCircuitFds(block, token, slotId, 1,\n        failureInjector.getSupportsReceiptVerification());\n    DataInputStream in = new DataInputStream(peer.getInputStream());\n    BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(\n        PBHelperClient.vintPrefixed(in));\n    DomainSocket sock = peer.getDomainSocket();\n    failureInjector.injectRequestFileDescriptorsFailure();\n    switch (resp.getStatus()) {\n    case SUCCESS:\n      byte buf[] = new byte[1];\n      FileInputStream[] fis = new FileInputStream[2];\n      sock.recvFileInputStreams(fis, buf, 0, buf.length);\n      ShortCircuitReplica replica = null;\n      try {\n        if (fis[0] == null || fis[1] == null) {\n          throw new IOException(\"the datanode \" + datanode + \" failed to \" +\n              \"pass a file descriptor (might have reached open file limit).\");\n        }\n\n        ExtendedBlockId key =\n            new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId());\n        if (buf[0] == USE_RECEIPT_VERIFICATION.getNumber()) {\n          LOG.trace(\"Sending receipt verification byte for slot {}\", slot);\n          sock.getOutputStream().write(0);\n        }\n        replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,\n            Time.monotonicNow(), slot);\n        return new ShortCircuitReplicaInfo(replica);\n      } catch (IOException e) {\n        // This indicates an error reading from disk, or a format error.  Since\n        // it's not a socket communication problem, we return null rather than\n        // throwing an exception.\n        LOG.warn(\"{}: error creating ShortCircuitReplica.\", this, e);\n        return null;","sourceCodeStart":589,"sourceCodeEnd":625,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L589-L625","documentation":"With short-circuit reads over a UNIX domain socket, the datanode passes the block and metadata file descriptors to the client via SCM_RIGHTS. After a successful READ_BLOCK_SHORT_CIRCUIT response the client calls recvFileInputStreams; if either returned FileInputStream is null, the datanode failed to duplicate or pass the descriptor — classically because its open-file limit (ulimit -n) is exhausted.","triggerScenarios":"dfs.client.read.shortcircuit=true with dfs.domain.socket.path configured while the local datanode process is at or near its nofile limit — fd exhaustion from heavy scanner/balancer load, many short-circuit slots, or an fd leak.","commonSituations":"Datanodes with OS-default 64k fd limits under heavy directory scanner or DataNode-scanner load; fd counts creeping up over long uptime (leak); hosts where the systemd unit's LimitNOFILE is low; short-circuit traffic ramping up after a new cache tier.","solutions":["Raise the datanode's open-file limit (ulimit -n 131072 in hadoop-env.sh, or LimitNOFILE=131072 in the systemd unit) and restart the datanode.","On the datanode host, compare live fd usage with the limit: ls /proc/$(pidof datanode)/fd | wc -l versus ulimit -n; a steadily climbing count means a leak — restart the datanode to release descriptors.","As an immediate client-side mitigation set dfs.client.read.shortcircuit=false so reads use the network path.","Check datanode logs around the failure for other fd-related errors (Too many open files)."],"exampleFix":"# before: datanode service unit sets no fd cap -> inherits OS default (often 65536)\n# after: raise it, then reload and restart\n[Service]\nLimitNOFILE=131072\n# systemctl daemon-reload && systemctl restart hadoop-hdfs-datanode","handlingStrategy":"fallback","validationCode":"// pre-flight: check datanode fd headroom via JMX before enabling heavy\n// short-circuit workloads\n// GET http://<dn-host>:<dn-http-port>/jmx?qry=java.lang:type=OperatingSystem\n// compare OpenFileDescriptorCount vs MaxFileDescriptorCount;\n// treat > ~80% usage as unsafe for short-circuit traffic.","typeGuard":null,"tryCatchPattern":"try {\n  return readViaShortCircuit(dfs, path);\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"failed to pass a file descriptor\")) {\n    // datanode fd exhaustion: fall back to network reads for this job\n    conf.setBoolean(\"dfs.client.read.shortcircuit\", false);\n    return readViaNetwork(dfs, path);\n  }\n  throw e;\n}","preventionTips":["Set generous datanode nofile limits (LimitNOFILE / ulimit -n) as part of base host setup.","Monitor OpenFileDescriptorCount vs MaxFileDescriptorCount on every datanode and alert before exhaustion.","Track whether fd counts grow monotonically over weeks — that is a leak, restart during a window.","Know how to disable dfs.client.read.shortcircuit quickly as a client-side mitigation."],"tags":["hdfs","datanode","short-circuit-read","file-descriptors","ulimit","domain-socket"],"backgroundTag":"file-descriptor-limit-exhausted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}