apache/hadoop · error · IOException

the datanode {} failed to pass a file descriptor (might have

Error message

the datanode {} failed to pass a file descriptor (might have reached open file limit).

What it means

With short-circuit reads over a UNIX domain socket, the datanode passes the block and metadata file descriptors to the client via SCM_RIGHTS. After a successful READ_BLOCK_SHORT_CIRCUIT response the client calls recvFileInputStreams; if either returned FileInputStream is null, the datanode failed to duplicate or pass the descriptor — classically because its open-file limit (ulimit -n) is exhausted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java:607

    final DataOutputStream out =
        new DataOutputStream(new BufferedOutputStream(peer.getOutputStream(), SMALL_BUFFER_SIZE));
    SlotId slotId = slot == null ? null : slot.getSlotId();
    new Sender(out).requestShortCircuitFds(block, token, slotId, 1,
        failureInjector.getSupportsReceiptVerification());
    DataInputStream in = new DataInputStream(peer.getInputStream());
    BlockOpResponseProto resp = BlockOpResponseProto.parseFrom(
        PBHelperClient.vintPrefixed(in));
    DomainSocket sock = peer.getDomainSocket();
    failureInjector.injectRequestFileDescriptorsFailure();
    switch (resp.getStatus()) {
    case SUCCESS:
      byte buf[] = new byte[1];
      FileInputStream[] fis = new FileInputStream[2];
      sock.recvFileInputStreams(fis, buf, 0, buf.length);
      ShortCircuitReplica replica = null;
      try {
        if (fis[0] == null || fis[1] == null) {
          throw new IOException("the datanode " + datanode + " failed to " +
              "pass a file descriptor (might have reached open file limit).");
        }

        ExtendedBlockId key =
            new ExtendedBlockId(block.getBlockId(), block.getBlockPoolId());
        if (buf[0] == USE_RECEIPT_VERIFICATION.getNumber()) {
          LOG.trace("Sending receipt verification byte for slot {}", slot);
          sock.getOutputStream().write(0);
        }
        replica = new ShortCircuitReplica(key, fis[0], fis[1], cache,
            Time.monotonicNow(), slot);
        return new ShortCircuitReplicaInfo(replica);
      } catch (IOException e) {
        // This indicates an error reading from disk, or a format error.  Since
        // it's not a socket communication problem, we return null rather than
        // throwing an exception.
        LOG.warn("{}: error creating ShortCircuitReplica.", this, e);
        return null;

View on GitHub (pinned to 2add963021)

Solutions

Raise the datanode's open-file limit (ulimit -n 131072 in hadoop-env.sh, or LimitNOFILE=131072 in the systemd unit) and restart the datanode.
On the datanode host, compare live fd usage with the limit: ls /proc/$(pidof datanode)/fd | wc -l versus ulimit -n; a steadily climbing count means a leak — restart the datanode to release descriptors.
As an immediate client-side mitigation set dfs.client.read.shortcircuit=false so reads use the network path.
Check datanode logs around the failure for other fd-related errors (Too many open files).

Example fix

# before: datanode service unit sets no fd cap -> inherits OS default (often 65536)
# after: raise it, then reload and restart
[Service]
LimitNOFILE=131072
# systemctl daemon-reload && systemctl restart hadoop-hdfs-datanode

Defensive patterns

Strategy: fallback

Validate before calling

// pre-flight: check datanode fd headroom via JMX before enabling heavy
// short-circuit workloads
// GET http://<dn-host>:<dn-http-port>/jmx?qry=java.lang:type=OperatingSystem
// compare OpenFileDescriptorCount vs MaxFileDescriptorCount;
// treat > ~80% usage as unsafe for short-circuit traffic.

Try / catch

try {
  return readViaShortCircuit(dfs, path);
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("failed to pass a file descriptor")) {
    // datanode fd exhaustion: fall back to network reads for this job
    conf.setBoolean("dfs.client.read.shortcircuit", false);
    return readViaNetwork(dfs, path);
  }
  throw e;
}

Prevention

Set generous datanode nofile limits (LimitNOFILE / ulimit -n) as part of base host setup.
Monitor OpenFileDescriptorCount vs MaxFileDescriptorCount on every datanode and alert before exhaustion.
Track whether fd counts grow monotonically over weeks — that is a leak, restart during a window.
Know how to disable dfs.client.read.shortcircuit quickly as a client-side mitigation.

When it happens

Trigger: dfs.client.read.shortcircuit=true with dfs.domain.socket.path configured while the local datanode process is at or near its nofile limit — fd exhaustion from heavy scanner/balancer load, many short-circuit slots, or an fd leak.

Common situations: Datanodes with OS-default 64k fd limits under heavy directory scanner or DataNode-scanner load; fd counts creeping up over long uptime (leak); hosts where the systemd unit's LimitNOFILE is low; short-circuit traffic ramping up after a new cache tier.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d5b176f90a1d6398. Report an issue: GitHub.