apache/hadoop · critical · RuntimeException

Although a UNIX domain socket path is configured as {domainS

Error message

Although a UNIX domain socket path is configured as {domainSocketPath}, we cannot start a localDataXceiverServer because {loadingFailureReason}

What it means

When dfs.domain.socket.path is configured, initDomainSocketPeerServer needs the native DomainSocket binding; DomainSocket.getLoadingFailureReason() is non-null exactly when the native Hadoop library (libhadoop.so) failed to load, and the DN then aborts startup with this RuntimeException rather than run a half-working short-circuit stack. This is about native library availability, not socket-path permissions (those fail later at bind time).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:1723

  private static DomainPeerServer getDomainPeerServer(Configuration conf,
      int port) throws IOException {
    String domainSocketPath =
        conf.getTrimmed(DFSConfigKeys.DFS_DOMAIN_SOCKET_PATH_KEY,
            DFSConfigKeys.DFS_DOMAIN_SOCKET_PATH_DEFAULT);
    if (domainSocketPath.isEmpty()) {
      if (conf.getBoolean(HdfsClientConfigKeys.Read.ShortCircuit.KEY,
            HdfsClientConfigKeys.Read.ShortCircuit.DEFAULT) &&
         (!conf.getBoolean(HdfsClientConfigKeys.DFS_CLIENT_USE_LEGACY_BLOCKREADERLOCAL,
          HdfsClientConfigKeys.DFS_CLIENT_USE_LEGACY_BLOCKREADERLOCAL_DEFAULT))) {
        LOG.warn("Although short-circuit local reads are configured, " +
            "they are disabled because you didn't configure {}",
            DFSConfigKeys.DFS_DOMAIN_SOCKET_PATH_KEY);
      }
      return null;
    }
    if (DomainSocket.getLoadingFailureReason() != null) {
      throw new RuntimeException("Although a UNIX domain socket " +
          "path is configured as " + domainSocketPath + ", we cannot " +
          "start a localDataXceiverServer because " +
          DomainSocket.getLoadingFailureReason());
    }
    DomainPeerServer domainPeerServer =
      new DomainPeerServer(domainSocketPath, port);
    int recvBufferSize = conf.getInt(
        DFSConfigKeys.DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_KEY,
        DFSConfigKeys.DFS_DATANODE_TRANSFER_SOCKET_RECV_BUFFER_SIZE_DEFAULT);
    if (recvBufferSize > 0) {
      domainPeerServer.setReceiveBufferSize(recvBufferSize);
    }
    return domainPeerServer;
  }
  
  // calls specific to BP
  public void notifyNamenodeReceivedBlock(ExtendedBlock block, String delHint,
      String storageUuid, boolean isOnTransientStorage) {

View on GitHub (pinned to 2add963021)

Solutions

Fix native support on the DN host: 'hadoop checknative -a' must report native:true; deploy matching libhadoop.so under $HADOOP_HOME/lib/native and set LD_LIBRARY_PATH accordingly
If short-circuit reads are not required, remove dfs.domain.socket.path from hdfs-site.xml — the DN then only logs a warning and skips the local xceiver server
After fixing, restart the DN and confirm the socket file appears: ls -l /var/lib/hadoop-hdfs/dn_socket

Example fix

<!-- before -->
<property><name>dfs.domain.socket.path</name><value>/var/lib/hadoop-hdfs/dn_socket</value></property>
<!-- after: no native libs, short-circuit reads not needed -->
<!-- property removed; set dfs.client.read.shortcircuit=false on clients -->

Defensive patterns

Strategy: validation

Validate before calling

String path = conf.get(DFSConfigKeys.DFS_DOMAIN_SOCKET_PATH_KEY, "");
if (!path.isEmpty() && DomainSocket.getLoadingFailureReason() != null) {
  throw new IllegalStateException("dfs.domain.socket.path set but native DomainSocket unavailable: "
      + DomainSocket.getLoadingFailureReason());
}

Try / catch

catch (RuntimeException e) {
  if (e.getMessage().contains("cannot start a localDataXceiverServer")) {
    // run hadoop checknative -a, fix libhadoop.so, or unset dfs.domain.socket.path and restart
  }
}

Prevention

Run 'hadoop checknative -a' on every DN host before enabling short-circuit reads
Bake libhadoop.so into deployment images and verify after OS/JDK upgrades
If native support is uncertain, leave dfs.domain.socket.path unset — the DN only warns

When it happens

Trigger: dfs.domain.socket.path set (for short-circuit reads / DomainPeerServer traffic) on a JVM where NativeCodeLoader failed: libhadoop.so missing, wrong architecture, glibc mismatch, LD_LIBRARY_PATH/JAVA_LIBRARY_PATH unset, or a platform without UNIX domain socket support (e.g. Windows).

Common situations: Short-circuit read rollouts where docs set dfs.domain.socket.path but native libs were never installed; slim Docker images stripping .so files; OS/JDK upgrades leaving stale native artifacts; hardened base images.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ac01b6fa9a9ee12b. Report an issue: GitHub.