apache/hadoop · critical · RuntimeException

Unable to bind on specified streaming port in secure context

Error message

Unable to bind on specified streaming port in secure context. Needed {}, got {}

What it means

In secure (Kerberos + jsvc) startup, SecureDataNodeStarter binds the DataNode streaming socket itself so it can run privileged. After ss.bind(streamingAddr) succeeds, it verifies the kernel actually gave it the port requested; if the local port differs (typical when the configured port is 0, so the OS assigned an ephemeral port), it throws this RuntimeException because secure data transfer requires a fixed, known, often privileged port.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java:142

        DFSConfigKeys.DFS_DATANODE_SOCKET_WRITE_TIMEOUT_KEY,
        HdfsConstants.WRITE_TIMEOUT);
    int backlogLength = conf.getInt(
        CommonConfigurationKeysPublic.IPC_SERVER_LISTEN_QUEUE_SIZE_KEY,
        CommonConfigurationKeysPublic.IPC_SERVER_LISTEN_QUEUE_SIZE_DEFAULT);

    ServerSocket ss = (socketWriteTimeout > 0) ? 
        ServerSocketChannel.open().socket() : new ServerSocket();
    try {
      ss.bind(streamingAddr, backlogLength);
    } catch (BindException e) {
      BindException newBe = appendMessageToBindException(e,
          streamingAddr.toString());
      throw newBe;
    }

    // Check that we got the port we need
    if (ss.getLocalPort() != streamingAddr.getPort()) {
      throw new RuntimeException(
          "Unable to bind on specified streaming port in secure "
              + "context. Needed " + streamingAddr.getPort() + ", got "
              + ss.getLocalPort());
    }
    isRpcPrivileged = SecurityUtil.isPrivilegedPort(ss.getLocalPort());
    System.err.println("Opened streaming server at " + streamingAddr);

    // Bind a port for the web server. The code intends to bind HTTP server to
    // privileged port only, as the client can authenticate the server using
    // certificates if they are communicating through SSL.
    final ServerSocketChannel httpChannel;
    if (policy.isHttpEnabled()) {
      httpChannel = ServerSocketChannel.open();
      InetSocketAddress infoSocAddr = DataNode.getInfoAddr(conf);
      try {
        httpChannel.socket().bind(infoSocAddr);
      } catch (BindException e) {
        BindException newBe = appendMessageToBindException(e,

View on GitHub (pinned to 2add963021)

Solutions

1. Set an explicit streaming port in hdfs-site.xml: dfs.datanode.address = 0.0.0.0:9866 (or your chosen fixed port), then restart the datanode under jsvc
2. Ensure that port is free and, if the deployment requires privileged ports, < 1024 so SecurityUtil.isPrivilegedPort() sees it as privileged
3. Keep dfs.datanode.address identical on all DNs so clients can address them consistently

Example fix

# before (hdfs-site.xml)
<property><name>dfs.datanode.address</name><value>0.0.0.0:0</value></property>

# after
<property><name>dfs.datanode.address</name><value>0.0.0.0:9866</value></property>

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import java.net.InetSocketAddress;
void assertFixedStreamingPort(Configuration conf) {
  InetSocketAddress a = InetSocketAddress.createUnresolved(
      conf.get(DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY,
               DFSConfigKeys.DFS_DATANODE_ADDRESS_DEFAULT),
      -1 /* placeholder */);
  // simpler: parse the port from the raw string
  String v = conf.get(DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY,
                      DFSConfigKeys.DFS_DATANODE_ADDRESS_DEFAULT);
  int port = Integer.parseInt(v.substring(v.lastIndexOf(':') + 1));
  if (port == 0) throw new IllegalStateException(
      "dfs.datanode.address must use a fixed port in secure mode: " + v);
}

When it happens

Trigger: jsvc secure startup with dfs.datanode.address resolving to port 0 (e.g. set to 0.0.0.0:0 or a hostname with :0), so bind() picks a random ephemeral port and ss.getLocalPort() != streamingAddr.getPort(). Also reachable with exotic socket/OS behavior, but port 0 is the practical trigger.

Common situations: Securing a cluster and enabling SASL data transfer (dfs.data.transfer.saslproperties.resolver.class) so the DN must start under jsvc, while dfs.datanode.address was left as/specified with port 0; port clashes that push admins to 'just use 0' on non-secure setups then enabling security; config drift after upgrade to 3.x default 9866.

Related errors

• Unable to bind on specified info port in secure context. Nee
• Security is enabled but block access tokens (via dfs.block.a
• Failed to create SecretManager
• Fetch of delegation token failed
• Cannot create a secured connection if DataNode listens on un

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fa623935a228b98d. Report an issue: GitHub.