apache/hadoop · error · ConnectException

Localhost targeted connection resulted in a loopback. No dae

Error message

Localhost targeted connection resulted in a loopback. No daemon is listening on the target port.

What it means

ConnectException from NetUtils.connect when a connection to a local endpoint 'succeeded' but the local port equals the remote port and the local address equals the remote address — the rare TCP self-connect permitted by the RFC, possible only when the OS hands you an ephemeral source port identical to the (dead) destination port. Hadoop detects the loopback, closes the socket, and reports it as connection refused because the target daemon cannot have been listening. The log line 'Detected a loopback TCP socket, disconnecting it' precedes the throw.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java:631

        SocketIOWithTimeout.connect(ch, endpoint, timeout);
      }
    } catch (SocketTimeoutException ste) {
      throw new ConnectTimeoutException(ste.getMessage());
    }  catch (UnresolvedAddressException uae) {
      throw new UnknownHostException(endpoint.toString());
    }

    // There is a very rare case allowed by the TCP specification, such that
    // if we are trying to connect to an endpoint on the local machine,
    // and we end up choosing an ephemeral port equal to the destination port,
    // we will actually end up getting connected to ourself (ie any data we
    // send just comes right back). This is only possible if the target
    // daemon is down, so we'll treat it like connection refused.
    if (socket.getLocalPort() == socket.getPort() &&
        socket.getLocalAddress().equals(socket.getInetAddress())) {
      LOG.info("Detected a loopback TCP socket, disconnecting it");
      socket.close();
      throw new ConnectException(
        "Localhost targeted connection resulted in a loopback. " +
        "No daemon is listening on the target port.");
    }
  }
  
  /** 
   * Given a string representation of a host, return its ip address
   * in textual presentation.
   * 
   * @param name a string representation of a host:
   *             either a textual representation its IP address or its host name
   * @return its IP address in the string format
   */
  public static String normalizeHostName(String name) {
    try {
      return InetAddress.getByName(name).getHostAddress();
    } catch (UnknownHostException e) {
      return name;

View on GitHub (pinned to 2add963021)

Solutions

1. Treat it exactly like connection refused: retry the connection with backoff — the ephemeral collision is transient
2. Verify the target daemon is actually listening: 'ss -ltnp | grep <port>'
3. In tests, drain/wait on the port between stop and start, or use SO_REUSEADDR-style hygiene, and shrink the ephemeral window collisions by not reusing the same port instantly

Example fix

// before
NetUtils.connect(socket, addr, timeout); // rare: ConnectException "Localhost targeted connection resulted in a loopback" -> test flakes

// after
for (int i = 0; i < 3; i++) {
  try { NetUtils.connect(socket, addr, timeout); break; }
  catch (ConnectException e) { /* loopback self-connect == refused: retry */ sleep(backoff); }
}

// Cannot be pre-validated deterministically (OS-dependent ephemeral pick),
// but you can check the daemon is listening first:
// ss -ltn | grep <port>  (shell-level)

catch (ConnectException e) { /* loopback self-connect == refused */ wait(backoff); retry connect with a fresh socket; after N attempts, surface 'daemon not listening on <port>'; }

When it happens

Trigger: Connecting to localhost:<port> while no daemon listens on <port> and the OS happens to allocate <port> as the ephemeral source port (probability ~1/ephemeral-range per attempt); repeated rapid stop/start of daemons in tests increases the chance.

Common situations: Integration tests that kill and restart NameNode/DataNode/ResourceManager in tight loops; a daemon crashing between the client's port check and its connect; flaky CI where the failure appears as 'connection refused' to a port the test just closed.

Related errors

• Illegal argument for connect()
• Bad format: {}
• src.toString() + ": No such file or directory"
• Range starts beyond the file length ({l}): {last}
• getXAttrs on path `{}' is not within a mount point

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1ca8cc283419ba57. Report an issue: GitHub.