apache/hadoop · error · IOException

Unable to map logical nameservice URI '{}' to a NameNode. Lo

Error message

Unable to map logical nameservice URI '{}' to a NameNode. Local configuration does not have a failover proxy provider configured.

What it means

To renew/cancel a delegation token whose service is a logical HA nameservice URI (e.g. hdfs://mycluster), DFSClient must build a failover proxy from local configuration. If the local config has no failover proxy provider for that nameservice (missing dfs.client.failover.proxy.provider.<ns> and dfs.ha.namenodes.<ns>), it cannot reach any NameNode and throws this IOException. The source comment cites exactly this MR scenario: a ResourceManager that lacks other clusters' HA config.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java:861

        nn.cancelDelegationToken(delToken);
      } catch (RemoteException re) {
        throw re.unwrapRemoteException(InvalidToken.class,
            AccessControlException.class);
      }
    }

    private static ClientProtocol getNNProxy(
        Token<DelegationTokenIdentifier> token, Configuration conf)
        throws IOException {
      URI uri = HAUtilClient.getServiceUriFromToken(
          HdfsConstants.HDFS_URI_SCHEME, token);
      if (HAUtilClient.isTokenForLogicalUri(token) &&
          !HAUtilClient.isLogicalUri(conf, uri)) {
        // If the token is for a logical nameservice, but the configuration
        // we have disagrees about that, we can't actually renew it.
        // This can be the case in MR, for example, if the RM doesn't
        // have all of the HA clusters configured in its configuration.
        throw new IOException("Unable to map logical nameservice URI '" +
            uri + "' to a NameNode. Local configuration does not have " +
            "a failover proxy provider configured.");
      }

      ProxyAndInfo<ClientProtocol> info =
          NameNodeProxiesClient.createProxyWithClientProtocol(conf, uri, null);
      assert info.getDelegationTokenService().equals(token.getService()) :
          "Returned service '" + info.getDelegationTokenService().toString() +
              "' doesn't match expected service '" +
              token.getService().toString() + "'";

      return info.getProxy();
    }

    @Override
    public boolean isManaged(Token<?> token) throws IOException {
      return true;
    }

View on GitHub (pinned to 2add963021)

Solutions

Add the missing nameservice's client HA config to the renewing component: dfs.ha.namenodes.<ns>, dfs.client.failover.proxy.provider.<ns> (ConfiguredFailoverProxyProvider), and dfs.namenode.rpc-address.<ns>.<nn> for each NameNode.
Distribute the union of all clusters' HDFS client configs to every host/service that may renew tokens (config management or YARN's config delivery).
Validate before renewing: parse the token's service URI host and check the provider key exists in conf; fail fast with a clear config-oriented message.
As a design alternative, let the RM renew with its own keytab instead of renewing borrowed tokens it cannot resolve.

Example fix

// before: token service is hdfs://ns2 but local config only knows ns1
client.renewDelegationToken(token);
// IOException: Unable to map logical nameservice URI 'hdfs://ns2' ...

// after: add to the renewing component's hdfs-site.xml
// <property><name>dfs.ha.namenodes.ns2</name><value>nn1,nn2</value></property>
// <property><name>dfs.client.failover.proxy.provider.ns2</name>
//   <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value></property>
// <property><name>dfs.namenode.rpc-address.ns2.nn1</name><value>host1:8020</value></property>
// <property><name>dfs.namenode.rpc-address.ns2.nn2</name><value>host2:8020</value></property>

Defensive patterns

Strategy: validation

Validate before calling

// fail fast if local config cannot resolve the token's logical nameservice
URI uri = HAUtilClient.getServiceUriFromToken(HdfsConstants.HDFS_URI_SCHEME, token);
if (HAUtilClient.isTokenForLogicalUri(token)) {
  String ns = uri.getHost();
  if (conf.get("dfs.client.failover.proxy.provider." + ns) == null) {
    throw new IOException("No failover proxy provider configured for nameservice " + ns);
  }
}

Try / catch

catch (IOException e) {
  if (e.getMessage().contains("failover proxy provider")) {
    // config gap: report which nameservice is missing HA entries instead of retrying
  }
  throw e;
}

Prevention

Ship the union of all HA clusters' client configs to every service that renews tokens.
Add a startup lint: for each known nameservice, assert dfs.ha.namenodes.<ns> and the provider key exist.
Prefer RM-side keytab renewal designs over renewing borrowed foreign tokens.

When it happens

Trigger: A component (YARN RM, MR job client, Oozie, History Server) renewing a token minted by cluster B while its own hdfs-site.xml only configures cluster A; distcp or federation setups where tokens cross cluster boundaries; minimal-config daemons handed foreign tokens.

Common situations: Multi-HA-cluster enterprises where RMs renew job tokens on behalf of jobs from other clusters; onboarding a new HA namespace without rolling its client config to all services; token-based workflows (Oozie coordinators) spanning trust domains.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0c4ed91327d92332. Report an issue: GitHub.