apache/hadoop · critical · RuntimeException

Error in instantiating YarnClient

Error message

Error in instantiating YarnClient

What it means

Thrown from the YARNRunner constructor when FileContext.getFileContext(conf) raises UnsupportedFileSystemException — the URI scheme of the configured default filesystem (fs.defaultFS) has no FileSystem implementation available. It is wrapped in a RuntimeException with this message and fails the whole job-submission path because YARNRunner needs a FileContext to stage job files. This is a configuration/classpath problem, not a cluster-side one.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java:181

   this(conf, resMgrDelegate, new ClientCache(conf, resMgrDelegate));
  }

  /**
   * Similar to {@link YARNRunner#YARNRunner(Configuration, ResourceMgrDelegate)}
   * but allowing injecting {@link ClientCache}. Enable mocking and testing.
   * @param conf the configuration object
   * @param resMgrDelegate the resource manager delegate
   * @param clientCache the client cache object.
   */
  public YARNRunner(Configuration conf, ResourceMgrDelegate resMgrDelegate,
      ClientCache clientCache) {
    this.conf = conf;
    try {
      this.resMgrDelegate = resMgrDelegate;
      this.clientCache = clientCache;
      this.defaultFileContext = FileContext.getFileContext(this.conf);
    } catch (UnsupportedFileSystemException ufe) {
      throw new RuntimeException("Error in instantiating YarnClient", ufe);
    }
  }
  
  @Private
  /**
   * Used for testing mostly.
   * @param resMgrDelegate the resource manager delegate to set to.
   */
  public void setResourceMgrDelegate(ResourceMgrDelegate resMgrDelegate) {
    this.resMgrDelegate = resMgrDelegate;
  }
  
  @Override
  public void cancelDelegationToken(Token<DelegationTokenIdentifier> arg0)
      throws IOException, InterruptedException {
    throw new UnsupportedOperationException("Use Token.renew instead");
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.defaultFS correctly in core-site.xml (e.g. hdfs://namenode:8020) and confirm the file is on the client classpath
2. Verify the scheme resolves: hdfs fs -ls / (or a tiny FileContext.getFileContext(conf) smoke test) with the same classpath the job uses
3. Add the connector JARs for the scheme (hadoop-aws for s3a, hadoop-azure for abfs) to the client classpath
4. Check for scheme typos and that any viewfs mount table config is present

Example fix

<!-- before: scheme has no impl on classpath -->
<property><name>fs.defaultFS</name><value>abfs://container@acct.dfs.core.windows.net/</value></property>

<!-- after -->
<property><name>fs.defaultFS</name><value>hdfs://nn.example.com:8020</value></property>
<!-- plus ensure core-site.xml is on the client classpath -->

Configuration conf = new Configuration();
URI u = FileSystem.getDefaultUri(conf);
try {
  java.nio.file.Path impl = null; // smoke-test the scheme resolves
  org.apache.hadoop.fs.FileContext.getFileContext(conf);
} catch (org.apache.hadoop.fs.UnsupportedFileSystemException e) {
  throw new IllegalStateException("fs.defaultFS " + u
      + " has no filesystem implementation on the classpath", e);
}

When it happens

Trigger: Constructing YARNRunner (normally via JobClient/Cluster with mapreduce.framework.name=yarn) when fs.defaultFS is unset (falls back to file:/ semantics issues), malformed, or names a scheme (e.g. s3a:, abfs:, viewfs:) whose JARs are absent from the client classpath.

Common situations: fs.defaultFS still defaulting to file:/// because core-site.xml is not on the client classpath; Upgrading Hadoop where a scheme moved out of hadoop-common (e.g. s3n removal) and stale config points at it; Typo in the scheme: hdfs:// vs hdfs:/, or viewfs:// with missing mount-table config

Related errors

• dir + " is not a directory!"
• Could not locate MapReduce framework name '{}' in mapreduce.
• Couldn't open queue configuration at " + xmlInUrl
• Compression codec {} was not found.
• Could not connect to History server.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e6fdb2ac42a8e4e9. Report an issue: GitHub.