apache/hadoop · error · IOException

From option %s %s

Error message

From option %s %s

What it means

While creating the ObsClient, DefaultOBSClientFactory does conf.getClass("fs.obs.credentials.provider", null). If looking up that option throws a RuntimeException (classically ClassNotFoundException/RuntimeException from RunJar for a misspelled or missing class), it is wrapped in IOException('From option fs.obs.credentials.provider <cause>') with the cause preserved. The message therefore names the config key and embeds the underlying reason — read the tail of the message for the real class-loading error.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/DefaultOBSClientFactory.java:234

   * @param conf    Hadoop configuration
   * @param obsConf ObsConfiguration
   * @param name    URL
   * @return ObsClient client
   * @throws IOException on any failure to create Huawei OBS client
   */
  private static ObsClient createHuaweiObsClient(final Configuration conf,
      final ObsConfiguration obsConf, final URI name)
      throws IOException {
    Class<?> credentialsProviderClass;
    BasicSessionCredential credentialsProvider;
    ObsClient obsClient;

    try {
      credentialsProviderClass = conf.getClass(
          OBSConstants.OBS_CREDENTIALS_PROVIDER, null);
    } catch (RuntimeException e) {
      Throwable c = e.getCause() != null ? e.getCause() : e;
      throw new IOException(
          "From option " + OBSConstants.OBS_CREDENTIALS_PROVIDER + ' '
              + c, c);
    }

    if (credentialsProviderClass == null) {
      return createObsClientWithoutCredentialsProvider(conf, obsConf,
          name);
    }

    try {
      Constructor<?> cons =
          credentialsProviderClass.getDeclaredConstructor(URI.class,
              Configuration.class);
      credentialsProvider = (BasicSessionCredential) cons.newInstance(
          name, conf);
    } catch (NoSuchMethodException
        | SecurityException
        | IllegalAccessException

View on GitHub (pinned to 2add963021)

Solutions

1. Read the appended cause: 'class ... not found' means classpath, anything else (e.g. InstantiationException from abstract class) means the class itself.
2. Fix the FQCN in fs.obs.credentials.provider (e.g. com.obs.services.EnvironmentVariableObsCredentialsProvider or org.apache.hadoop.fs.obs.OBSCredentialsProvider*) and verify with a tiny program doing Class.forName(name).
3. Ship the provider jar to every node/executor (Spark: --jars or ship into hadoop lib dir), then restart.
4. If you did not intend a custom provider, delete the option — a null value falls back to createObsClientWithoutCredentialsProvider (static AK/SK from config).

Example fix

# before
<property><name>fs.obs.credentials.provider</name>
  <value>com.obs.MyCredentialsProivder</value></property> <!-- typo -->

# after
<property><name>fs.obs.credentials.provider</name>
  <value>com.obs.MyCredentialsProvider</value></property>
<!-- verify: java -cp ... Class.forName check, or spark-shell --jars provider.jar -->

static boolean providerClassLoads(String fqcn) {
  try { Class.forName(fqcn, false, Thread.currentThread().getContextClassLoader()); return true; }
  catch (Throwable t) { return false; }
}
String cls = conf.get("fs.obs.credentials.provider");
if (cls != null && !providerClassLoads(cls)) throw new ConfigException("missing provider class " + cls);

try {
  FileSystem.get(obsUri, conf);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).startsWith("From option fs.obs.credentials.provider")) {
    throw new ConfigException("fs.obs.credentials.provider misconfigured: " + e.getMessage(), e);
  }
  throw e;
}

When it happens

Trigger: fs.obs.credentials.provider points to a class not on the classpath (typo, wrong FQCN, jar missing from lib/ or the executor classpath); the value names an interface/abstract class; a shaded/relocated jar renamed the class between connector versions; the class exists but static initializers throw.

Common situations: Upgrading hadoop-huaweicloud where provider classes moved packages; deploying an OBS connector build that does not bundle the custom provider used in an older cluster; Spark executors missing the jar that is present on the driver; class name copy-pasted with spaces or from documentation of a different fork.

Related errors

• Proxy error: %s or %s set without the other.
• Cannot find password option {}
• Filesystem %s closed
• write has error. bs : pre upload obs[%s] has error.
• closed has error. bs : pre write obs[%s] has error.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/820b973adb7b832a. Report an issue: GitHub.