apache/beam · error · IOException

Unable to create configuration

Error message

Unable to create configuration: ${e}

What it means

SerializableConfiguration.readExternal reconstructs a Hadoop Configuration from a serialized stream. If instantiating the Configuration class reflectively fails (no default constructor, inaccessible class, constructor exception), it wraps the problem in this IOException. It means the deserialized configuration object could not be created.

Solutions

1. Ensure hadoop-common and its dependencies are on the worker classpath at the same version as submission time
2. Check the wrapped `e` in the message for the true reflective failure (InstantiationException/NoSuchMethodException etc.)
3. Verify no shading/relocation breaks org.apache.hadoop.conf.Configuration's no-arg constructor
4. Test Configuration c = new Configuration() directly on the worker to confirm the class loads

Example fix

// before (broken classpath)
conf = new SerializableConfiguration();
// after — ensure dependency present
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <scope>provided</scope>
</dependency>

try {
  Configuration probe = new Configuration();
  probe.get("fs.defaultFS");
} catch (Throwable t) {
  throw new IllegalStateException("hadoop Configuration unavailable on classpath", t);
}

try {
  serConf.readExternal(in);
} catch (IOException e) {
  if (e.getMessage().startsWith("Unable to create configuration")) {
    // check hadoop-common on worker classpath
  }
  throw e;
}

When it happens

Trigger: Deserializing a SerializableConfiguration (e.g. from a worker state or pipeline serialization) when Configuration cannot be instantiated: wrong Hadoop version, Configuration class not on classpath, or reflective constructor failure.

Common situations: Running Beam Hadoop IO connectors on runners where the Hadoop classpath differs from the one used at submission time, or shaded/missing hadoop-common dependency causing class initialization errors.

Related errors

• Cannot provide because does not implement the interface
• ${e}
• unable to deserialize
• unable to deserialize record
• unable to deserialize record

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/514a4235a151ac12. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/SerializableConfiguration.java:99

    out.write(serializationCache);
  }

  @Override
  public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException {
    confMutated = true;
    String className = in.readUTF();
    try {
      conf =
          Class.forName(className)
              .asSubclass(Configuration.class)
              .getDeclaredConstructor()
              .newInstance();
      conf.readFields(in);
    } catch (InstantiationException
        | IllegalAccessException
        | NoSuchMethodException
        | InvocationTargetException e) {
      throw new IOException("Unable to create configuration: " + e);
    }
  }

  /** Returns new configured {@link Job} object. */
  public static Job newJob(@Nullable SerializableConfiguration conf) throws IOException {
    if (conf == null) {
      return Job.getInstance();
    } else {
      // Don't reading configuration from slave thread, but only from master thread.
      Job job = Job.getInstance(new Configuration(false));
      for (Map.Entry<String, String> entry : conf.get()) {
        job.getConfiguration().set(entry.getKey(), entry.getValue());
      }
      return job;
    }
  }

  /** Returns a new configuration instance using provided flags. */

View on GitHub (pinned to 12126d8942)