apache/hadoop · error · RuntimeException

Failed to instantiate custom class " + name + " " + e

Error message

Failed to instantiate custom class " + name + " " + e

What it means

When fs.s3a.input.stream.type=Custom, StreamIntegration.loadCustomFactory() loads the class named by fs.s3a.input.stream.custom.factory and reflectively invokes its no-arg constructor. Any failure - property unset (factoryClass is null -> NPE in getConstructor()), class missing from classpath, class not implementing ObjectInputStreamFactory, missing public no-arg constructor, or constructor throwing - is wrapped in RuntimeException("Failed to instantiate custom class <name> <cause>").

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/streams/StreamIntegration.java:187

   * @throws RuntimeException any binding/loading/instantiation problem
   */
  static ObjectInputStreamFactory loadCustomFactory(Configuration conf) {

    // make sure the classname option is actually set
    final String name = conf.getTrimmed(INPUT_STREAM_CUSTOM_FACTORY, "");
    checkArgument(!isEmpty(name), E_EMPTY_CUSTOM_CLASSNAME);

    final Class<? extends ObjectInputStreamFactory> factoryClass =
        conf.getClass(INPUT_STREAM_CUSTOM_FACTORY,
            null,
            ObjectInputStreamFactory.class);

    try {
      final Constructor<? extends ObjectInputStreamFactory> ctor =
          factoryClass.getConstructor();
      return ctor.newInstance();
    } catch (Exception e) {
      throw new RuntimeException("Failed to instantiate custom class "
          + name + " " + e, e);
    }
  }

  /**
   * Populates the configurations related to vectored IO operations.
   * The context is still mutable at this point.
   * @param conf configuration object.
   * @return VectoredIOContext.
   */
  public static VectoredIOContext populateVectoredIOContext(Configuration conf) {
    final int minSeekVectored = (int) longBytesOption(conf, AWS_S3_VECTOR_READS_MIN_SEEK_SIZE,
        DEFAULT_AWS_S3_VECTOR_READS_MIN_SEEK_SIZE, 0);
    final int maxReadSizeVectored =
        (int) longBytesOption(conf, AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE,
            DEFAULT_AWS_S3_VECTOR_READS_MAX_MERGED_READ_SIZE, 0);
    final int vectoredActiveRangeReads = intOption(conf,
        AWS_S3_VECTOR_ACTIVE_RANGE_READS, DEFAULT_AWS_S3_VECTOR_ACTIVE_RANGE_READS, 1);

View on GitHub (pinned to 2add963021)

Solutions

1. Set fs.s3a.input.stream.custom.factory to the fully-qualified class name whenever fs.s3a.input.stream.type is Custom.
2. Give the factory a public no-arg constructor and make it implement org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStreamFactory.
3. Ship the factory jar on every JVM that opens the s3a filesystem (client classpath, YARN containers, or fs.s3a.aws.sdk classpath-style mechanisms).
4. Test instantiation standalone: Class.forName(name).getConstructor().newInstance() before shipping config.

Example fix

// before
<property><name>fs.s3a.input.stream.type</name><value>Custom</value></property>
<!-- factory property missing -> NPE wrapped in RuntimeException -->

// after
<property><name>fs.s3a.input.stream.type</name><value>Custom</value></property>
<property><name>fs.s3a.input.stream.custom.factory</name>
  <value>com.example.MyObjectInputStreamFactory</value></property>

// required shape of the class:
public class MyObjectInputStreamFactory implements ObjectInputStreamFactory {
  public MyObjectInputStreamFactory() { }
  // ... interface methods
}

String name = conf.getTrimmed("fs.s3a.input.stream.custom.factory", "");
if ("Custom".equalsIgnoreCase(conf.getTrimmed("fs.s3a.input.stream.type", ""))) {
  if (name.isEmpty()) throw new IllegalArgumentException("fs.s3a.input.stream.custom.factory is required");
  Class<?> c = Class.forName(name);                       // ClassNotFoundException if jar missing
  if (!org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStreamFactory.class.isAssignableFrom(c))
    throw new IllegalArgumentException(name + " does not implement ObjectInputStreamFactory");
  c.getConstructor();                                      // NoSuchMethodException if no no-arg ctor
}

try {
  FileSystem fs = path.getFileSystem(conf);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to instantiate custom class")) {
    // report classpath/jar/constructor issue; do not retry with same config
  } else { throw e; }
}

When it happens

Trigger: Setting fs.s3a.input.stream.type=Custom without fs.s3a.input.stream.custom.factory; custom factory jar not on the classpath of the JVM opening the filesystem (client, NM container, etc.); class lacking a public no-arg constructor; constructor throwing (e.g. reading config that is absent); class not assignable to ObjectInputStreamFactory so conf.getClass returns null.

Common situations: Teams shipping custom S3A input streams forgetting the jar in the cluster's share directory or the job's classpath; refactoring away the no-arg constructor; typos in the class name; version skew where the factory compiled against an older hadoop-aws API no longer implements the current ObjectInputStreamFactory interface.

Related errors

• Failed to instantiate class {auditClassname} defined in {key
• Errors on getting mount table loader class. The fs.viewfs.mo
• Error creating plugin: {}
• Failed to get method
• Unable to create the Decommission monitor from {cls}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/44fe058f3dbdb108. Report an issue: GitHub.