apache/flink · critical · RuntimeException

Unable to instantiate the hadoop input format

Error message

Unable to instantiate the hadoop input format

What it means

Thrown during Java deserialization of HadoopInputFormatBase to a TaskManager. readObject() reconstructs the wrapped InputFormat by name using Class.forName(className, true, contextClassLoader).newInstance(). If the class is missing, abstract, non-instantiable, or lacks an accessible no-arg constructor, the RuntimeException aborts deserialization and the task cannot start.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapreduce/HadoopInputFormatBase.java:323

        org.apache.hadoop.conf.Configuration configuration =
                new org.apache.hadoop.conf.Configuration();
        configuration.readFields(in);

        if (this.configuration == null) {
            this.configuration = configuration;
        }

        try {
            this.mapreduceInputFormat =
                    (org.apache.hadoop.mapreduce.InputFormat<K, V>)
                            Class.forName(
                                            hadoopInputFormatClassName,
                                            true,
                                            Thread.currentThread().getContextClassLoader())
                                    .newInstance();
        } catch (Exception e) {
            throw new RuntimeException("Unable to instantiate the hadoop input format", e);
        }
        try {
            this.keyClass =
                    (Class<K>)
                            Class.forName(
                                    keyClassName,
                                    true,
                                    Thread.currentThread().getContextClassLoader());
        } catch (Exception e) {
            throw new RuntimeException("Unable to find key class.", e);
        }
        try {
            this.valueClass =
                    (Class<V>)
                            Class.forName(
                                    valueClassName,
                                    true,
                                    Thread.currentThread().getContextClassLoader());

View on GitHub (pinned to 2f3c205e92)

Solutions

  1. Package the concrete Hadoop InputFormat class (and its dependencies) into the user-code jar submitted with the job.
  2. Confirm the InputFormat class has a public no-arg constructor required by newInstance().
  3. Verify the cluster's Hadoop version exposes the same class name as at build time (no relocation/removal).
  4. If shading, ensure the className written at serialization time matches the resolvable name on the TaskManager.
Defensive patterns

Strategy: validation

Validate before calling

// Before submitting, confirm the InputFormat class is loadable + instantiable on the runtime classpath
String className = myInputFormat.getClass().getName();
try {
    Class<?> c = Class.forName(className, true, Thread.currentThread().getContextClassLoader());
    if (java.lang.reflect.Modifier.isAbstract(c.getModifiers()) || c.isInterface())
        throw new IllegalStateException(className + " is abstract/interface — cannot instantiate");
    c.getDeclaredConstructor().newInstance(); // mirrors readObject's newInstance()
} catch (Exception e) {
    throw new IllegalStateException("InputFormat class " + className
        + " cannot be instantiated on the cluster (missing jar / no public no-arg ctor)", e);
}

Prevention

When it happens

Trigger: Triggered on the TaskManager when the serialized HadoopInputFormat is rebuilt and Class.forName(hadoopInputFormatClassName, true, contextClassLoader).newInstance() fails — e.g. the InputFormat class is not in the user jar shipped to the cluster, has no public no-arg constructor, is abstract/interface, or the context classloader cannot resolve it.

Common situations: The Hadoop InputFormat implementation (e.g. a custom or connector InputFormat) is not included in the submitted user-code jar; class relocated/shaded differently than at build time; Hadoop version on cluster differs and a class was removed/renamed; SecurityManager or access rules block reflective instantiation.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/7f89cdcea2f255a1. Report an issue: GitHub.