apache/flink · critical · RuntimeException

Unable to instantiate Hadoop InputSplit

Error message

Unable to instantiate Hadoop InputSplit

What it means

Thrown during Java deserialization of the mapred-API HadoopInputSplit. On the TaskManager side, readObject() rebuilds the wrapped Hadoop split via WritableFactories.newInstance(splitType), which relies on the split class having a no-arg constructor (or a registered WritableFactory). If the class cannot be instantiated, the deserialization fails and the RuntimeException aborts split recovery, failing the task.

Source

Thrown at flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/wrapper/HadoopInputSplit.java:119

        if (needsJobConf(hadoopInputSplit)) {
            // the job conf knows how to serialize itself
            // noinspection ConstantConditions
            jobConf.write(out);
        }

        // write the input split
        hadoopInputSplit.write(out);
    }

    private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException {
        // read the parent fields and the final fields
        in.defaultReadObject();

        try {
            hadoopInputSplit =
                    (org.apache.hadoop.mapred.InputSplit) WritableFactories.newInstance(splitType);
        } catch (Exception e) {
            throw new RuntimeException("Unable to instantiate Hadoop InputSplit", e);
        }

        if (needsJobConf(hadoopInputSplit)) {
            // the job conf knows how to deserialize itself
            jobConf = new JobConf();
            jobConf.readFields(in);

            if (hadoopInputSplit instanceof Configurable) {
                ((Configurable) hadoopInputSplit).setConf(this.jobConf);
            } else if (hadoopInputSplit instanceof JobConfigurable) {
                ((JobConfigurable) hadoopInputSplit).configure(this.jobConf);
            }
        }

        hadoopInputSplit.readFields(in);
    }

    private static boolean needsJobConf(org.apache.hadoop.mapred.InputSplit split) {

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the concrete Hadoop InputSplit class is on the TaskManager classpath (ship the user jar with all required classes via the job's user-code jar).
2. Verify the InputSplit implementation class has a public no-arg constructor, which WritableFactories.newInstance relies on.
3. If the split is a custom type, register a WritableFactory for it via WritableFactories.registerFactory(splitClass, factory) before the job runs, or extend an existing Hadoop split that is already Writable-factory-enabled.
4. Confirm the Hadoop dependency version on the cluster matches the one used to build the job.

Example fix

// before — custom split with no default constructor
public class MySplit implements Writable {
    public MySplit(Path p) { ... }
}
// after — add a no-arg constructor used by WritableFactories
public class MySplit implements Writable {
    public MySplit() {}
    public MySplit(Path p) { ... }
}

// Before shipping the job, verify the concrete split class is instantiable via WritableFactories
Class<?> splitType = myInputSplit.getClass();
try {
    Object probe = org.apache.hadoop.io.WritableFactories.newInstance(splitType);
    if (probe == null) throw new IllegalStateException("WritableFactories returned null for " + splitType);
} catch (Exception e) {
    throw new IllegalStateException("Split class " + splitType.getName()
        + " is not instantiable by WritableFactories (needs public no-arg ctor or registered factory)", e);
}

When it happens

Trigger: Triggered when Flink ships a serialized HadoopInputSplit to a TaskManager and WritableFactories.newInstance(splitType) throws — e.g. the InputSplit implementation class has no public no-arg constructor, is abstract, or is not visible/loadable by WritableFactories' reflection. The splitType is the concrete class captured at construction on the client.

Common situations: A custom mapred InputSplit whose Writable class lacks a default constructor; the split class lives in a user jar that is not shipped to the TaskManagers; Hadoop version mismatch where a split class changed its constructor requirements; a split class that is package-private or lacks the WritableFactory registration Hadoop expects.

Related errors

• Unable to instantiate the Hadoop InputSplit
• Unable to instantiate the hadoop input format
• Unable to find key class.
• Unable to find value class.
• Unable to instantiate the hadoop output format

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/9bae6b0b721da912. Report an issue: GitHub.