apache/flink · error · RuntimeException

Could not duplicate SimpleVersionedSerializer.

Error message

Could not duplicate SimpleVersionedSerializer.

What it means

Thrown as a RuntimeException by SimpleVersionedSerializerTypeSerializerProxy.duplicate() when cloning the serializerSupplier via InstantiationUtil.clone fails with ClassNotFoundException or IOException. duplicate() deep-clones the SerializableSupplier so each duplicated TypeSerializer has an independent supplier; if the supplier's class or its dependencies are not on the classloader, or Java serialization of the supplier fails, this wrapping error is thrown.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java:63

    public SimpleVersionedSerializerTypeSerializerProxy(
            SerializableSupplier<SimpleVersionedSerializer<T>> serializerSupplier) {
        this.serializerSupplier = checkNotNull(serializerSupplier, "serializerSupplier");
    }

    @Override
    public boolean isImmutableType() {
        return false;
    }

    @Override
    public TypeSerializer<T> duplicate() {
        try {
            return new SimpleVersionedSerializerTypeSerializerProxy<>(
                    InstantiationUtil.clone(
                            serializerSupplier, serializerSupplier.getClass().getClassLoader()));
        } catch (ClassNotFoundException | IOException e) {
            throw new RuntimeException("Could not duplicate SimpleVersionedSerializer.", e);
        }
    }

    @Override
    public T createInstance() {
        return null;
    }

    @Override
    public T copy(T from) {
        SimpleVersionedSerializer<T> serializer = getSerializer();
        try {
            byte[] serializedFrom = serializer.serialize(from);
            return serializer.deserialize(serializer.getVersion(), serializedFrom);
        } catch (IOException e) {
            throw new RuntimeException("Could not copy element.", e);
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Ensure the SerializableSupplier and all captured objects are Java-Serializable and available on the classloader performing duplicate().
2. If the supplier is a lambda, make sure it only captures Serializable-compatible state.
3. Verify the JAR containing the supplier class is deployed to all TaskManagers.
4. If duplicate() is not needed, consider whether the proxy is necessary at all.

Example fix

// before — supplier captures non-serializable state
SerializableSupplier<MySerializer> supplier = () -> new MySerializer(nonSerializableConfig);

// after — supplier captures only serializable state
SerializableSupplier<MySerializer> supplier = (SerializableSupplier<MySerializer>)
    () -> new MySerializer(serializableConfig);

try {
    TypeSerializer<T> dup = proxy.duplicate();
} catch (RuntimeException e) {
    if (e.getCause() instanceof ClassNotFoundException) {
        // ensure supplier class is on the classloader
    }
    throw e;
}

When it happens

Trigger: Calling TypeSerializer.duplicate() on a SimpleVersionedSerializerTypeSerializerProxy whose supplier is not Java-serializable or whose class is missing from the classloader; custom supplier classes referencing types not available at clone time.

Common situations: Classloader isolation issues (e.g. user-code classloader vs system classloader in a cluster); suppliers that capture non-Serializable lambdas or hold references to non-cloneable objects; deploying a job where the supplier's defining JAR is missing on some TaskManagers.

Related errors

• Could not serialize serializer into the configuration.
• Cannot deserialize and unwrap accumulators properly.
• Failed to deserialize coordination response
• Cannot deserialize and unwrap accumulators properly.
• Unable to instantiate Hadoop InputSplit

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/e25c439f48abbf2b. Report an issue: GitHub.