apache/flink · error · RuntimeException

Cannot copy serializer

Error message

Cannot copy serializer

What it means

PojoComparator's copy constructor clones its PojoSerializer by Java-serializing and deserializing it (InstantiationUtil round-trip with the context classloader). If the serializer cannot be Java-serialized or its class is not loadable by the thread-context classloader, this RuntimeException is thrown.

Source

Thrown at flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/PojoComparator.java:152

        for (int i = 0; i < toClone.comparators.length; i++) {
            this.comparators[i] = toClone.comparators[i].duplicate();
        }

        this.normalizedKeyLengths = toClone.normalizedKeyLengths;
        this.numLeadingNormalizableKeys = toClone.numLeadingNormalizableKeys;
        this.normalizableKeyPrefixLen = toClone.normalizableKeyPrefixLen;
        this.invertNormKey = toClone.invertNormKey;

        this.type = toClone.type;

        try {
            this.serializer =
                    (TypeSerializer<T>)
                            InstantiationUtil.deserializeObject(
                                    InstantiationUtil.serializeObject(toClone.serializer),
                                    Thread.currentThread().getContextClassLoader());
        } catch (IOException | ClassNotFoundException e) {
            throw new RuntimeException("Cannot copy serializer", e);
        }
    }

    private void writeObject(ObjectOutputStream out) throws IOException, ClassNotFoundException {
        out.defaultWriteObject();
        out.writeInt(keyFields.length);
        for (Field field : keyFields) {
            FieldSerializer.serializeField(field, out);
        }
    }

    private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException {
        in.defaultReadObject();
        int numKeyFields = in.readInt();
        keyFields = new Field[numKeyFields];
        for (int i = 0; i < numKeyFields; i++) {
            keyFields[i] = FieldSerializer.deserializeField(in);
        }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the cause: NotSerializableException points at the non-serializable field to make transient/serializable; ClassNotFoundException points at a classloader issue
2. Ensure POJO and any custom serializer classes are inside the job jar (not only on the server classpath) so the user-code classloader can load them
3. Mark non-serializable cached resources (like Kryo instances) transient and lazily reinitialize them in readObject/open

Example fix

// before
class MyPojoSerializer extends TypeSerializer<MyPojo> {
    private final Kryo kryo = new Kryo(); // not serializable
}

// after
class MyPojoSerializer extends TypeSerializer<MyPojo> implements Serializable {
    private transient Kryo kryo;
    private Kryo kryo() {
        if (kryo == null) kryo = new Kryo();
        return kryo;
    }
}

// Verify the comparator's serializer can be Java-serialized before duplicate()
try {
    InstantiationUtil.serializeObject(comparator.getSerializer());
} catch (IOException e) {
    throw new IllegalStateException("Serializer not Java-serializable: " + e, e);
}

try {
    TypeComparator<T> dup = comparator.duplicate();
} catch (RuntimeException e) {
    Throwable cause = e.getCause();
    if (cause instanceof ClassNotFoundException) {
        throw new IllegalStateException("Serializer class not on context classloader", cause);
    }
    throw e;
}

When it happens

Trigger: comparator.duplicate() on a PojoComparator where the wrapped PojoSerializer contains state (e.g. cached Kryo instance or custom serializer classes) that fails Java serialization, or when Thread.currentThread().getContextClassLoader() cannot see the serializer's class - typical in classloader-isolated deployments (sessions, libraries directory, SQL planner loader).

Common situations: Running jobs on mini-cluster/session with user-code classloader separation; Kryo serializer cached inside PojoSerializer is not serializable; ClassNotFoundException after moving POJO/serializer classes between job submissions; custom subclasses of PojoSerializer not on the context classloader.

Related errors

• Comparator {} specifies an invalid length for the normalized
• Unable to access field {} on object {}
• A NullPointerException occurred while accessing a key field
• {}
• Could not serialize comparator into the configuration.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/c0cfa0b4e38abb9e. Report an issue: GitHub.