apache/iceberg · error · RuntimeException

Failed to initialize serializerCache for reading data with…

Error message

Failed to initialize serializerCache for reading data with old serializer

What it means

In DynamicRecordInternalSerializer.resolveSchemaCompatibility, when restoring serializer state written by an old snapshot format, the code reflectively invokes the hidden initializeSerializerCache method on DynamicRecordInternalTypeSerializerSnapshot. Any reflection failure is wrapped in a RuntimeException with this message, so old-version data cannot be read for migration.

Solutions

1. Align the iceberg-flink runtime version used at restore time with the version that wrote the checkpoint, or use a version that supports migration from it.
2. Ensure only one consistent iceberg-flink jar is on the classpath (no shaded duplicates).
3. If the old checkpoint cannot be migrated, start a new job without state and let the sink rewrite in-flight data.

// verify class presence and consistent jars before restore
Class.forName("org.apache.iceberg.flink.sink.dynamic.DynamicRecordInternalTypeSerializerSnapshot");

try { compat = serializer.resolveSchemaCompatibility(oldSnapshot); } catch (RuntimeException e) { if (e.getMessage().contains("Failed to initialize serializerCache")) { /* plan full restart without state or with matching versions */ } throw e; }

When it happens

Trigger: Restoring from a checkpoint/savepoint written by an older Iceberg-Flink release whose DynamicRecordInternalTypeSerializerSnapshot layout differs, causing the hidden-method invocation (or cache initialization) to fail — seen in testRestoreFromOldVersion scenarios.

Common situations: Cross-version upgrades (e.g. 1.9 -> 2.x) where the internal snapshot class changed; jars on the classpath mixing old and new iceberg-flink classes.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

• Failed to initialize serializerCache for reading data with…
• Could not deserialize the WriteResult object
• Could not deserialize the WriteResult object
• Could not deserialize the WriteResult object
• Could not deserialize the WriteResult object

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/30717c6b5a0288a8. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java:334

    @Override
    public TypeSerializerSchemaCompatibility<DynamicRecordInternal> resolveSchemaCompatibility(
        TypeSerializerSnapshot<DynamicRecordInternal> oldSerializerSnapshot) {
      if (oldSerializerSnapshot.getCurrentVersion() == getCurrentVersion()) {
        return TypeSerializerSchemaCompatibility.compatibleAsIs();
      }

      // Old TypeSerializerSnapshots do not contain the serializer cache, but the newest one does.
      // This will also ensure that we always use the up-to-date cache alongside with its catalog
      // configuration.
      Preconditions.checkNotNull(serializerCache, "serializerCache should not be null");
      try {
        DynMethods.builder("initializeSerializerCache")
            .hiddenImpl(
                DynamicRecordInternalTypeSerializerSnapshot.class, TableSerializerCache.class)
            .build()
            .invoke(oldSerializerSnapshot, serializerCache);
      } catch (Exception e) {
        throw new RuntimeException(
            "Failed to initialize serializerCache for reading data with old serializer", e);
      }

      // This will first read data with the old serializer, then switch to the most recent one.
      return TypeSerializerSchemaCompatibility.compatibleAfterMigration();
    }

    @Override
    public TypeSerializer<DynamicRecordInternal> restoreSerializer() {
      if (getCurrentVersion() < MOST_RECENT_VERSION) {
        // If this serializer is not the most recent one, we need to read old data with the correct
        // parameters.
        return new DynamicRecordInternalSerializer(serializerCache, writeSchemaAndSpec, false);
      }

      // In all other cases, we just use the newest serializer.
      return new DynamicRecordInternalSerializer(serializerCache, writeSchemaAndSpec, true);
    }

View on GitHub (pinned to 86d9c8fc54)