apache/iceberg · error · RuntimeException

Failed to initialize serializerCache for reading data with o

Error message

Failed to initialize serializerCache for reading data with old serializer

What it means

During schema-compatibility resolution, DynamicRecordInternalSerializer may restore an OLD serializer snapshot (pre serializerCache). It initializes the cache reflectively via DynMethods on the hidden snapshot class; any failure there is wrapped in a RuntimeException with this message, aborting restore from old-version state.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/dynamic/DynamicRecordInternalSerializer.java:334

    @Override
    public TypeSerializerSchemaCompatibility<DynamicRecordInternal> resolveSchemaCompatibility(
        TypeSerializerSnapshot<DynamicRecordInternal> oldSerializerSnapshot) {
      if (oldSerializerSnapshot.getCurrentVersion() == getCurrentVersion()) {
        return TypeSerializerSchemaCompatibility.compatibleAsIs();
      }

      // Old TypeSerializerSnapshots do not contain the serializer cache, but the newest one does.
      // This will also ensure that we always use the up-to-date cache alongside with its catalog
      // configuration.
      Preconditions.checkNotNull(serializerCache, "serializerCache should not be null");
      try {
        DynMethods.builder("initializeSerializerCache")
            .hiddenImpl(
                DynamicRecordInternalTypeSerializerSnapshot.class, TableSerializerCache.class)
            .build()
            .invoke(oldSerializerSnapshot, serializerCache);
      } catch (Exception e) {
        throw new RuntimeException(
            "Failed to initialize serializerCache for reading data with old serializer", e);
      }

      // This will first read data with the old serializer, then switch to the most recent one.
      return TypeSerializerSchemaCompatibility.compatibleAfterMigration();
    }

    @Override
    public TypeSerializer<DynamicRecordInternal> restoreSerializer() {
      if (getCurrentVersion() < MOST_RECENT_VERSION) {
        // If this serializer is not the most recent one, we need to read old data with the correct
        // parameters.
        return new DynamicRecordInternalSerializer(serializerCache, writeSchemaAndSpec, false);
      }

      // In all other cases, we just use the newest serializer.
      return new DynamicRecordInternalSerializer(serializerCache, writeSchemaAndSpec, true);
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Restore using the exact Iceberg version that wrote the savepoint, drain, then upgrade on a clean start.
2. Ensure the iceberg-flink runtime version is consistent across all nodes and matches the code performing the restore.
3. If reflective access fails due to module restrictions, add the required --add-opens for org.apache.iceberg.flink.sink.dynamic.
4. Avoid resuming in-flight dynamic-sink state across major serializer changes; restart the pipeline without the old snapshot.

try {
  compatibility = serializerSnapshot.resolveSchemaCompatibility(newSerializer);
} catch (RuntimeException e) {
  if (e.getMessage().contains("Failed to initialize serializerCache")) {
    // restore with the original Iceberg version, or restart without the old snapshot
  }
  throw e;
}

When it happens

Trigger: Restoring a Flink job from a savepoint/checkpoint written by an older Iceberg version whose DynamicRecordInternalTypeSerializerSnapshot lacks the expected initializeSerializerCache method or TableSerializerCache class.

Common situations: Iceberg runtime upgrades across restore with dynamic-sink operator state; mismatched jar versions where the hidden method signature changed; reflective access blocked by JPMS/module settings.

Related errors

• Unrecognized version or corrupt state: ${version}
• The Avro schema is not a nullable type: ${schema}
• Fail to serialize at field: %s.
• Unrecognized version or corrupt state: <version>
• Failed to serialize PK index key

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/2b751a8cd043015f. Report an issue: GitHub.