{"record":{"id":"273f03546119d7d2","repo":"apache/iceberg","slug":"fail-to-deserialize-aggregated-statistics-273f03","errorCode":null,"errorMessage":"Fail to deserialize aggregated statistics","messagePattern":"Fail to deserialize aggregated statistics","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java","lineNumber":97,"sourceCode":"      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);\n      if (!completedStatistics.isValid()) {\n        throw new RuntimeException(\"Fail to deserialize aggregated statistics,change to v1\");\n      }\n\n      return completedStatistics;\n    } catch (Exception e) {\n      try {\n        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly\n        // parse the checkpointData, so we need to first switch the version to v1. Once the state\n        // data is successfully parsed, we need to switch the serialization version to the latest\n        // version to parse the subsequent data passed from the TM.\n        statisticsSerializer.changeSortKeySerializerVersion(1);\n        DataInputDeserializer input = new DataInputDeserializer(bytes);\n        CompletedStatistics deserialize = statisticsSerializer.deserialize(input);\n        statisticsSerializer.changeSortKeySerializerVersionLatest();\n        return deserialize;\n      } catch (IOException ioException) {\n        throw new UncheckedIOException(\"Fail to deserialize aggregated statistics\", ioException);\n      }\n    }\n  }\n\n  static byte[] serializeGlobalStatistics(\n      GlobalStatistics globalStatistics, TypeSerializer<GlobalStatistics> statisticsSerializer) {\n    try {\n      DataOutputSerializer out = new DataOutputSerializer(1024);\n      statisticsSerializer.serialize(globalStatistics, out);\n      return out.getCopyOfBuffer();\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Fail to serialize aggregated statistics\", e);\n    }\n  }\n\n  static GlobalStatistics deserializeGlobalStatistics(\n      byte[] bytes, TypeSerializer<GlobalStatistics> statisticsSerializer) {\n    try {","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java#L79-L115","documentation":"StatisticsUtil.deserializeCompletedStatistics's final fallback: after retrying deserialization with sort-key serializer version 1, any remaining IOException becomes UncheckedIOException \"Fail to deserialize aggregated statistics\". This means the bytes are unreadable with both the latest and the v1 serializer — the payload is corrupt, truncated, or from an unrecognized format.","triggerScenarios":"deserializeCompletedStatistics where the v1-fallback path's statisticsSerializer.deserialize throws IOException, e.g. truncated checkpoint bytes, wrong byte array passed in, or bytes from a format neither serializer version handles.","commonSituations":"Corrupted savepoint/checkpoint files; passing statistics bytes to the wrong deserializer (e.g. GlobalStatistics bytes fed to CompletedStatisticsSerializer); hand-migrated state.","solutions":["Verify the byte array actually contains completed statistics from the same serializer family (not global/other statistics).","Restore from an intact checkpoint/savepoint; discard the corrupted state file.","If corruption is systemic, disable restore and start fresh — sort-key statistics are advisory and rebuilt automatically by the sink."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (bytes == null || bytes.length == 0) { throw new IllegalArgumentException(\"No statistics bytes\"); }","typeGuard":null,"tryCatchPattern":"try {\n  stats = StatisticsUtil.deserializeCompletedStatistics(bytes, serializer);\n} catch (UncheckedIOException e) {\n  LOG.warn(\"Statistics unreadable in both serializer versions; discarding\", e);\n  stats = null; // sink rebuilds statistics\n}","preventionTips":["Guard checkpoint/savepoint integrity (checksums, no manual edits)","Only feed matching statistics bytes to each deserializer","Accept statistics loss gracefully — they are advisory and self-rebuilding"],"tags":["flink","deserialization","unchecked-io","statistics"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}