{"record":{"id":"b663cc556896301f","repo":"apache/iceberg","slug":"fail-to-serialize-aggregated-statistics-b663cc","errorCode":null,"errorMessage":"Fail to serialize aggregated statistics","messagePattern":"Fail to serialize aggregated statistics","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java","lineNumber":71,"sourceCode":"  static DataStatistics deserializeDataStatistics(\n      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {\n    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);\n    try {\n      return statisticsSerializer.deserialize(input);\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Fail to deserialize data statistics\", e);\n    }\n  }\n\n  static byte[] serializeCompletedStatistics(\n      CompletedStatistics completedStatistics,\n      TypeSerializer<CompletedStatistics> statisticsSerializer) {\n    try {\n      DataOutputSerializer out = new DataOutputSerializer(1024);\n      statisticsSerializer.serialize(completedStatistics, out);\n      return out.getCopyOfBuffer();\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Fail to serialize aggregated statistics\", e);\n    }\n  }\n\n  static CompletedStatistics deserializeCompletedStatistics(\n      byte[] bytes, CompletedStatisticsSerializer statisticsSerializer) {\n    try {\n      DataInputDeserializer input = new DataInputDeserializer(bytes);\n      CompletedStatistics completedStatistics = statisticsSerializer.deserialize(input);\n      if (!completedStatistics.isValid()) {\n        throw new RuntimeException(\"Fail to deserialize aggregated statistics,change to v1\");\n      }\n\n      return completedStatistics;\n    } catch (Exception e) {\n      try {\n        // If we restore from a lower version, the new version of SortKeySerializer cannot correctly\n        // parse the checkpointData, so we need to first switch the version to v1. Once the state\n        // data is successfully parsed, we need to switch the serialization version to the latest","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java#L53-L89","documentation":"StatisticsUtil.serializeCompletedStatistics serializes aggregated (completed) statistics via the provided serializer into bytes; IOException becomes UncheckedIOException \"Fail to serialize aggregated statistics\". Used when persisting/broadcasting final sort-key statistics after shuffle aggregation.","triggerScenarios":"Calling serializeCompletedStatistics when the underlying CompletedStatisticsSerializer.serialize throws IOException — usually buffer growth failure or memory pressure, since the flow is in-memory.","commonSituations":"Very large sketches (many sort-key distributions) exhausting the DataOutputSerializer buffer under constrained task-manager memory.","solutions":["Check the cause; increase task-manager memory or reduce sketch size/num samples configured for the sink.","Confirm the CompletedStatistics object is fully initialized (non-null sort key serializer) before serialization.","Retry after transient backpressure resolves."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Preconditions.checkNotNull(completedStatistics, \"completedStatistics must not be null\");","typeGuard":null,"tryCatchPattern":"try {\n  byte[] out = StatisticsUtil.serializeCompletedStatistics(stats, serializer);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Failed to serialize aggregated statistics\", e.getCause());\n}","preventionTips":["Bound sketch size via sink write-aerosol/sketch configuration to limit serialized size","Monitor task-manager memory before large aggregations","Retry after transient backpressure"],"tags":["flink","serialization","unchecked-io","statistics"],"backgroundTag":"json-serialization-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}