{"record":{"id":"cb831bc5d3398241","repo":"apache/iceberg","slug":"fail-to-serialize-data-statistics-cb831b","errorCode":null,"errorMessage":"Fail to serialize data statistics","messagePattern":"Fail to serialize data statistics","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java","lineNumber":49,"sourceCode":"\n  static DataStatistics createTaskStatistics(\n      StatisticsType type, int operatorParallelism, int numPartitions) {\n    if (type == StatisticsType.Map) {\n      return new MapDataStatistics();\n    } else {\n      return new SketchDataStatistics(\n          SketchUtil.determineOperatorReservoirSize(operatorParallelism, numPartitions));\n    }\n  }\n\n  static byte[] serializeDataStatistics(\n      DataStatistics dataStatistics, TypeSerializer<DataStatistics> statisticsSerializer) {\n    DataOutputSerializer out = new DataOutputSerializer(64);\n    try {\n      statisticsSerializer.serialize(dataStatistics, out);\n      return out.getCopyOfBuffer();\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Fail to serialize data statistics\", e);\n    }\n  }\n\n  static DataStatistics deserializeDataStatistics(\n      byte[] bytes, TypeSerializer<DataStatistics> statisticsSerializer) {\n    DataInputDeserializer input = new DataInputDeserializer(bytes, 0, bytes.length);\n    try {\n      return statisticsSerializer.deserialize(input);\n    } catch (IOException e) {\n      throw new UncheckedIOException(\"Fail to deserialize data statistics\", e);\n    }\n  }\n\n  static byte[] serializeCompletedStatistics(\n      CompletedStatistics completedStatistics,\n      TypeSerializer<CompletedStatistics> statisticsSerializer) {\n    try {\n      DataOutputSerializer out = new DataOutputSerializer(1024);","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/StatisticsUtil.java#L31-L67","documentation":"StatisticsUtil.serializeDataStatistics serializes a DataStatistics object through the provided Flink TypeSerializer into a byte array; any IOException becomes UncheckedIOException \"Fail to serialize data statistics\". It is a low-level utility used when shipping sort-key statistics through Flink operator state/broadcast.","triggerScenarios":"Calling StatisticsUtil.serializeDataStatistics when the underlying TypeSerializer.serialize throws IOException — essentially only from stream/buffer failures or a malformed statistics object.","commonSituations":"Task-manager memory pressure during shuffle statistics collection; a custom/incorrect statistics serializer passed in.","solutions":["Check the IOException cause; increase task-manager heap/network memory if it is a buffer failure.","Ensure the serializer passed matches the DataStatistics implementation (SortKeyStatisticsSerializer for sort-key stats).","Retry the job; transient buffer issues usually do not recur after backpressure eases."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Preconditions.checkNotNull(dataStatistics, \"dataStatistics must not be null\");\nPreconditions.checkNotNull(statisticsSerializer, \"statisticsSerializer must not be null\");","typeGuard":null,"tryCatchPattern":"try {\n  byte[] out = StatisticsUtil.serializeDataStatistics(stats, serializer);\n} catch (UncheckedIOException e) {\n  LOG.error(\"Failed to serialize data statistics\", e.getCause());\n}","preventionTips":["Pass the serializer matching the DataStatistics implementation","Ensure adequate task-manager heap/network memory","Inspect the IOException cause for buffer failures"],"tags":["flink","serialization","unchecked-io","statistics"],"backgroundTag":"json-serialization-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}