apache/iceberg · error · UncheckedIOException

Failed to encode partition

Error message

Failed to encode partition

What it means

encodePartition() serializes a StructLike partition tuple to bytes (length-prefixed, per-field). An IOException from the underlying write path is wrapped in UncheckedIOException. It almost always signals that a partition value cannot be converted to the declared partition field type via Conversions.toByteBuffer.

Solutions

  1. Ensure the StructLike partition values match partitionType exactly (use PartitionData built from the same Types.StructType)
  2. Re-derive partitions from the table's current spec rather than reusing cached ones from before a spec change
  3. Check for type mismatches like Integer vs Long for int/long partition fields and fix the producer
  4. Catch the UncheckedIOException and log the partition type and values to identify the offending field

Example fix

// before
PartitionData p = new PartitionData(oldSpecPartitionType);
p.set(0, "2024-01-01"); // string for a date field
// after
PartitionData p = new PartitionData(table.spec().partitionType());
p.set(0, LocalDate.parse("2024-01-01"));
Defensive patterns

Strategy: validation

Validate before calling

Types.StructType partType = table.spec().partitionType();
for (int i = 0; i < partType.fields().size(); i++) {
  Object v = partition.get(i, Object.class);
  Preconditions.checkState(v == null || v.getClass() == expectedJavaClass(partType.fields().get(i).type()),
      "Partition field %s expects %s but got %s", partType.fields().get(i).name(),
      partType.fields().get(i).type(), v == null ? null : v.getClass());
}

Try / catch

try {
  byte[] encoded = serializer.encodePartition(partition, partitionType);
} catch (UncheckedIOException e) {
  LOG.error("Partition encode failed for type {} values {}", partitionType, partition, e);
  throw e;
}

Prevention

When it happens

Trigger: Passing a partition StructLike whose values don't match partitionType (e.g. mismatched types after spec/schema changes); a null-typed or custom StructLike returning an unexpected object type.

Common situations: Partition spec evolved and the encoder is given old partition objects with a new partition type; manually constructed PartitionData with wrong value types; rows produced by an engine writing partition values in a different representation.

Understand the failure class

Background: "JSON serialization failed", "not JSON serializable", "Failed to serialize": why JSON marshaling errors happen and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d21a5d1c318fbb89. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/StructLikeSerializer.java:88

    return new SerializedEqualityValues(baos.toByteArray());
  }

  public byte[] encodePartition(StructLike partition, Types.StructType partitionType) {
    List<Types.NestedField> fields = partitionType.fields();
    if (fields.isEmpty()) {
      return EMPTY_PARTITION;
    }

    baos.reset();
    try {
      for (int i = 0; i < fields.size(); i++) {
        writeField(partition, i, fields.get(i).type());
      }

      dos.flush();
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to encode partition", e);
    }

    return baos.toByteArray();
  }

  public static StructLike decodePartition(byte[] encoded, Types.StructType partitionType) {
    PartitionData partition = new PartitionData(partitionType);
    List<Types.NestedField> fields = partitionType.fields();
    if (fields.isEmpty()) {
      return partition;
    }

    try (DataInputStream dis = new DataInputStream(new ByteArrayInputStream(encoded))) {
      for (int i = 0; i < fields.size(); i++) {
        boolean isNull = dis.readBoolean();
        if (isNull) {
          partition.set(i, null);
        } else {

View on GitHub (pinned to 86d9c8fc54)