apache/iceberg · error · UncheckedIOException

Failed to decode partition

Error message

Failed to decode partition

What it means

StructLikeSerializer.decodePartition reads back an encoded partition byte[] into a mutable StructLike, field by field, using the supplied partition Types.StructType. An IOException while reading (truncated buffer, unexpected end of stream, or a field written with a different type/layout than readField expects) is wrapped in an UncheckedIOException 'Failed to decode partition'.

Solutions

1. Ensure the byte[] comes from the same serializer/schema version; rebuild the index after partition-spec or schema evolution instead of decoding stale keys.
2. Validate the byte array is non-empty and the expected length before calling decodePartition.
3. Pass the same Types.StructType used at encode time (store specId alongside the encoded partition).
4. Inspect the wrapped cause to identify at which field the read failed and compare against the encoding schema.

Example fix

// before
StructLike p = StructLikeSerializer.decodePartition(bytes, table.spec().partitionType());
// stale bytes encoded with an older spec

// after
if (bytes == null || bytes.length == 0) {
  throw new IllegalArgumentException("No encoded partition stored");
}
StructLike p = StructLikeSerializer.decodePartition(bytes, specTypeAtEncodeTime);

if (bytes == null || bytes.length == 0) {
  throw new IllegalArgumentException("No encoded partition bytes to decode");
}

try {
  StructLike p = StructLikeSerializer.decodePartition(bytes, partitionType);
} catch (UncheckedIOException e) {
  LOG.error("Partition decode failed (stale or corrupt bytes): {}", e.getCause().getMessage());
  // drop the stale index entry and rebuild it
}

When it happens

Trigger: decodePartition receives bytes not produced by the matching encodePartition call — e.g. empty/truncated byte arrays, bytes encoded with a different partition type, or state restored from a different schema version.

Common situations: Partition spec/schema changed between encode and decode (index entries persisted across a schema evolution); corrupted or empty byte arrays read back from RocksDB/state or checkpoint; wrong partitionType argument causing field-type mismatch mid-read.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to encode partition
• Failed to encode partition
• Could not deserialize the WriteResult object
• Could not deserialize the WriteResult object
• Could not deserialize the WriteResult object

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f3bfeb23d5f910a8. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/StructLikeSerializer.java:121

        boolean isNull = dis.readBoolean();
        if (isNull) {
          partition.set(i, null);
        } else {
          int length = dis.readInt();
          byte[] bytes = new byte[length];
          dis.readFully(bytes);
          Object value = Conversions.fromByteBuffer(fields.get(i).type(), ByteBuffer.wrap(bytes));
          // Conversions returns CharBuffer for STRING; PartitionData (and the manifest writer)
          // expects String.
          if (value instanceof CharSequence cs && !(value instanceof String)) {
            value = cs.toString();
          }

          partition.set(i, value);
        }
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to decode partition", e);
    }

    return partition;
  }

  private void writeField(StructLike struct, int pos, Type fieldType) throws IOException {
    Object value = struct.get(pos, Object.class);
    if (value == null) {
      dos.writeBoolean(true);
      return;
    }

    dos.writeBoolean(false);
    ByteBuffer buf = Conversions.toByteBuffer(fieldType, value);
    dos.writeInt(buf.remaining());
    dos.write(buf.array(), buf.arrayOffset() + buf.position(), buf.remaining());
  }
}

View on GitHub (pinned to 86d9c8fc54)