apache/iceberg · error · UncheckedIOException

Failed to decode partition

Error message

Failed to decode partition

What it means

decodePartition() reads back bytes produced by encodePartition() into a PartitionData. IOExceptions (EOFException, corrupt data, wrong format) are wrapped in UncheckedIOException. It means the byte array does not match the expected layout: a null flag, length, and value per field in the given partitionType order.

Solutions

1. Ensure encode and decode use the same Types.StructType partitionType (from the same table spec version)
2. Verify the byte[] is not truncated — check the source of the bytes (state, message, event)
3. Rebuild state/reprocess the affected partition if state was written before a spec change
4. Add a defensive check that encoded.length > 0 before decodePartition for non-empty partition types

Example fix

// before: decoding with a changed spec
StructLike p = StructLikeSerializer.decodePartition(bytes, table.spec().partitionType());
// after: decode with the spec active when the bytes were written
StructLike p = StructLikeSerializer.decodePartition(bytes, specAtWriteTime.partitionType());

if (partitionType.fields().isEmpty()) {
  return; // empty partition: no bytes expected
}
if (encoded == null || encoded.length == 0) {
  throw new IllegalArgumentException("No encoded bytes for non-empty partition type " + partitionType);
}

try {
  StructLike p = StructLikeSerializer.decodePartition(encoded, partitionType);
} catch (UncheckedIOException e) {
  LOG.error("Corrupt/truncated partition bytes for type {}", partitionType, e);
  throw e;
}

When it happens

Trigger: Decoding bytes encoded with a different partitionType (field count/type changed); truncated or corrupted byte arrays from state; decoding EMPTY_PARTITION bytes with a non-empty partitionType.

Common situations: Partition spec evolution between the job that encoded and the one decoding; state written by a different Iceberg version; deserializing from a state backend after a failed restore.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Fail to deserialize aggregated statistics
• Fail to deserialize aggregated statistics
• Fail to deserialize aggregated statistics,change to v1
• Fail to deserialize aggregated statistics,change to v1
• Fail to deserialize aggregated statistics,change to v1

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1c1e63df42e2f4bc. Report an issue: GitHub.

Appendix: source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/StructLikeSerializer.java:121

        boolean isNull = dis.readBoolean();
        if (isNull) {
          partition.set(i, null);
        } else {
          int length = dis.readInt();
          byte[] bytes = new byte[length];
          dis.readFully(bytes);
          Object value = Conversions.fromByteBuffer(fields.get(i).type(), ByteBuffer.wrap(bytes));
          // Conversions returns CharBuffer for STRING; PartitionData (and the manifest writer)
          // expects String.
          if (value instanceof CharSequence cs && !(value instanceof String)) {
            value = cs.toString();
          }

          partition.set(i, value);
        }
      }
    } catch (IOException e) {
      throw new UncheckedIOException("Failed to decode partition", e);
    }

    return partition;
  }

  private void writeField(StructLike struct, int pos, Type fieldType) throws IOException {
    Object value = struct.get(pos, Object.class);
    if (value == null) {
      dos.writeBoolean(true);
      return;
    }

    dos.writeBoolean(false);
    ByteBuffer buf = Conversions.toByteBuffer(fieldType, value);
    dos.writeInt(buf.remaining());
    dos.write(buf.array(), buf.arrayOffset() + buf.position(), buf.remaining());
  }
}

View on GitHub (pinned to 86d9c8fc54)