apache/iceberg · error · UnsupportedOperationException

Invalid unit for shredded timestamp

Error message

Invalid unit for shredded timestamp: <unit>

What it means

ParquetVariantUtil.convert() maps a Parquet timestamp annotation to a variant PhysicalType, supporting MICROS (TIMESTAMPTZ/TIMESTAMPNTZ) and NANOS (TIMESTAMPTZ_NANOS/TIMESTAMPNTZ_NANOS). Any other unit hits the default branch and throws UnsupportedOperationException naming the unit.

Solutions

1. Rewrite the column's logical type to MICROS or NANOS before shredding
2. Convert MILLIS values to MICROS in a preprocessing step and annotate accordingly
3. Upgrade Iceberg if support for the unit was added in a newer release

Example fix

// before: millis timestamp in shredded variant schema
Types.TimestampType ts = Types.TimestampType.withZone(); // written as MILLIS
// after: normalize to micros
TimestampLogicalTypeAnnotation.timestampType(true, TimeUnit.MICROS)

TimestampLogicalTypeAnnotation ts = (TimestampLogicalTypeAnnotation) annotation;
Preconditions.checkArgument(
    ts.getUnit() == TimeUnit.MICROS || ts.getUnit() == TimeUnit.NANOS,
    "Variant shredded timestamps must be MICROS or NANOS, got %s", ts.getUnit());

boolean isShreddableTimestampUnit(TimestampLogicalTypeAnnotation ts) {
  return ts.getUnit() == TimeUnit.MICROS || ts.getUnit() == TimeUnit.NANOS;
}

try {
  physicalType = ParquetVariantUtil.convert(timestampAnnotation);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().startsWith("Invalid unit for shredded timestamp")) {
    // rewrite the column to micros, or keep the value unshredded
  } else throw e;
}

When it happens

Trigger: Converting a shredded variant timestamp whose TimestampLogicalTypeAnnotation unit is MILLIS (or any unit other than MICROS/NANOS) during variant shredding schema conversion.

Common situations: Files written with millis-precision timestamps shredded into variant columns; older writers emitting MILLIS units; schema defined before variant shredding spec restricted units to micros/nanos.

Related errors

• errMsg(variant, "timestamp")
• Failed to read Variant
• Invalid bit width for int
• Invalid bit width for int
• Invalid bound type

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/83b54b549564e98a. Report an issue: GitHub.

Appendix: source

Thrown at parquet/src/main/java/org/apache/iceberg/parquet/ParquetVariantUtil.java:111

  /**
   * Convert a Parquet {@link TimestampLogicalTypeAnnotation} to the equivalent Variant {@link
   * PhysicalType}.
   *
   * @param timestamp a Parquet {@link TimestampLogicalTypeAnnotation}
   * @return a Variant {@link PhysicalType}
   * @throws UnsupportedOperationException if the timestamp unit is not MICROS or NANOS
   */
  static PhysicalType convert(TimestampLogicalTypeAnnotation timestamp) {
    switch (timestamp.getUnit()) {
      case MICROS:
        return timestamp.isAdjustedToUTC() ? PhysicalType.TIMESTAMPTZ : PhysicalType.TIMESTAMPNTZ;
      case NANOS:
        return timestamp.isAdjustedToUTC()
            ? PhysicalType.TIMESTAMPTZ_NANOS
            : PhysicalType.TIMESTAMPNTZ_NANOS;
      default:
        throw new UnsupportedOperationException(
            "Invalid unit for shredded timestamp: " + timestamp.getUnit());
    }
  }

  /**
   * Serialize Variant metadata and value in a single concatenated buffer.
   *
   * @param metadata a {VariantMetadata}
   * @param value a {VariantValue}
   * @return a buffer containing the metadata and value serialized and concatenated
   */
  static ByteBuffer toByteBuffer(VariantMetadata metadata, VariantValue value) {
    ByteBuffer buffer =
        ByteBuffer.allocate(metadata.sizeInBytes() + value.sizeInBytes())
            .order(ByteOrder.LITTLE_ENDIAN);
    metadata.writeTo(buffer, 0);
    value.writeTo(buffer, metadata.sizeInBytes());
    return buffer;

View on GitHub (pinned to 86d9c8fc54)