apache/beam · error · UnknownPartitionException

Only one of the min/max was was null, for field

Error message

Only one of the min/max was was null, for field {columnName}

What it means

AddFiles.getPartitionFromMetrics derives a DataFile's partition from its lower/upper bound metrics. If exactly one of lowerBounds/upperBounds is null for a partition field, the partition cannot be determined safely, so it throws UnknownPartitionException. The file's statistics are incomplete for that column.

Solutions

Rewrite the affected DataFiles with a full rewrite (rewrite_data_files / RewriteDataFiles) so complete min/max stats are produced
Set write.metadata.metrics.default appropriately (e.g. 'truncate' or 'full') on the table before writing new files
Exclude/skip the offending DataFile or route it to a fallback path instead of auto-replicating to a partition
Inspect the file's metrics via Iceberg's MetadataTableColumns to confirm which column lacks bounds

Example fix

// before
AddFiles.getDefaultTableLoader(catalogConfig)
// after (Hive/park route): first run in Spark
spark.sql("CALL catalog.system.rewrite_data_files(table => 'db.tbl')")

Defensive patterns

Strategy: try-catch

Validate before calling

Metrics m = file.metrics(); if (m.lowerBounds()==null || m.upperBounds()==null || m.lowerBounds().get(fieldId)==null || m.upperBounds().get(fieldId)==null) skip(file);

Type guard

null

Try / catch

try { partition = getPartitionFromMetrics(...); } catch (UnknownPartitionException e) { routeToFallback(file, e); }

Prevention

Enable full metrics on write (write.metadata.metrics.default)
Avoid writing files whose partition columns are all-null
Rewrite legacy files with complete stats
Check table property write.metadata.metrics.default=none

When it happens

Trigger: Processing a DataFile whose metrics (written by the writing engine) lack either a min or a max for a partition-key column; metrics set to null because the column had all-nulls or stats were dropped (write.metadata.metrics.default=none).

Common situations: Tables written by engines with partial statistics collection; partition columns that are entirely NULL in some files; metrics disabled for performance on wide tables; files produced by older Iceberg writers.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e9a79f5978673be0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:603

      }

      PartitionKey pk = new PartitionKey(table.spec(), table.schema());

      // read metadata from footer and set partition based on min/max transformed values
      for (int i = 0; i < fields.size(); i++) {
        PartitionField field = fields.get(i);
        Type type = table.schema().findType(field.sourceId());
        Transform<?, ?> transform = field.transform();

        // Make a best effort estimate by comparing the lower and upper transformed values.
        // If the transformed values are equal, assume that the DataFile's data safely
        // aligns with the same partition.
        ByteBuffer lowerBytes = partitionMetrics.lowerBounds().get(field.sourceId());
        ByteBuffer upperBytes = partitionMetrics.upperBounds().get(field.sourceId());
        if (lowerBytes == null && upperBytes == null) {
          continue;
        } else if (lowerBytes == null || upperBytes == null) {
          throw new UnknownPartitionException(
              "Only one of the min/max was was null, for field "
                  + table.schema().findColumnName(field.sourceId()));
        }
        Object lowerTransformedValue = transformValue(transform, type, lowerBytes);
        Object upperTransformedValue = transformValue(transform, type, upperBytes);

        if (!Objects.deepEquals(lowerTransformedValue, upperTransformedValue)) {
          // The DataFile contains values that align to different partitions, so we cannot
          // safely determine a partition.
          throw new UnknownPartitionException(
              "Min and max transformed values were not equal, for column: " + field.name());
        }

        pk.set(i, lowerTransformedValue);
      }

      return pk.toPath();
    }

View on GitHub (pinned to 12126d8942)