apache/beam · error · UnknownPartitionException

Min and max transformed values were not equal, for column

Error message

Min and max transformed values were not equal, for column: {columnName}

What it means

getPartitionFromMetrics requires that, after applying the partition transform, the transformed min and max of each partition column are equal; otherwise the DataFile spans multiple partitions and no single partition can be assigned. The library throws UnknownPartitionException naming the column.

Solutions

Re-partition/split the source files by the partition key before invoking AddFiles (e.g. sort-and-rewrite with Spark)
Use a coarser partition transform that matches the file granularity so min==max per file
Handle UnknownPartitionException by routing such files to a manual/repair workflow instead of failing the whole job
Rewrite data files first with rewrite_data_files to consolidate values per partition

Example fix

// before: files contain multiple days, transform days(ts)
AddFiles.applicableFormat(...)
// after: coarsen transform to months(ts), or rewrite files per-partition first
spark.sql("CALL cat.system.rewrite_data_files(table => 'db.t', strategy => 'sort', sort_order => 'ts')")

Defensive patterns

Strategy: try-catch

Validate before calling

if (lower!=null && upper!=null && !Objects.equals(transform.apply(lower), transform.apply(upper))) routeToFallback(file);

Type guard

null

Try / catch

try { partition = getPartitionFromMetrics(...); } catch (UnknownPartitionException e) { splitFile(file); }

Prevention

Sort source data by partition key before AddFiles
Match partition transform granularity to file contents
Rewrite files per-partition before import
Provide a fallback path for multi-partition files

When it happens

Trigger: Running AddFiles on a DataFile whose rows span multiple values of a partition column (e.g. a day-granularity file containing rows from two dates under a days(ts) partition transform).

Common situations: Importing legacy/unpartitioned files whose contents were not pre-sorted by partition key; overly fine-grained partition transforms (hours(ts)) on coarsely written files; month/day transforms applied to long-lived files.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/247a4c1239f6718b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:613

        // Make a best effort estimate by comparing the lower and upper transformed values.
        // If the transformed values are equal, assume that the DataFile's data safely
        // aligns with the same partition.
        ByteBuffer lowerBytes = partitionMetrics.lowerBounds().get(field.sourceId());
        ByteBuffer upperBytes = partitionMetrics.upperBounds().get(field.sourceId());
        if (lowerBytes == null && upperBytes == null) {
          continue;
        } else if (lowerBytes == null || upperBytes == null) {
          throw new UnknownPartitionException(
              "Only one of the min/max was was null, for field "
                  + table.schema().findColumnName(field.sourceId()));
        }
        Object lowerTransformedValue = transformValue(transform, type, lowerBytes);
        Object upperTransformedValue = transformValue(transform, type, upperBytes);

        if (!Objects.deepEquals(lowerTransformedValue, upperTransformedValue)) {
          // The DataFile contains values that align to different partitions, so we cannot
          // safely determine a partition.
          throw new UnknownPartitionException(
              "Min and max transformed values were not equal, for column: " + field.name());
        }

        pk.set(i, lowerTransformedValue);
      }

      return pk.toPath();
    }
  }

  /**
   * Writes batches of {@link SerializableDataFile}s (grouped by Partition Spec ID) into {@link
   * ManifestFile}s.
   *
   * <p>Returns the byte-encoded {@link ManifestFile}, to be reconstructed and committed by
   * downstream {@link CommitManifestFilesDoFn}.
   */
  static class CreateManifests

View on GitHub (pinned to 12126d8942)