apache/beam · error · IllegalArgumentException

Cannot serialize DeleteFile: its partition spec id

Error message

Cannot serialize DeleteFile: its partition spec id {} does not match the provided spec id {}.

What it means

SerializableDeleteFile.from() mirrors SerializableDataFile: it requires the provided PartitionSpec's id to equal the DeleteFile's specId and throws IllegalArgumentException otherwise. Delete files (equality or position deletes) carry the spec id they were written under; serializing with a mismatched spec would misplace the delete in partition space.

Solutions

Resolve the spec via the table's spec map: table.specs().get(deleteFile.specId()).
When listing delete files via scans, carry the spec associated with the scan task rather than table.spec().
Refresh the table/collection point so stale spec ids are re-resolved after evolution.

Example fix

// before
SerializableDeleteFile.from(deleteFile, table.spec(), true);
// after
SerializableDeleteFile.from(deleteFile, table.specs().get(deleteFile.specId()), true);

Defensive patterns

Strategy: validation

Validate before calling

if (spec.specId() != deleteFile.specId()) {
  spec = table.specs().get(deleteFile.specId());
  if (spec == null) throw new IllegalStateException("No spec found for delete file specId " + deleteFile.specId());
}

Try / catch

try {
  SerializableDeleteFile s = SerializableDeleteFile.from(deleteFile, spec, true);
} catch (IllegalArgumentException e) {
  // re-resolve spec by deleteFile.specId() and retry
}

Prevention

Resolve specs per delete file via table.specs().get(deleteFile.specId()).
Refresh table snapshots before serializing delete files in maintenance jobs.
Test round-trips on tables that evolved their partition spec.

When it happens

Trigger: Calling SerializableDeleteFile.from(deleteFile, spec, includeMetrics) with a spec whose specId differs from deleteFile.specId(), commonly using table.spec() for delete files created under an earlier, evolved partition spec.

Common situations: Tables that evolved their partition spec while having delete files from the old spec; pairing all delete files with the table's current spec during a compaction/cleanup job; deserializing plan results from a checkpointed older table version.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b6a82dcca7395a5c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDeleteFile.java:210

      DeleteFile deleteFile, Map<Integer, PartitionSpec> specs, boolean includeMetrics) {
    return from(
        deleteFile,
        checkStateNotNull(
            specs.get(deleteFile.specId()),
            "Could not create a SerializableDeleteFile because DeleteFile is written using a partition spec id '%s' that is not found in the provided specs: %s",
            deleteFile.specId(),
            specs.keySet()),
        includeMetrics);
  }

  public static SerializableDeleteFile from(DeleteFile deleteFile, PartitionSpec spec) {
    return from(deleteFile, spec, true);
  }

  public static SerializableDeleteFile from(
      DeleteFile deleteFile, PartitionSpec spec, boolean includeMetrics) {
    if (spec.specId() != deleteFile.specId()) {
      throw new IllegalArgumentException(
          String.format(
              "Cannot serialize DeleteFile: its partition spec id %s does not match the provided "
                  + "spec id %s.",
              deleteFile.specId(), spec.specId()));
    }
    // jsonPartition is the primary (handles evolved specs, special characters).
    // partitionPath is the fallback for values that don't round-trip through JSON.
    String jsonPartition = SingleValueParser.toJson(spec.partitionType(), deleteFile.partition());
    String partitionPath = spec.partitionToPath(deleteFile.partition());

    SerializableDeleteFile.Builder builder =
        SerializableDeleteFile.builder()
            .setLocation(deleteFile.location())
            .setFileFormat(deleteFile.format().name())
            .setFileSizeInBytes(deleteFile.fileSizeInBytes())
            .setPartitionPath(partitionPath)
            .setJsonPartition(jsonPartition)
            .setPartitionSpecId(deleteFile.specId())

View on GitHub (pinned to 12126d8942)