apache/beam · error · IllegalArgumentException

Cannot serialize DataFile: its partition spec id

Error message

Cannot serialize DataFile: its partition spec id {} does not match the provided spec id {}. Serialize the file with the exact spec it was written with.

What it means

SerializableDataFile.from() validates that the PartitionSpec supplied for serialization is the exact spec the DataFile was written with, comparing spec.specId() to f.specId(). If they differ it throws IllegalArgumentException, because serializing with the wrong spec would corrupt the partition metadata. This guards against schema/spec evolution mismatches.

Solutions

Look up each file's spec by its specId from table.specs() (e.g. table.specs().get(f.specId())) instead of using the current spec.
Iterate scan tasks/partitions and fetch the matching spec from the table's spec map.
After partition evolution, refresh the table and never reuse a stale spec id.

Example fix

// before
SerializableDataFile.from(dataFile, table.spec(), true);
// after
SerializableDataFile.from(dataFile, table.specs().get(dataFile.specId()), true);

Defensive patterns

Strategy: validation

Validate before calling

if (spec.specId() != dataFile.specId()) {
  spec = table.specs().get(dataFile.specId());
  if (spec == null) throw new IllegalStateException("No spec found for file specId " + dataFile.specId());
}

Try / catch

try {
  SerializableDataFile s = SerializableDataFile.from(dataFile, spec, true);
} catch (IllegalArgumentException e) {
  // re-resolve spec by the file's own specId and retry
}

Prevention

Never pass table.spec() blindly; always resolve per-file via table.specs().get(file.specId()).
After partition evolution, refresh the table before serializing files.
Add unit tests covering tables with multiple partition specs.

When it happens

Trigger: Calling SerializableDataFile.from(dataFile, spec, includeMetrics) where the spec's id does not match dataFile.specId(), e.g. passing table.spec() (current spec) for a file written under an older, evolved spec.

Common situations: Tables with evolved partition specs (partition evolution) where code assumes all files use the current spec; iterating table.files() and pairing every file with the latest spec; caching one spec object for a whole scan task.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/24a58c703560553c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDataFile.java:187

        checkStateNotNull(
            specs.get(f.specId()),
            "Could not create a SerializableDataFile because DataFile is written using a partition spec id '%s' that is not found in the provided specs: %s",
            f.specId(),
            specs.keySet()),
        true);
  }

  public static SerializableDataFile from(DataFile f, PartitionSpec spec) {
    return from(f, spec, true);
  }

  /**
   * Create a {@link SerializableDataFile} from a {@link DataFile} and its associated {@link
   * PartitionKey}.
   */
  public static SerializableDataFile from(DataFile f, PartitionSpec spec, boolean includeMetrics) {
    if (spec.specId() != f.specId()) {
      throw new IllegalArgumentException(
          String.format(
              "Cannot serialize DataFile: its partition spec id %s does not match the provided "
                  + "spec id %s. Serialize the file with the exact spec it was written with.",
              f.specId(), spec.specId()));
    }
    // jsonPartition is the primary (handles evolved specs, special characters).
    // partitionPath is the fallback for values that don't round-trip through JSON.
    String jsonPartition = SingleValueParser.toJson(spec.partitionType(), f.partition());
    String partitionPath = spec.partitionToPath(f.partition());

    SerializableDataFile.Builder builder =
        SerializableDataFile.builder()
            .setPath(f.location())
            .setFileFormat(f.format().toString())
            .setRecordCount(f.recordCount())
            .setFileSizeInBytes(f.fileSizeInBytes())
            .setPartitionPath(partitionPath)
            .setJsonPartition(jsonPartition)

View on GitHub (pinned to 12126d8942)