{"record":{"id":"24a58c703560553c","repo":"apache/beam","slug":"cannot-serialize-datafile-its-partition-spec-id-does-not","errorCode":null,"errorMessage":"Cannot serialize DataFile: its partition spec id {} does not match the provided spec id {}. Serialize the file with the exact spec it was written with.","messagePattern":"Cannot serialize DataFile: its partition spec id (.+?) does not match the provided spec id (.+?)\\. Serialize the file with the exact spec it was written with\\.","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDataFile.java","lineNumber":187,"sourceCode":"        checkStateNotNull(\n            specs.get(f.specId()),\n            \"Could not create a SerializableDataFile because DataFile is written using a partition spec id '%s' that is not found in the provided specs: %s\",\n            f.specId(),\n            specs.keySet()),\n        true);\n  }\n\n  public static SerializableDataFile from(DataFile f, PartitionSpec spec) {\n    return from(f, spec, true);\n  }\n\n  /**\n   * Create a {@link SerializableDataFile} from a {@link DataFile} and its associated {@link\n   * PartitionKey}.\n   */\n  public static SerializableDataFile from(DataFile f, PartitionSpec spec, boolean includeMetrics) {\n    if (spec.specId() != f.specId()) {\n      throw new IllegalArgumentException(\n          String.format(\n              \"Cannot serialize DataFile: its partition spec id %s does not match the provided \"\n                  + \"spec id %s. Serialize the file with the exact spec it was written with.\",\n              f.specId(), spec.specId()));\n    }\n    // jsonPartition is the primary (handles evolved specs, special characters).\n    // partitionPath is the fallback for values that don't round-trip through JSON.\n    String jsonPartition = SingleValueParser.toJson(spec.partitionType(), f.partition());\n    String partitionPath = spec.partitionToPath(f.partition());\n\n    SerializableDataFile.Builder builder =\n        SerializableDataFile.builder()\n            .setPath(f.location())\n            .setFileFormat(f.format().toString())\n            .setRecordCount(f.recordCount())\n            .setFileSizeInBytes(f.fileSizeInBytes())\n            .setPartitionPath(partitionPath)\n            .setJsonPartition(jsonPartition)","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableDataFile.java#L169-L205","documentation":"SerializableDataFile.from() validates that the PartitionSpec supplied for serialization is the exact spec the DataFile was written with, comparing spec.specId() to f.specId(). If they differ it throws IllegalArgumentException, because serializing with the wrong spec would corrupt the partition metadata. This guards against schema/spec evolution mismatches.","triggerScenarios":"Calling SerializableDataFile.from(dataFile, spec, includeMetrics) where the spec's id does not match dataFile.specId(), e.g. passing table.spec() (current spec) for a file written under an older, evolved spec.","commonSituations":"Tables with evolved partition specs (partition evolution) where code assumes all files use the current spec; iterating table.files() and pairing every file with the latest spec; caching one spec object for a whole scan task.","solutions":["Look up each file's spec by its specId from table.specs() (e.g. table.specs().get(f.specId())) instead of using the current spec.","Iterate scan tasks/partitions and fetch the matching spec from the table's spec map.","After partition evolution, refresh the table and never reuse a stale spec id."],"exampleFix":"// before\nSerializableDataFile.from(dataFile, table.spec(), true);\n// after\nSerializableDataFile.from(dataFile, table.specs().get(dataFile.specId()), true);","handlingStrategy":"validation","validationCode":"if (spec.specId() != dataFile.specId()) {\n  spec = table.specs().get(dataFile.specId());\n  if (spec == null) throw new IllegalStateException(\"No spec found for file specId \" + dataFile.specId());\n}","typeGuard":null,"tryCatchPattern":"try {\n  SerializableDataFile s = SerializableDataFile.from(dataFile, spec, true);\n} catch (IllegalArgumentException e) {\n  // re-resolve spec by the file's own specId and retry\n}","preventionTips":["Never pass table.spec() blindly; always resolve per-file via table.specs().get(file.specId()).","After partition evolution, refresh the table before serializing files.","Add unit tests covering tables with multiple partition specs."],"tags":["iceberg","java","serialization","partition-spec"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}