apache/beam · error · IllegalStateException

PartitionSpec with id

Error message

PartitionSpec with id {} not found in partitionSpecs map

What it means

SerializableTableSpec.getPartitionSpec() resolves the stored specId against the serialized partitionSpecs map and throws IllegalStateException if absent. Like the schema lookup, the map is a snapshot of table.specs(); a miss means the serialized state is inconsistent or the spec was removed/evolved between snapshot and use.

Solutions

Rebuild via SerializableTableSpec.from(currentTable) so partitionSpecs comes from table.specs().
Confirm the stored specId exists in the table's spec map; if the spec was dropped, remap to the current spec.
Refresh the table and re-serialize state after partition evolution.
Verify producer and consumer use the same connector/table snapshot.

Example fix

// before
long specId = 0; // stale id after partition evolution
SerializableTableSpec spec = ...withSpecId(specId)...;
// after
SerializableTableSpec spec = SerializableTableSpec.from(table); // uses table.currentSpec().specId() with matching map

Defensive patterns

Strategy: validation

Validate before calling

PartitionSpec ps = spec.getPartitionSpecs().get(spec.getSpecId());
if (ps == null) {
  // spec id missing — rebuild from current table or remap to an existing spec
  spec = SerializableTableSpec.from(table);
}

Try / catch

try {
  PartitionSpec ps = spec.getPartitionSpec();
} catch (IllegalStateException e) {
  // rebuild from table / fall back to currentSpec
}

Prevention

Re-serialize specs after any partition evolution.
Don't reuse spec objects across different table snapshots.
Include multi-spec tables in round-trip tests.

When it happens

Trigger: Calling getPartitionSpec() when partitionSpecs lacks the entry for getSpecId(), e.g. deserializing a spec produced by a different table version after partition evolution, or building the object with a specId that was never added.

Common situations: Partition spec evolution making an old specId unavailable; restoring stale checkpoint/beam state; tests constructing the object manually with only some fields populated.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4a518bae7f0537e1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableTableSpec.java:168

        local = cachedPartitionSpecs;
        if (local == null) {
          ImmutableMap.Builder<Integer, PartitionSpec> builder = ImmutableMap.builder();
          for (Map.Entry<Integer, String> entry : getPartitionSpecsJson().entrySet()) {
            builder.put(
                entry.getKey(), PartitionSpecParser.fromJson(getSchema(), entry.getValue()));
          }
          cachedPartitionSpecs = local = builder.build();
        }
      }
    }
    return local;
  }

  @SchemaIgnore
  public PartitionSpec getPartitionSpec() {
    PartitionSpec spec = getPartitionSpecs().get(getSpecId());
    if (spec == null) {
      throw new IllegalStateException(
          "PartitionSpec with id " + getSpecId() + " not found in partitionSpecs map");
    }
    return spec;
  }

  @SchemaIgnore
  public @Nullable PartitionSpec getPartitionSpec(int specId) {
    return getPartitionSpecs().get(specId);
  }

  @SchemaIgnore
  public Map<Integer, SortOrder> getSortOrders() {
    Map<Integer, SortOrder> local = cachedSortOrders;
    if (local == null) {
      synchronized (this) {
        local = cachedSortOrders;
        if (local == null) {
          ImmutableMap.Builder<Integer, SortOrder> builder = ImmutableMap.builder();

View on GitHub (pinned to 12126d8942)