apache/beam · error · IllegalStateException

Schema with id not found in schemas map

Error message

Schema with id {} not found in schemas map

What it means

SerializableTableSpec.getSchema() looks up the table's current schema id in the serialized schemas map and throws IllegalStateException when it is absent. The schemas map is a snapshot taken from the table at serialization time, so a missing schema id means the serialized spec state is incomplete or inconsistent.

Solutions

1. Rebuild the SerializableTableSpec from the current table via the standard from(Table) factory instead of hand-constructing it.
2. Verify the schemas map keys include the schemaId stored on the object.
3. Refresh the table (table.refresh()) and re-serialize after schema evolution.
4. Check for mixed connector versions between the code producing and consuming the spec.

Example fix

// before
SerializableTableSpec spec = new SerializableTableSpec(); // schemas not populated
// after
SerializableTableSpec spec = SerializableTableSpec.from(table); // schemas map populated from table.schemas()

Schema schema = spec.getSchemas().get(spec.getSchemaId());
if (schema == null) {
  // rebuild from the table instead of proceeding
  spec = SerializableTableSpec.from(table);
}

try {
  Schema schema = spec.getSchema();
} catch (IllegalStateException e) {
  // rebuild spec from the current table and retry once
}

When it happens

Trigger: Calling getSchema() on a SerializableTableSpec whose schemas map does not contain getSchemaId() — e.g. a spec built with an inconsistent or truncated schemas map, or deserialized from an older writer that stored a different schema id.

Common situations: Schema evolution between serialization and deserialization with a hand-built or stale SerializableTableSpec; corrupted checkpoint state; constructing the object in tests without populating all schemas.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• PartitionSpec with id
• SortOrder with id not found in sortOrders map
• Cannot serialize DataFile: its partition spec id
• Cannot serialize DeleteFile: its partition spec id
• Could not encode message as bytes

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/55c85a872277a355. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableTableSpec.java:134

      synchronized (this) {
        local = cachedSchemas;
        if (local == null) {
          ImmutableMap.Builder<Integer, Schema> builder = ImmutableMap.builder();
          for (Map.Entry<Integer, String> entry : getSchemasJson().entrySet()) {
            builder.put(entry.getKey(), SchemaParser.fromJson(entry.getValue()));
          }
          cachedSchemas = local = builder.build();
        }
      }
    }
    return local;
  }

  @SchemaIgnore
  public Schema getSchema() {
    Schema schema = getSchemas().get(getSchemaId());
    if (schema == null) {
      throw new IllegalStateException(
          "Schema with id " + getSchemaId() + " not found in schemas map");
    }
    return schema;
  }

  @SchemaIgnore
  public @Nullable Schema getSchema(int schemaId) {
    return getSchemas().get(schemaId);
  }

  @SchemaIgnore
  public Map<Integer, PartitionSpec> getPartitionSpecs() {
    Map<Integer, PartitionSpec> local = cachedPartitionSpecs;
    if (local == null) {
      synchronized (this) {
        local = cachedPartitionSpecs;
        if (local == null) {
          ImmutableMap.Builder<Integer, PartitionSpec> builder = ImmutableMap.builder();

View on GitHub (pinned to 12126d8942)