apache/beam · error · IllegalStateException

Schema fields count: '%s' does not fit columnsMapping count:

Error message

Schema fields count: '%s' does not fit columnsMapping count: '%s'

What it means

In flat schema mode the TBLPROPERTIES columnsMapping must map exactly one Bigtable qualifier per non-key schema field. validateColumnsMappingCount sums all mapped qualifiers and compares with schema field count minus the key; a mismatch throws IllegalStateException.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTable.java:190

              + " 'googleapis.com/bigtable/projects/projectId/instances/instanceId/tables/tableId'"
              + " but was: "
              + location);
    }
  }

  private static void validateColumnsMapping(
      Map<String, Set<String>> columnsMapping, Schema schema) {
    validateColumnsMappingCount(columnsMapping, schema);
    validateColumnsMappingFields(columnsMapping, schema);
  }

  private static void validateColumnsMappingCount(
      Map<String, Set<String>> columnsMapping, Schema schema) {
    int mappingCount = columnsMapping.values().stream().mapToInt(Set::size).sum();
    // Don't count the key field
    int qualifiersCount = schema.getFieldCount() - 1;
    if (qualifiersCount != mappingCount) {
      throw new IllegalStateException(
          String.format(
              "Schema fields count: '%s' does not fit columnsMapping count: '%s'",
              qualifiersCount, mappingCount));
    }
  }

  private static void validateColumnsMappingFields(
      Map<String, Set<String>> columnsMapping, Schema schema) {
    Set<String> allMappingQualifiers =
        columnsMapping.values().stream().flatMap(Collection::stream).collect(toSet());

    Set<String> schemaFieldNames =
        schema.getFieldNames().stream().filter(field -> !KEY.equals(field)).collect(toSet());

    if (!schemaFieldNames.equals(allMappingQualifiers)) {
      throw new IllegalStateException(
          String.format(
              "columnsMapping '%s' does not fit to schema field names '%s'",

View on GitHub (pinned to 12126d8942)

Solutions

Make columnsMapping contain exactly one qualifier per non-key schema field, matching counts on both sides.
After changing the schema, update the columnsMapping TBLPROPERTIES in the same change.
Remove the 'key' field from columnsMapping (it is never counted as a qualifier).

Example fix

-- before (schema has key, val1, val2; mapping only has 1 qualifier)
TBLPROPERTIES '{"columnsMapping": {"cf": ["val1"]}}'

-- after
TBLPROPERTIES '{"columnsMapping": {"cf": ["val1", "val2"]}}'

Defensive patterns

Strategy: validation

Validate before calling

Schema schema = table.getSchema();
Map<String, Set<String>> mapping = parseColumnsMapping(props);
int qualifiers = mapping.values().stream().mapToInt(Set::size).sum();
if (schema.getFieldCount() - 1 != qualifiers) {
  throw new IllegalArgumentException("columnsMapping qualifier count must equal schema fields minus the key");
}

Try / catch

try {
  BigtableTable t = new BigtableTable(table);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("does not fit columnsMapping count")) {
    // align schema fields and columnsMapping, then retry
  }
}

Prevention

Change schema and columnsMapping together in one commit/DDL update.
Never list the 'key' field inside columnsMapping.
Write a startup validation that compares field count minus key with mapping size.

When it happens

Trigger: Total number of qualifiers across all column families in columnsMapping differs from (number of schema fields - 1): extra mapped qualifiers, missing ones, or schema fields added/removed without updating columnsMapping.

Common situations: Adding a new column to the schema but forgetting to add it to columnsMapping; listing the same qualifier twice across families inflating the count; including the 'key' field in columnsMapping.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/588893149a510a57. Report an issue: GitHub.