apache/beam · error · IllegalStateException

columnsMapping '%s' does not fit to schema field names '%s'

Error message

columnsMapping '%s' does not fit to schema field names '%s'

What it means

The set of qualifiers in columnsMapping must equal the set of non-key schema field names exactly. validateColumnsMappingFields throws IllegalStateException when the two sets differ — even if counts match, names must correspond one-to-one.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTable.java:206

    int qualifiersCount = schema.getFieldCount() - 1;
    if (qualifiersCount != mappingCount) {
      throw new IllegalStateException(
          String.format(
              "Schema fields count: '%s' does not fit columnsMapping count: '%s'",
              qualifiersCount, mappingCount));
    }
  }

  private static void validateColumnsMappingFields(
      Map<String, Set<String>> columnsMapping, Schema schema) {
    Set<String> allMappingQualifiers =
        columnsMapping.values().stream().flatMap(Collection::stream).collect(toSet());

    Set<String> schemaFieldNames =
        schema.getFieldNames().stream().filter(field -> !KEY.equals(field)).collect(toSet());

    if (!schemaFieldNames.equals(allMappingQualifiers)) {
      throw new IllegalStateException(
          String.format(
              "columnsMapping '%s' does not fit to schema field names '%s'",
              allMappingQualifiers, schemaFieldNames));
    }
  }

  private BigtableIO.Read readTransform() {
    BigtableIO.Read readTransform =
        BigtableIO.read().withProjectId(projectId).withInstanceId(instanceId).withTableId(tableId);
    if (!emulatorHost.isEmpty()) {
      readTransform = readTransform.withEmulator(emulatorHost);
    }
    return readTransform;
  }

  private PTransform<PCollection<com.google.bigtable.v2.Row>, PCollection<Row>> bigtableRowToRow() {
    return useFlatSchema
        ? new BigtableRowToBeamRowFlat(schema, columnsMapping)

View on GitHub (pinned to 12126d8942)

Solutions

1. Rename mapped qualifiers (or schema fields) so the two sets are identical, ignoring the 'key' field.
2. Compare both lists (schema fields minus key vs all columnsMapping qualifiers) and fix typos.
3. If the Bigtable qualifier must differ from the SQL column name, flat-schema mode does not allow it — rename the column in Bigtable or use a non-flat access path via BigtableIO instead.

Example fix

-- before
CREATE EXTERNAL TABLE bt (key STRING, userName STRING)
TBLPROPERTIES '{"columnsMapping": {"cf": ["user_name"]}}'

-- after
CREATE EXTERNAL TABLE bt (key STRING, user_name STRING)
TBLPROPERTIES '{"columnsMapping": {"cf": ["user_name"]}}'

Set<String> mapped = mapping.values().stream().flatMap(Set::stream).collect(java.util.stream.Collectors.toSet());
Set<String> fields = new HashSet<>(java.util.Arrays.asList(schema.getFieldNames()));
fields.remove("key");
if (!fields.equals(mapped)) {
  throw new IllegalArgumentException("columnsMapping qualifiers must exactly match schema fields (excluding key)");
}

try {
  BigtableTable t = new BigtableTable(table);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("does not fit to schema field names")) {
    // rename qualifiers/fields until the sets match
  }
}

When it happens

Trigger: A mapped qualifier name does not match any schema field name (or vice versa): typos in either the schema or columnsMapping, a field renamed in one place only, or mapping to Bigtable qualifier names that differ from schema field names.

Common situations: Schema field 'user_name' mapped as qualifier 'userName'; adding a field to schema and reusing an old mapping; expecting qualifier names in Bigtable to differ from SQL column names (flat schema requires them to be identical sets).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Schema fields count: '%s' does not fit columnsMapping count:
• LOCATION is required
• Schema has to contain '%s' field
• key field type should be STRING but was %s
• The specified 'event-time.timestamp-column' ('%s') does not

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b2fb77e8318b6263. Report an issue: GitHub.