apache/beam · error · IllegalStateException

Schema has to contain '%s' field

Error message

Schema has to contain '%s' field

What it means

Every Bigtable SQL table schema must contain a 'key' field, which becomes the Bigtable row key. validateSchema throws IllegalStateException when schema.hasField(KEY) is false, i.e. the declared schema lacks the mandatory 'key' column.

Source

Thrown at sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTable.java:158

    Splitter.on(",")
        .splitToList(commaSeparatedMapping)
        .forEach(
            colonSeparatedValues -> {
              List<String> pair = Splitter.on(":").splitToList(colonSeparatedValues);
              columnsMapping.putIfAbsent(pair.get(0), newHashSet());
              columnsMapping.get(pair.get(0)).add(pair.get(1));
            });
    return columnsMapping;
  }

  private static String getMatcherValue(Matcher matcher, String field) {
    String value = matcher.group(field);
    return value == null ? "" : value;
  }

  private static void validateSchema(Schema schema) {
    if (!schema.hasField(KEY)) {
      throw new IllegalStateException(String.format("Schema has to contain '%s' field", KEY));
    } else {
      Schema.Field keyField = schema.getField(KEY);
      if (keyField != null && !(Schema.TypeName.STRING == keyField.getType().getTypeName())) {
        throw new IllegalArgumentException(
            "key field type should be STRING but was " + keyField.getType().getTypeName());
      }
    }
  }

  private static void validateMatcher(Matcher matcher, String location) {
    if (!matcher.matches()) {
      throw new InvalidTableException(
          "Bigtable location must be in the following format:"
              + " 'googleapis.com/bigtable/projects/projectId/instances/instanceId/tables/tableId'"
              + " but was: "
              + location);
    }
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Add a 'key' field of type STRING to the table schema: CREATE EXTERNAL TABLE ... (key STRING, ...).
2. Rename your intended row-key column to exactly 'key' in the schema definition.
3. If the key lives under a different name in source data, alias it to 'key' in the columnsMapping TBLPROPERTIES is not enough — the schema field itself must be named 'key'.

Example fix

-- before
CREATE EXTERNAL TABLE bt (id STRING, cf_val STRING) TYPE 'bigtable' LOCATION '...';

-- after
CREATE EXTERNAL TABLE bt (key STRING, cf_val STRING) TYPE 'bigtable' LOCATION '...';

Schema schema = table.getSchema();
if (!schema.hasField("key")) {
  throw new IllegalArgumentException("Bigtable table schema must contain a 'key' field");
}

try {
  BigtableTable t = new BigtableTable(table);
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Schema has to contain")) {
    // add the 'key' field to the schema before retrying
  }
}

When it happens

Trigger: Declaring a Bigtable external table whose schema has no field named 'key' — the constructor's validateSchema fails before the table can be used for read or write.

Common situations: Authoring CREATE EXTERNAL TABLE DDL for bigtable provider and forgetting the 'key' column; renaming the key column to something like 'rowkey' or 'id'; auto-generated schemas from data that omitted the row-key field.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• key field type should be STRING but was %s
• Schema fields count: '%s' does not fit columnsMapping count:
• columnsMapping '%s' does not fit to schema field names '%s'
• Unsupported type: {fieldType}
• LOCATION is required

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c40533c546f99f1e. Report an issue: GitHub.