apache/beam · error · RuntimeException

Schema field not found: %s

Error message

Schema field not found: %s

What it means

SchemaInformation.getSchemaForField(String) looks up a nested field by (lowercased) name in subFieldsByName and throws RuntimeException("Schema field not found: ...") when absent. During TableRow-to-proto conversion, every value key must correspond to a field declared in the BigQuery schema.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:752

      return tableFieldSchema.getType();
    }

    public boolean isNullable() {
      return tableFieldSchema.getMode().equals(TableFieldSchema.Mode.NULLABLE);
    }

    public boolean isRepeated() {
      return tableFieldSchema.getMode().equals(TableFieldSchema.Mode.REPEATED);
    }

    public long getTimestampPrecision() {
      return tableFieldSchema.getTimestampPrecision().getValue();
    }

    public SchemaInformation getSchemaForField(String name) {
      SchemaInformation schemaInformation = subFieldsByName.get(name.toLowerCase());
      if (schemaInformation == null) {
        throw new RuntimeException("Schema field not found: " + name.toLowerCase());
      }
      return schemaInformation;
    }

    public SchemaInformation getSchemaForField(int i) {
      SchemaInformation schemaInformation = subFields.get(i);
      if (schemaInformation == null) {
        throw new RuntimeException("Schema field not found: " + i);
      }
      return schemaInformation;
    }

    public static SchemaInformation fromTableSchema(TableSchema tableSchema) {
      TableFieldSchema root =
          TableFieldSchema.newBuilder()
              .addAllFields(tableSchema.getFieldsList())
              .setName("root")
              .build();

View on GitHub (pinned to 12126d8942)

Solutions

  1. Update the destination BigQuery table schema (bq update or auto-detect) to include the missing field.
  2. Fix the TableRow key so it matches the schema field name exactly (case-insensitive after lowercase).
  3. Strip or route unknown fields before the write (the sink partially drops empty unknown fields, but non-empty ones surface this).

Example fix

// before
row.set("userId", id); // schema has "user_id"
// after
row.set("user_id", id);
Defensive patterns

Strategy: validation

Validate before calling

Set<String> schemaFields = schema.getFields().stream()
    .map(f -> f.getName().toLowerCase()).collect(java.util.stream.Collectors.toSet());
for (String key : row.keySet()) {
  if (!schemaFields.contains(key.toLowerCase())) {
    throw new IllegalArgumentException("Row key not in schema: " + key);
  }
}

Try / catch

try {
  convert(row);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Schema field not found")) {
    deadLetter(row); return;
  }
  throw e;
}

Prevention

When it happens

Trigger: Converting a TableRow whose keys contain a field name not present in the destination table's schema (extra/renamed/misspelled keys, wrong-case keys not matching after lowercasing), via fieldSchemaInformation or fieldSchema.

Common situations: Schema drift: upstream data gained a new column not yet added to the BigQuery table; typos in key names; nested record keys mismatched with the declared schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/de5ab1be7ff6a536. Report an issue: GitHub.