{"record":{"id":"de5ab1be7ff6a536","repo":"apache/beam","slug":"schema-field-not-found-s","errorCode":null,"errorMessage":"Schema field not found: %s","messagePattern":"Schema field not found: (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java","lineNumber":752,"sourceCode":"      return tableFieldSchema.getType();\n    }\n\n    public boolean isNullable() {\n      return tableFieldSchema.getMode().equals(TableFieldSchema.Mode.NULLABLE);\n    }\n\n    public boolean isRepeated() {\n      return tableFieldSchema.getMode().equals(TableFieldSchema.Mode.REPEATED);\n    }\n\n    public long getTimestampPrecision() {\n      return tableFieldSchema.getTimestampPrecision().getValue();\n    }\n\n    public SchemaInformation getSchemaForField(String name) {\n      SchemaInformation schemaInformation = subFieldsByName.get(name.toLowerCase());\n      if (schemaInformation == null) {\n        throw new RuntimeException(\"Schema field not found: \" + name.toLowerCase());\n      }\n      return schemaInformation;\n    }\n\n    public SchemaInformation getSchemaForField(int i) {\n      SchemaInformation schemaInformation = subFields.get(i);\n      if (schemaInformation == null) {\n        throw new RuntimeException(\"Schema field not found: \" + i);\n      }\n      return schemaInformation;\n    }\n\n    public static SchemaInformation fromTableSchema(TableSchema tableSchema) {\n      TableFieldSchema root =\n          TableFieldSchema.newBuilder()\n              .addAllFields(tableSchema.getFieldsList())\n              .setName(\"root\")\n              .build();","sourceCodeStart":734,"sourceCodeEnd":770,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L734-L770","documentation":"SchemaInformation.getSchemaForField(String) looks up a nested field by (lowercased) name in subFieldsByName and throws RuntimeException(\"Schema field not found: ...\") when absent. During TableRow-to-proto conversion, every value key must correspond to a field declared in the BigQuery schema.","triggerScenarios":"Converting a TableRow whose keys contain a field name not present in the destination table's schema (extra/renamed/misspelled keys, wrong-case keys not matching after lowercasing), via fieldSchemaInformation or fieldSchema.","commonSituations":"Schema drift: upstream data gained a new column not yet added to the BigQuery table; typos in key names; nested record keys mismatched with the declared schema.","solutions":["Update the destination BigQuery table schema (bq update or auto-detect) to include the missing field.","Fix the TableRow key so it matches the schema field name exactly (case-insensitive after lowercase).","Strip or route unknown fields before the write (the sink partially drops empty unknown fields, but non-empty ones surface this)."],"exampleFix":"// before\nrow.set(\"userId\", id); // schema has \"user_id\"\n// after\nrow.set(\"user_id\", id);","handlingStrategy":"validation","validationCode":"Set<String> schemaFields = schema.getFields().stream()\n    .map(f -> f.getName().toLowerCase()).collect(java.util.stream.Collectors.toSet());\nfor (String key : row.keySet()) {\n  if (!schemaFields.contains(key.toLowerCase())) {\n    throw new IllegalArgumentException(\"Row key not in schema: \" + key);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  convert(row);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Schema field not found\")) {\n    deadLetter(row); return;\n  }\n  throw e;\n}","preventionTips":["Keep table schema and row-producing code in sync; add new columns to the table first.","Run schema-conformance checks in CI against the live BigQuery schema.","Drop or dead-letter unknown fields before the write."],"tags":["java","beam","bigquery","schema","field-not-found"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}