{"record":{"id":"4bc8edf9628e9924","repo":"apache/beam","slug":"couldn-t-convert-schema-for-s","errorCode":null,"errorMessage":"Couldn't convert schema for %s","messagePattern":"Couldn't convert schema for (.+?)","errorType":"validation","errorClass":"SchemaDoesntMatchException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java","lineNumber":1044,"sourceCode":"              Preconditions.checkStateNotNull(changeSequenceNum));\n    }\n\n    if (requiredFieldsRemaining != null && !requiredFieldsRemaining.isEmpty()) {\n      String prefix = schemaInformation.getFullName();\n      Set<String> missingFields =\n          requiredFieldsRemaining.stream()\n              .map(key -> prefix.isEmpty() ? key : String.join(\".\", prefix, key))\n              .collect(Collectors.toSet());\n      SchemaConversionException e = new SchemaMissingRequiredFieldException(missingFields);\n      collectedExceptions.collect(e); // Throws if not collected.\n    }\n    if (!collectedExceptions.isEmpty()) {\n      return null;\n    }\n    try {\n      return Preconditions.checkArgumentNotNull(builder).build();\n    } catch (Exception e) {\n      throw new SchemaDoesntMatchException(\n          \"Couldn't convert schema for \" + schemaInformation.getFullName(), e);\n    }\n  }\n\n  /**\n   * Forwards {@code changeSequenceNum} to {@link #messageFromTableRow(SchemaInformation,\n   * Descriptor, TableRow, boolean, boolean, TableRow, String, String, ErrorCollector)} via {@link\n   * Long#toHexString}.\n   */\n  public static @Nullable DynamicMessage messageFromTableRow(\n      SchemaInformation schemaInformation,\n      @Nullable Descriptor descriptor,\n      TableRow tableRow,\n      boolean ignoreUnknownValues,\n      boolean allowMissingRequiredFields,\n      final @Nullable TableRow unknownFields,\n      @Nullable String changeType,\n      long changeSequenceNum,","sourceCodeStart":1026,"sourceCodeEnd":1062,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L1026-L1062","documentation":"TableRowToStorageApiProto wraps any exception raised while building the final protobuf DynamicMessage for a row into a SchemaDoesntMatchException, adding the full table name. It means the TableRow did not match the BigQuery table's schema at the top level (after per-field errors were collected). The underlying cause is chained as the exception's cause.","triggerScenarios":"Calling the BigQueryIO Storage Write API sink (via tableRowToStorageApiProto / messageFromTableRow) when one or more fields failed schema conversion, so the builder was never produced, or DynamicMessage.build() itself rejected an unset required field.","commonSituations":"Rows missing required (non-nullable) columns; nested row structures not matching the declared RECORD schema; value types (e.g. Long vs String for INT64) that converters reject; schema drift where the table was altered but the Beam schema or data wasn't updated.","solutions":["Log the chained cause (e.getCause()) to find the exact failing field.","Validate that every non-nullable schema field is present and non-null in each TableRow.","Align nested TableRow maps with the declared schema (field names lowercased, matching RECORD types).","Re-read the table schema (bigquery.tables().get) and rebuild the Beam schema before writing.","Enable getNestedUnknown handling or update the pipeline schema to match actual data."],"exampleFix":"// before\nrow.get(\"created_at\") may be null for a REQUIRED field\n// after\nif (row.get(\"created_at\") == null) { row.set(\"created_at\", Instant.now().toString()); }","handlingStrategy":"validation","validationCode":"List<String> missing = schema.getFields().stream().filter(f -> !f.getMode().equals(\"NULLABLE\") && row.get(f.getName().toLowerCase()) == null).map(Field::getName).collect(toList()); if (!missing.isEmpty()) throw new IllegalArgumentException(\"Missing required fields: \" + missing);","typeGuard":"boolean matchesSchema(Map<String,Object> row, TableSchema schema) { return schema.getFields().stream().allMatch(f -> row.containsKey(f.getName().toLowerCase()) || \"NULLABLE\".equals(f.getMode())); }","tryCatchPattern":"try { ProtoRow proto = TableRowToStorageApiProto.tableRowToProto(tableSchema, row); } catch (SchemaDoesntMatchException e) { LOG.error(\"Row rejected for table {}: cause={}\", table, e.getCause(), e); deadLetter(row, e); }","preventionTips":["Always fetch the live table schema instead of maintaining a copy","Keep field-name casing consistent (BQ schema names are lowercased in proto descriptors)","Validate a sample of rows against the schema before committing the pipeline","Log SchemaDoesntMatchException.getCause() — the root reason is chained"],"tags":["bigquery","schema","java","storage-write-api"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}