apache/beam · warning

Failure getting proto descriptor. Successful output will…

Error message

Failure getting proto descriptor. Successful output will not be produced.

What it means

In StorageApiWriteUnshardedRecords.flush, when finalizing a batch of rows written via the Storage Write API, Beam converts the proto descriptor to a runtime Descriptor to build failed-row output. If descriptor validation fails (DescriptorValidationException), it logs this warning and skips producing successful/failed row output for that batch.

Solutions

1. Inspect the chained DescriptorValidationException to find the offending field and fix the TableSchema (rename invalid fields, use supported types).
2. Verify the schema is fetched fresh via the Storage API (use storage write's default-schema mode if possible) to avoid stale descriptors.
3. Avoid characters in BigQuery column names that are invalid in protobuf identifiers.
4. Restart the pipeline after correcting the schema so a valid descriptor is loaded.

Example fix

// before
// BigQuery column name: "my-column" (hyphen invalid in proto)
// after
// BigQuery column name: "my_column" (valid proto identifier)

// Validate schema field names are valid proto identifiers before writing:
boolean protoSafe(String name) { return name != null && name.matches("[A-Za-z_][A-Za-z0-9_]*"); }

try { descriptor = TableRowToStorageApiProto.wrapDescriptorProto(proto); } catch (DescriptorValidationException e) { LOG.warn("Failure getting proto descriptor...", e); /* route batch to failed-row output */ }

When it happens

Trigger: flush() calls TableRowToStorageApiProto.wrapDescriptorProto on the TableSchemaProto descriptor and validation fails — typically a malformed or inconsistent TableSchema (invalid field names/types) for a Storage API write with error/warning output configured.

Common situations: Schemas with field names that violate proto naming rules (invalid identifiers for dynamic protos); unsupported BigQuery types in dynamic destinations; schema updated mid-stream causing mismatched descriptor proto.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Failure parsing TableRow
• Problem converting field
• BigQuery storage source must be split before being read
• bigquery write error
• Both a query and an output type of 'BEAM_ROW' were…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/91f39c0cb5adf7f7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:895

              int numRecordsAppended = c.protoRows.getSerializedRowsCount();
              recordsAppended.inc(numRecordsAppended);
              BigQuerySinkMetrics.appendRowsRowStatusCounter(
                      BigQuerySinkMetrics.RowStatus.SUCCESSFUL,
                      BigQuerySinkMetrics.OK,
                      shortTableUrn)
                  .inc(numRecordsAppended);

              BigQuerySinkMetrics.reportSuccessfulRpcMetrics(
                  c, BigQuerySinkMetrics.RpcMethod.APPEND_ROWS, shortTableUrn);

              if (successfulRowsReceiver != null) {
                Descriptor descriptor = null;
                try {
                  descriptor =
                      TableRowToStorageApiProto.wrapDescriptorProto(
                          Preconditions.checkStateNotNull(appendClientInfo).getDescriptor());
                } catch (DescriptorValidationException e) {
                  LOG.warn(
                      "Failure getting proto descriptor. Successful output will not be produced.",
                      e);
                }
                if (descriptor != null) {
                  for (int i = 0; i < c.protoRows.getSerializedRowsCount(); ++i) {
                    ByteString rowBytes = c.protoRows.getSerializedRowsList().get(i);
                    try {
                      TableRow row =
                          TableRowToStorageApiProto.tableRowFromMessage(
                              Preconditions.checkStateNotNull(appendClientInfo)
                                  .getSchemaInformation(),
                              DynamicMessage.parseFrom(descriptor, rowBytes),
                              true,
                              successfulRowsPredicate);
                      org.joda.time.Instant timestamp = c.timestamps.get(i);
                      successfulRowsReceiver.outputWithTimestamp(row, timestamp);
                    } catch (Exception e) {
                      LOG.warn("Failure parsing TableRow", e);

View on GitHub (pinned to 12126d8942)