apache/beam · warning

Append to stream by client # failed with error, operations…

Error message

Append to stream {} by client #{} failed with error, operations will be retried.
{}

What it means

StorageApiWriteUnshardedRecords.flush() logs this when an AppendRows RPC to the BigQuery Storage Write API fails and the error is deemed retryable — the failed operations will be replayed. The message includes the stream name, client generation number, and error details retrieved from the response contexts. Failure count per context is incremented before retry.

Solutions

1. Read retrieveErrorDetails(...) in the log to see the underlying gRPC status and error details.
2. For UNAVAILABLE/INTERNAL errors, rely on built-in retry; ensure the pipeline isn't being cancelled mid-retry.
3. Check for schema evolution conflicts; enable autoSchemaUpdate or align schemas with the destination table.
4. Reduce batch sizes if errors indicate payload limits; verify VPC/Service Networking allows storage.googleapis.com.

Example fix

// before: append fails repeatedly due to stale client
// handled internally; ensure fresh clients
// after: at pipeline level, ensure retries and schema freshness
.withAutoSchemaUpdate(true) // and keep default retry behavior

// rely on built-in retry; guard application code around the write
try {
  rows.apply("StorageApiWrite", storageApiWrite);
} catch (Exception e) {
  // inspect retrieveErrorDetails-style gRPC status in logs before deciding to fail
}

When it happens

Trigger: AppendRows returns a retryable error (e.g. UNAVAILABLE, INTERNAL) not covered by earlier branches that return RETRY_ALL_OPERATIONS for client-recreation cases; the client will retry the append on the same or recreated client.

Common situations: Transient BigQuery Storage Write outages; schema mismatch errors retried after client recreation; network interruptions between worker and Google APIs; too-large append batches hitting message size limits.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• Flush of stream to offset failed
• Append to stream failed with invalid offset of
• Append to stream failed with Status Code . The stream may…
• Caught exception whilw trying to close append client…
• expansion failed

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8eee501181ef29b8. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:766

                failedContext.protoRows = retryRows.build();
                failedContext.timestamps = retryTimestamps;
                int numRowsRetried = failedContext.protoRows.getSerializedRowsCount();
                BigQuerySinkMetrics.appendRowsRowStatusCounter(
                        BigQuerySinkMetrics.RowStatus.RETRIED, errorCode, shortTableUrn)
                    .inc(numRowsRetried);

                // Since we removed rows, we need to update the insert offsets for all remaining
                // rows.
                long newOffset = failedContext.offset;
                for (AppendRowsContext context : contexts) {
                  context.offset = newOffset;
                  newOffset += context.protoRows.getSerializedRowsCount();
                }
                this.currentOffset = newOffset;
                return RetryType.RETRY_ALL_OPERATIONS;
              }

              LOG.warn(
                  "Append to stream {} by client #{} failed with error, operations will be retried.\n{}",
                  streamName,
                  clientNumber,
                  retrieveErrorDetails(contexts));
              failedContext.failureCount += 1;

              boolean quotaError = false;
              Throwable error = failedContext.getError();
              Status.Code statusCode = Status.Code.OK;
              if (error != null) {
                statusCode = Status.fromThrowable(error).getCode();
                quotaError = statusCode.equals(Status.Code.RESOURCE_EXHAUSTED);
              }

              int allowedRetry;

              if (!quotaError) {
                // This forces us to close and reopen all gRPC connections to Storage API on error,

View on GitHub (pinned to 12126d8942)