{"record":{"id":"8eee501181ef29b8","repo":"apache/beam","slug":"append-to-stream-by-client-failed-with-error-operations-will","errorCode":null,"errorMessage":"Append to stream {} by client #{} failed with error, operations will be retried.\n{}","messagePattern":"Append to stream (.+?) by client #(.+?) failed with error, operations will be retried\\.\n(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java","lineNumber":766,"sourceCode":"                failedContext.protoRows = retryRows.build();\n                failedContext.timestamps = retryTimestamps;\n                int numRowsRetried = failedContext.protoRows.getSerializedRowsCount();\n                BigQuerySinkMetrics.appendRowsRowStatusCounter(\n                        BigQuerySinkMetrics.RowStatus.RETRIED, errorCode, shortTableUrn)\n                    .inc(numRowsRetried);\n\n                // Since we removed rows, we need to update the insert offsets for all remaining\n                // rows.\n                long newOffset = failedContext.offset;\n                for (AppendRowsContext context : contexts) {\n                  context.offset = newOffset;\n                  newOffset += context.protoRows.getSerializedRowsCount();\n                }\n                this.currentOffset = newOffset;\n                return RetryType.RETRY_ALL_OPERATIONS;\n              }\n\n              LOG.warn(\n                  \"Append to stream {} by client #{} failed with error, operations will be retried.\\n{}\",\n                  streamName,\n                  clientNumber,\n                  retrieveErrorDetails(contexts));\n              failedContext.failureCount += 1;\n\n              boolean quotaError = false;\n              Throwable error = failedContext.getError();\n              Status.Code statusCode = Status.Code.OK;\n              if (error != null) {\n                statusCode = Status.fromThrowable(error).getCode();\n                quotaError = statusCode.equals(Status.Code.RESOURCE_EXHAUSTED);\n              }\n\n              int allowedRetry;\n\n              if (!quotaError) {\n                // This forces us to close and reopen all gRPC connections to Storage API on error,","sourceCodeStart":748,"sourceCodeEnd":784,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java#L748-L784","documentation":"StorageApiWriteUnshardedRecords.flush() logs this when an AppendRows RPC to the BigQuery Storage Write API fails and the error is deemed retryable — the failed operations will be replayed. The message includes the stream name, client generation number, and error details retrieved from the response contexts. Failure count per context is incremented before retry.","triggerScenarios":"AppendRows returns a retryable error (e.g. UNAVAILABLE, INTERNAL) not covered by earlier branches that return RETRY_ALL_OPERATIONS for client-recreation cases; the client will retry the append on the same or recreated client.","commonSituations":"Transient BigQuery Storage Write outages; schema mismatch errors retried after client recreation; network interruptions between worker and Google APIs; too-large append batches hitting message size limits.","solutions":["Read retrieveErrorDetails(...) in the log to see the underlying gRPC status and error details.","For UNAVAILABLE/INTERNAL errors, rely on built-in retry; ensure the pipeline isn't being cancelled mid-retry.","Check for schema evolution conflicts; enable autoSchemaUpdate or align schemas with the destination table.","Reduce batch sizes if errors indicate payload limits; verify VPC/Service Networking allows storage.googleapis.com."],"exampleFix":"// before: append fails repeatedly due to stale client\n// handled internally; ensure fresh clients\n// after: at pipeline level, ensure retries and schema freshness\n.withAutoSchemaUpdate(true) // and keep default retry behavior","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// rely on built-in retry; guard application code around the write\ntry {\n  rows.apply(\"StorageApiWrite\", storageApiWrite);\n} catch (Exception e) {\n  // inspect retrieveErrorDetails-style gRPC status in logs before deciding to fail\n}","preventionTips":["Keep append batches within gRPC message size limits.","Enable autoSchemaUpdate for evolving schemas.","Ensure egress to storage.googleapis.com is permitted.","Track failureCount in logs for persistent (non-transient) errors."],"tags":["bigquery","grpc","retry","append"],"backgroundTag":"api-request-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}