{"record":{"id":"c32bb9e479fa10f2","repo":"apache/beam","slug":"append-to-stream-s-failed-with-invalid-offset-of-s","errorCode":null,"errorMessage":"Append to stream %s failed with invalid offset of %s","messagePattern":"Append to stream (.+?) failed with invalid offset of (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java","lineNumber":811,"sourceCode":"              if (failedContext.failureCount > allowedRetry) {\n                String errorMessage =\n                    String.format(\n                        \"More than %d attempts to call AppendRows failed. Last encountered error: %s\",\n                        allowedRetry, error != null ? error.toString() : \"unknown\");\n                if (statusCode == Status.Code.PERMISSION_DENIED\n                    || statusCode == Status.Code.NOT_FOUND) {\n                  errorMessage +=\n                      \". Please check if the destination table exists and if the service account has the \"\n                          + \"bigquery.tables.updateData permission.\";\n                }\n                throw new RuntimeException(errorMessage, error);\n              }\n\n              // The following errors are known to be persistent, so always fail the work item in\n              // this case.\n              if (statusCode.equals(Status.Code.OUT_OF_RANGE)\n                  || statusCode.equals(Status.Code.ALREADY_EXISTS)) {\n                throw new RuntimeException(\n                    \"Append to stream \"\n                        + this.streamName\n                        + \" failed with invalid \"\n                        + \"offset of \"\n                        + failedContext.offset);\n              }\n\n              // Schema mismatched exceptions can happen if the table was recently updated. Since\n              // vortex caches schemas\n              // we might see the new schema before vortex does. In this case, we simply need to\n              // retry.\n              Exceptions.@Nullable StorageException storageException =\n                  (error == null) ? null : Exceptions.toStorageException(error);\n              boolean schemaMismatchError =\n                  (storageException instanceof Exceptions.SchemaMismatchedException);\n              if (!schemaMismatchError && error != null) {\n                // There's no special error code for missing required fields, and that can also\n                // happen due to vortex","sourceCodeStart":793,"sourceCodeEnd":829,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java#L793-L829","documentation":"RuntimeException thrown by StorageApiWriteUnshardedRecords.flush when an AppendRows append fails with a persistent error indicating an invalid offset on the write stream — specifically gRPC OUT_OF_RANGE or ALREADY_EXISTS statuses. This means the client's offset bookkeeping for the stream is out of sync with what BigQuery accepted, so the work item is failed immediately without retry.","triggerScenarios":"An append to this.streamName returns Status.Code.OUT_OF_RANGE (offset beyond the stream's next expected offset) or ALREADY_EXISTS (offset already committed), typically after retries replayed a partially acknowledged append or stream state diverged.","commonSituations":"Retry of an append that actually succeeded on the server (duplicate offset); resharding/stream rotation concurrent with appends; multiple writers sharing one stream with overlapping offsets; Dataflow worker restart replaying a bundle whose first attempt partially committed.","solutions":["Rerun the failing work item — the stream client recreates its offset state and typically recovers to a fresh position","Check logs for a prior successful append at the same offset (duplicate delivery)","If persistent, ensure only one writer owns the stream; avoid sharing write streams across pipelines","Update Beam — newer versions improved stream-append client offset recovery","As a last resort, recreate the table's write streams and restart the pipeline"],"exampleFix":"// before: shared default stream across two pipelines causing duplicate offsets\n// after: use a dedicated stream per pipeline\n.withMethod(BigQueryIO.Write.Method.STORAGE_API_WRITE)\n// ensure only one job writes to the destination table at a time","handlingStrategy":"retry","validationCode":"// Ensure single-writer per stream and no manual offset management\n// before writing, recreate/refresh the stream client so offsets resync:\nstreamAppendClientHolder.close(); // forces fresh connection + offset resync on next use","typeGuard":null,"tryCatchPattern":"try {\n  flushRecords();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"invalid offset\")) {\n    // re-run the work item; recreate the StreamAppendClient to resync offsets\n  } else { throw e; }\n}","preventionTips":["Never share write streams or destination tables across concurrent pipelines","Upgrade Beam: newer versions handle duplicate/replayed offsets better","Avoid replaying partially-acknowledged appends manually","Monitor for prior successful appends at the failing offset to detect duplicates"],"tags":["bigquery","storage-write","offset","grpc","gcp"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}