apache/beam · error · RuntimeException

Append to stream %s failed with invalid offset of %s

Error message

Append to stream %s failed with invalid offset of %s

What it means

RuntimeException thrown by StorageApiWriteUnshardedRecords.flush when an AppendRows append fails with a persistent error indicating an invalid offset on the write stream — specifically gRPC OUT_OF_RANGE or ALREADY_EXISTS statuses. This means the client's offset bookkeeping for the stream is out of sync with what BigQuery accepted, so the work item is failed immediately without retry.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:811

              if (failedContext.failureCount > allowedRetry) {
                String errorMessage =
                    String.format(
                        "More than %d attempts to call AppendRows failed. Last encountered error: %s",
                        allowedRetry, error != null ? error.toString() : "unknown");
                if (statusCode == Status.Code.PERMISSION_DENIED
                    || statusCode == Status.Code.NOT_FOUND) {
                  errorMessage +=
                      ". Please check if the destination table exists and if the service account has the "
                          + "bigquery.tables.updateData permission.";
                }
                throw new RuntimeException(errorMessage, error);
              }

              // The following errors are known to be persistent, so always fail the work item in
              // this case.
              if (statusCode.equals(Status.Code.OUT_OF_RANGE)
                  || statusCode.equals(Status.Code.ALREADY_EXISTS)) {
                throw new RuntimeException(
                    "Append to stream "
                        + this.streamName
                        + " failed with invalid "
                        + "offset of "
                        + failedContext.offset);
              }

              // Schema mismatched exceptions can happen if the table was recently updated. Since
              // vortex caches schemas
              // we might see the new schema before vortex does. In this case, we simply need to
              // retry.
              Exceptions.@Nullable StorageException storageException =
                  (error == null) ? null : Exceptions.toStorageException(error);
              boolean schemaMismatchError =
                  (storageException instanceof Exceptions.SchemaMismatchedException);
              if (!schemaMismatchError && error != null) {
                // There's no special error code for missing required fields, and that can also
                // happen due to vortex

View on GitHub (pinned to 12126d8942)

Solutions

Rerun the failing work item — the stream client recreates its offset state and typically recovers to a fresh position
Check logs for a prior successful append at the same offset (duplicate delivery)
If persistent, ensure only one writer owns the stream; avoid sharing write streams across pipelines
Update Beam — newer versions improved stream-append client offset recovery
As a last resort, recreate the table's write streams and restart the pipeline

Example fix

// before: shared default stream across two pipelines causing duplicate offsets
// after: use a dedicated stream per pipeline
.withMethod(BigQueryIO.Write.Method.STORAGE_API_WRITE)
// ensure only one job writes to the destination table at a time

Defensive patterns

Strategy: retry

Validate before calling

// Ensure single-writer per stream and no manual offset management
// before writing, recreate/refresh the stream client so offsets resync:
streamAppendClientHolder.close(); // forces fresh connection + offset resync on next use

Try / catch

try {
  flushRecords();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("invalid offset")) {
    // re-run the work item; recreate the StreamAppendClient to resync offsets
  } else { throw e; }
}

Prevention

Never share write streams or destination tables across concurrent pipelines
Upgrade Beam: newer versions handle duplicate/replayed offsets better
Avoid replaying partially-acknowledged appends manually
Monitor for prior successful appends at the failing offset to detect duplicates

When it happens

Trigger: An append to this.streamName returns Status.Code.OUT_OF_RANGE (offset beyond the stream's next expected offset) or ALREADY_EXISTS (offset already committed), typically after retries replayed a partially acknowledged append or stream state diverged.

Common situations: Retry of an append that actually succeeded on the server (duplicate offset); resharding/stream rotation concurrent with appends; multiple writers sharing one stream with overlapping offsets; Dataflow worker restart replaying a bundle whose first attempt partially committed.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c32bb9e479fa10f2. Report an issue: GitHub.