apache/beam · error · RuntimeException

Append to stream %s failed with Status Code %s. The stream m

Error message

Append to stream %s failed with Status Code %s. The stream may not exist.

What it means

RuntimeException thrown by StorageApiWriteUnshardedRecords.flush when the append fails with a persistent error that is not a schema mismatch — a StreamFinalizedException, INVALID_ARGUMENT, NOT_FOUND on a non-default stream, or FAILED_PRECONDITION. These indicate the write stream itself is unusable, so the work item fails without further retry.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:854

                  schemaMismatchError =
                      description != null && description.contains("incompatible fields");
                }
              }
              if (schemaMismatchError) {
                LOG.info(
                    "Vortex failed stream open due to incompatible fields. This is likely because the BigTable "
                        + "schema was recently updated and Vortex hasn't noticed yet, so retrying. error {}",
                    Preconditions.checkStateNotNull(error).toString());
              }

              boolean hasPersistentErrors =
                  failedContext.getError() instanceof Exceptions.StreamFinalizedException
                      || statusCode.equals(Status.Code.INVALID_ARGUMENT)
                      || (!this.useDefaultStream && statusCode.equals(Status.Code.NOT_FOUND))
                      || statusCode.equals(Status.Code.FAILED_PRECONDITION);
              hasPersistentErrors = hasPersistentErrors && !schemaMismatchError;
              if (hasPersistentErrors) {
                throw new RuntimeException(
                    String.format(
                        "Append to stream %s failed with Status Code %s. The stream may not exist.",
                        this.streamName, statusCode),
                    error);
              }
              // TODO: Only do this on explicit NOT_FOUND errors once BigQuery reliably produces
              // them.
              try {
                tryCreateTable.call();
              } catch (Exception e) {
                throw new RuntimeException(e);
              }

              int numRowsRetried = failedContext.protoRows.getSerializedRowsCount();
              BigQuerySinkMetrics.appendRowsRowStatusCounter(
                      BigQuerySinkMetrics.RowStatus.RETRIED, errorCode, shortTableUrn)
                  .inc(numRowsRetried);

View on GitHub (pinned to 12126d8942)

Solutions

1. Rerun the work item — the code recreates the connection/stream (a non-default stream is recreated on NOT_FOUND)
2. Check whether the stream or table was finalized/deleted externally while the job ran
3. Ensure the table isn't being dropped/recreated during the pipeline run
4. Verify the Beam version — stream recreation handling has improved in later releases
5. Confirm the append payload matches the stream's schema to rule out INVALID_ARGUMENT causes

Example fix

// before: external job calls FlushRows/finalize on the stream mid-run
// after: only finalize streams after the pipeline completes
// (schedule stream finalization in a pipeline-completion callback) 

// Verify the write stream still accepts appends before flushing
StreamStats stats = bigQueryWriteClient.getWriteStream(streamName).getStats();
if (stats.getEndTimeMs() > 0) throw new IllegalStateException("Stream finalized: " + streamName);

try {
  flushRecords();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("The stream may not exist")) {
    // allow the framework to recreate the stream on rerun; verify table wasn't replaced
  } else { throw e; }
}

When it happens

Trigger: AppendRows returns one of the persistent status codes above and schemaMismatchError is false, meaning the destination write stream was finalized, does not exist, or is otherwise in an invalid state.

Common situations: Write stream finalized after its TTL or by a manual flush/finalize call while the pipeline still appends; stream deleted; NOT_FOUND on a non-default stream due to table recreation; INVALID_ARGUMENT from a stale or incompatible stream client; FAILED_PRECONDITION after table replacement.

Related errors

• More than %d attempts to call AppendRows failed. Last encoun
• Append to stream %s failed with invalid offset of %s
• More than %d attempts to call AppendRows failed. Last encoun
• Failed to patch table schema.
• Failed to flush elements on window expiration!

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b6203b11fb61bee8. Report an issue: GitHub.