{"record":{"id":"b6203b11fb61bee8","repo":"apache/beam","slug":"append-to-stream-s-failed-with-status-code-s-the-stream-may","errorCode":null,"errorMessage":"Append to stream %s failed with Status Code %s. The stream may not exist.","messagePattern":"Append to stream (.+?) failed with Status Code (.+?)\\. The stream may not exist\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java","lineNumber":854,"sourceCode":"                  schemaMismatchError =\n                      description != null && description.contains(\"incompatible fields\");\n                }\n              }\n              if (schemaMismatchError) {\n                LOG.info(\n                    \"Vortex failed stream open due to incompatible fields. This is likely because the BigTable \"\n                        + \"schema was recently updated and Vortex hasn't noticed yet, so retrying. error {}\",\n                    Preconditions.checkStateNotNull(error).toString());\n              }\n\n              boolean hasPersistentErrors =\n                  failedContext.getError() instanceof Exceptions.StreamFinalizedException\n                      || statusCode.equals(Status.Code.INVALID_ARGUMENT)\n                      || (!this.useDefaultStream && statusCode.equals(Status.Code.NOT_FOUND))\n                      || statusCode.equals(Status.Code.FAILED_PRECONDITION);\n              hasPersistentErrors = hasPersistentErrors && !schemaMismatchError;\n              if (hasPersistentErrors) {\n                throw new RuntimeException(\n                    String.format(\n                        \"Append to stream %s failed with Status Code %s. The stream may not exist.\",\n                        this.streamName, statusCode),\n                    error);\n              }\n              // TODO: Only do this on explicit NOT_FOUND errors once BigQuery reliably produces\n              // them.\n              try {\n                tryCreateTable.call();\n              } catch (Exception e) {\n                throw new RuntimeException(e);\n              }\n\n              int numRowsRetried = failedContext.protoRows.getSerializedRowsCount();\n              BigQuerySinkMetrics.appendRowsRowStatusCounter(\n                      BigQuerySinkMetrics.RowStatus.RETRIED, errorCode, shortTableUrn)\n                  .inc(numRowsRetried);\n","sourceCodeStart":836,"sourceCodeEnd":872,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java#L836-L872","documentation":"RuntimeException thrown by StorageApiWriteUnshardedRecords.flush when the append fails with a persistent error that is not a schema mismatch — a StreamFinalizedException, INVALID_ARGUMENT, NOT_FOUND on a non-default stream, or FAILED_PRECONDITION. These indicate the write stream itself is unusable, so the work item fails without further retry.","triggerScenarios":"AppendRows returns one of the persistent status codes above and schemaMismatchError is false, meaning the destination write stream was finalized, does not exist, or is otherwise in an invalid state.","commonSituations":"Write stream finalized after its TTL or by a manual flush/finalize call while the pipeline still appends; stream deleted; NOT_FOUND on a non-default stream due to table recreation; INVALID_ARGUMENT from a stale or incompatible stream client; FAILED_PRECONDITION after table replacement.","solutions":["Rerun the work item — the code recreates the connection/stream (a non-default stream is recreated on NOT_FOUND)","Check whether the stream or table was finalized/deleted externally while the job ran","Ensure the table isn't being dropped/recreated during the pipeline run","Verify the Beam version — stream recreation handling has improved in later releases","Confirm the append payload matches the stream's schema to rule out INVALID_ARGUMENT causes"],"exampleFix":"// before: external job calls FlushRows/finalize on the stream mid-run\n// after: only finalize streams after the pipeline completes\n// (schedule stream finalization in a pipeline-completion callback) ","handlingStrategy":"retry","validationCode":"// Verify the write stream still accepts appends before flushing\nStreamStats stats = bigQueryWriteClient.getWriteStream(streamName).getStats();\nif (stats.getEndTimeMs() > 0) throw new IllegalStateException(\"Stream finalized: \" + streamName);","typeGuard":null,"tryCatchPattern":"try {\n  flushRecords();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"The stream may not exist\")) {\n    // allow the framework to recreate the stream on rerun; verify table wasn't replaced\n  } else { throw e; }\n}","preventionTips":["Do not finalize, delete, or recreate the table/streams while the pipeline is running","Use the default stream if external finalization is unavoidable","Upgrade Beam for improved non-default stream recreation on NOT_FOUND","Keep append payloads schema-compatible to avoid INVALID_ARGUMENT"],"tags":["bigquery","storage-write","stream","grpc","gcp"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}