apache/beam · error · RuntimeException

Cannot call refreshSchema after the object has been stopped!

Error message

Cannot call refreshSchema after the object has been stopped!

What it means

TableSchemaCache.refreshSchema schedules a schema refresh for a table, but the cache object has a stopped flag that permanently disables it after shutdown. Calling refreshSchema after stop() throws this RuntimeException to prevent use of a torn-down resource.

Solutions

Ensure no refreshSchema calls happen after stop(); gate the call with a stopped/isAlive check
Fix lifecycle ordering so all schema refreshes complete before stop() is called
Do not retry refreshSchema after a failure during teardown; treat stop as terminal
Create a new TableSchemaCache instance if you legitimately need to refresh again
Guard with try-catch and log if the call is best-effort during shutdown

Example fix

// before
schemaCache.refreshSchema(tableReference, writeStreamService, options);
// after
if (!schemaCache.isStopped()) {
  schemaCache.refreshSchema(tableReference, writeStreamService, options);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (cacheStopped) { throw new IllegalStateException("refreshSchema called after stop()"); } // check before calling

Try / catch

try {
  schemaCache.refreshSchema(tableRef, writeStreamService, options);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("after the object has been stopped")) {
    LOG.warn("Ignoring refresh after stop"); // best-effort during teardown
  } else throw e;
}

Prevention

Treat stop() as terminal; never retry refresh afterwards
Complete all schema refreshes before teardown
Track cache lifecycle explicitly in your writer code
Avoid caching TableSchemaCache beyond its owning DoFn lifetime

When it happens

Trigger: Calling refreshSchema on a TableSchemaCache after its stop() method has been invoked, e.g. during pipeline teardown, a late retry, or an element still being processed while the writer is closing.

Common situations: DoFn teardown racing with pending schema refresh calls, reusing a cached writer/schema-cache object across bundle or worker lifetimes, or error-handling code that retries refresh after the stream was aborted.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b86eb60e01167232. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableSchemaCache.java:220

    final String key = tableKey(tableReference);
    Optional<SchemaHolder> existing =
        runUnderMonitor(
            () ->
                Optional.ofNullable(
                    this.cachedSchemas.putIfAbsent(key, SchemaHolder.of(tableSchema, 0))));
    return existing.map(SchemaHolder::getTableSchema).orElse(null);
  }

  public void refreshSchema(
      TableReference tableReference,
      DatasetService datasetService,
      BigQueryServices.WriteStreamService writeStreamService,
      BigQueryOptions options) {
    int targetVersion =
        runUnderMonitor(
            () -> {
              if (stopped) {
                throw new RuntimeException(
                    "Cannot call refreshSchema after the object has been stopped!");
              }
              String key = tableKey(tableReference);
              @Nullable SchemaHolder schemaHolder = cachedSchemas.get(key);
              int nextVersion = schemaHolder != null ? schemaHolder.getVersion() + 1 : 0;
              @Nullable Refresh existing =
                  tablesToRefresh.putIfAbsent(
                      key, Refresh.of(datasetService, writeStreamService, options, nextVersion));
              // Wait at least until the next version.
              return (existing == null) ? nextVersion : existing.getTargetVersion();
            });
    waitForRefresh(tableReference, targetVersion);
  }

  private void waitForRefresh(TableReference tableReference, int version) {
    tableUpdateMonitor.enterWhenUninterruptibly(
        new Guard(tableUpdateMonitor) {
          @Override

View on GitHub (pinned to 12126d8942)