apache/beam · error · RuntimeException

Unexpected null schema for

Error message

Unexpected null schema for ${entry.getKey()}

What it means

TableSchemaCache's refresh thread updates cached schemas from a map of fetched results. Every key returned by the refresh loop must already exist in cachedSchemas; if it does not, the internal invariant is broken and this RuntimeException is thrown naming the missing table key.

Solutions

1. Report this as a bug to the Apache Beam project with pipeline details — it indicates an internal race/invariant violation
2. Avoid stopping or mutating the cache concurrently with active refreshes; check lifecycle ordering
3. Upgrade Beam to pick up any fix for the cache race
4. Log the table key and reproduce with a minimal pipeline before filing the issue
5. As a workaround, restart the affected worker/pipeline stage

SchemaHolder holder = cachedSchemas.get(key); if (holder == null) { /* skip or re-register before update */ }

try {
  schemaCache.refreshSchema(tableRef, service, options);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unexpected null schema for")) {
    LOG.error("Cache invariant violation for table; restarting stage", e);
    // fail pipeline or recreate cache
  } else throw e;
}

When it happens

Trigger: The background refresh thread completes schema fetches and iterates the result map, but a table key in the results is absent from cachedSchemas — typically because the entry was removed or never registered while refreshes were in flight.

Common situations: Concurrent stop/eviction racing with the refresh thread, a table being removed from the cache while its refresh was queued, or a bug in cache bookkeeping during dynamic destination churn.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• 2xx codes should not be exceptions. Got status code
• BigQuery test was not shutdown previously. Table is
• Cannot call refreshSchema after the object has been stopped!
• Column has array of arrays which is prohibited in Spanner.
• CompletionException

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ceb59544e906060c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableSchemaCache.java:292

                    @Nullable SchemaHolder schemaHolder = cachedSchemas.get(entry.getKey());
                    return schemaHolder != null
                        && schemaHolder.getVersion() >= entry.getValue().getTargetVersion();
                  });
        } finally {
          tableUpdateMonitor.leave();
        }
      }

      // Query all the tables for their schema.
      final Map<String, @Nullable TableSchema> schemas = refreshAll(localTablesToRefresh);

      runUnderMonitor(
          () -> {
            // Update the cache schemas.
            for (Map.Entry<String, @Nullable TableSchema> entry : schemas.entrySet()) {
              SchemaHolder schemaHolder = cachedSchemas.get(entry.getKey());
              if (schemaHolder == null) {
                throw new RuntimeException("Unexpected null schema for " + entry.getKey());
              }

              if (entry.getValue() == null) {
                // There was an error fetching the schema. Reschedule it.
                Refresh oldRefresh =
                    Preconditions.checkStateNotNull(localTablesToRefresh.get(entry.getKey()));
                Refresh existingRefresh = this.tablesToRefresh.get(entry.getKey());
                if (existingRefresh == null
                    || oldRefresh.getTargetVersion() > existingRefresh.getTargetVersion()) {
                  this.tablesToRefresh.put(entry.getKey(), oldRefresh);
                }
              } else {
                SchemaHolder newSchema =
                    SchemaHolder.of(entry.getValue(), schemaHolder.getVersion() + 1);
                cachedSchemas.put(entry.getKey(), newSchema);
              }
            }
          });

View on GitHub (pinned to 12126d8942)