apache/beam · error · IllegalStateException

Unexpected null pipeline option for DynamicDestination…

Error message

Unexpected null pipeline option for DynamicDestination object. Need to call setSideInputAccessorFromProcessContext(context) before use it.

What it means

UpdateSchemaDestination.getTableWithDefaultProject resolves the destination table via DynamicDestinations, which needs PipelineOptions that are injected by calling setSideInputAccessorFromProcessContext(processContext). If getPipelineOptions() returns null, the object was used before that call, so it throws IllegalStateException instructing the developer to inject the context first.

Solutions

1. Call dynamicDestinations.setSideInputAccessorFromProcessContext(context) before any getTable/getDestination usage, typically at the start of processElement
2. Move destination resolution out of setup()/constructor into processElement after context binding
3. If implementing a custom path, mimic BigQueryIO's WriteTables/CreateTables flow which performs the binding first
4. Verify the DynamicDestinations instance isn't shared across DoFn instances in a way that loses the binding
5. Guard with a null check and fail fast with a clear message in your own wrapper

Example fix

// before (in setup)
TableDestination dest = updateSchemaDestination.getTableWithDefaultProject(destination);
// after (in processElement)
public void processElement(ProcessContext context) {
  updateSchemaDestination.setSideInputAccessorFromProcessContext(context);
  TableDestination dest = updateSchemaDestination.getTableWithDefaultProject(destination);
}

if (dynamicDestinations.getPipelineOptions() == null) {
  dynamicDestinations.setSideInputAccessorFromProcessContext(context);
}

boolean optionsBound(DynamicDestinations<?> dd) {
  return dd.getPipelineOptions() != null;
}

try {
  TableDestination dest = updateSchemaDestination.getTableWithDefaultProject(destination);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("setSideInputAccessorFromProcessContext")) {
    updateSchemaDestination.setSideInputAccessorFromProcessContext(context);
    // retry once
  } else throw e;
}

When it happens

Trigger: dynamicDestinations.getTable (via tableDestination) is invoked before setSideInputAccessorFromProcessContext(context) was called — e.g. in setup, startBundle, or a custom path that bypasses processElement initialization.

Common situations: Calling getTable/getDestination in setup() instead of processElement, subclassing DynamicDestinations and using table resolution too early, or reusing an UpdateSchemaDestination across bundles without re-binding the context.

Related errors

• BigQuery test was not shutdown previously. Table is
• Cannot call refreshSchema after the object has been stopped!
• Buffer counter not initialized for UUID:
• Cannot access FinishBundleContext outside of @FinishBundle…
• Cannot access StartBundleContext outside of @StartBundle…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/54fa4dbff87b4530. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/UpdateSchemaDestination.java:121

    this.zeroLoadJobIdPrefixView = zeroLoadJobIdPrefixView;
    this.bqServices = bqServices;
    this.maxRetryJobs = maxRetryJobs;
    this.kmsKey = kmsKey;
    this.schemaUpdateOptions = schemaUpdateOptions;
    this.createDisposition = createDisposition;
    this.writeDisposition = writeDisposition;
    this.dynamicDestinations = dynamicDestinations;
  }

  @StartBundle
  public void startBundle(StartBundleContext c) {
    pendingJobs.clear();
  }

  TableDestination getTableWithDefaultProject(DestinationT destination) {
    PipelineOptions pipelineOptions = dynamicDestinations.getPipelineOptions();
    if (pipelineOptions == null) {
      throw new IllegalStateException(
          "Unexpected null pipeline option for DynamicDestination object. "
              + "Need to call setSideInputAccessorFromProcessContext(context) before use it.");
    }
    BigQueryOptions options = pipelineOptions.as(BigQueryOptions.class);
    TableDestination tableDestination = dynamicDestinations.getTable(destination);
    TableReference tableReference = tableDestination.getTableReference();

    if (Strings.isNullOrEmpty(tableReference.getProjectId())) {
      tableReference.setProjectId(
          options.getBigQueryProject() == null
              ? options.getProject()
              : options.getBigQueryProject());
      tableDestination = tableDestination.withTableReference(tableReference);
    }

    return tableDestination;
  }

View on GitHub (pinned to 12126d8942)