apache/beam · error · RuntimeError

Dependency computation failed or was cancelled.

Error message

Dependency computation failed or was cancelled.

What it means

In the background thread run by _run_async_computation, if wait_for_inputs is set and _wait_for_dependencies reports that a dependency PCollection's async computation failed or was cancelled, a RuntimeError is raised so the current computation is marked failed rather than computing on missing inputs.

Solutions

1. Re-run the upstream computation first and confirm it succeeds before computing the dependent PCollection
2. Inspect logs for the original failure of the dependency (the root pipeline error is logged by the manager)
3. Cancel and restart the interactive environment (ib.options) if stale computation state persists
4. Reduce chain depth: compute dependencies with blocking mode to surface errors immediately

Example fix

// before
ib.collect(downstream_pcoll)  # fails because upstream async compute failed
// after
ib.collect(upstream_pcoll)   # ensure the dependency computes successfully first
ib.collect(downstream_pcoll)

from apache_beam.runners.interactive import interactive_environment as ie
env = ie.current_env()
if any(not env.is_pcollection_computed(p) for p in upstream_pcolls):
    compute_upstreams_first()

try:
    ib.collect(downstream_pcoll)
except RuntimeError as e:
    if 'Dependency computation failed' in str(e):
        ib.collect(upstream_pcoll)  # recompute dependency, then retry

When it happens

Trigger: Computing a PCollection asynchronously whose upstream PCollection was itself computed asynchronously and failed/cancelled; chained ib.collect calls where an earlier async compute did not reach a DONE state.

Common situations: Multi-step interactive pipelines in notebooks where a downstream collect depends on an upstream computation that hit a pipeline error, was cancelled, or exceeded its timeout.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Asynchronous computation was cancelled.
• Cannot record because a dependency failed to compute…
• Timeout waiting for asynchronous computation completion.
• Blocking computation failed. State
• Dependencies must be a list of strings, got

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c4243d1ac04db147. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/recording_manager.py:510

    if async_result:
      async_result.set_pipeline_result(pipeline_result)

    pipeline_result.wait_until_finish()
    return pipeline_result

  def _run_async_computation(
      self,
      pcolls_to_compute: set[beam.pvalue.PCollection],
      async_result: 'AsyncComputationResult',
      wait_for_inputs: bool,
      runner: runner.PipelineRunner = None,
      options: pipeline_options.PipelineOptions = None,
  ):
    """The function to be run in the thread pool for async computation."""
    try:
      if wait_for_inputs:
        if not self._wait_for_dependencies(pcolls_to_compute, async_result):
          raise RuntimeError('Dependency computation failed or was cancelled.')

      _LOGGER.info(
          'Starting asynchronous computation for %d PCollections.',
          len(pcolls_to_compute))

      pipeline_result = self._execute_pipeline_fragment(
          pcolls_to_compute, async_result, runner, options)

      return pipeline_result
    except Exception as e:
      _LOGGER.exception('Exception during asynchronous computation: %s', e)
      raise

  def _watch(self, pcolls: list[beam.pvalue.PCollection]) -> None:
    """Watch any pcollections not being watched.

    This allows for the underlying caching layer to identify the PCollection as
    something to be cached.

View on GitHub (pinned to 12126d8942)