apache/beam · error · DataflowRuntimeException

Failed to cancel job

Error message

Failed to cancel job %s, please go to the Developers Console to cancel it manually.

What it means

DataflowRuntimeException raised by cancel() when the Dataflow service's modify_job_state() call fails to move the job to JOB_STATE_CANCELLED (returns falsy). The error tells the user to cancel the job manually in the Developers Console.

Solutions

1. Open the job in the GCP Developers Console and cancel it manually as the message instructs
2. Retry cancel(); if the job already terminated the is_in_terminal_state() check will short-circuit with a warning
3. Verify the caller's IAM permissions include dataflow.jobs.updateInstanceState on the project

Example fix

# before
result.cancel()
# after
try:
    result.cancel()
except DataflowRuntimeException as e:
    print('Manual cancellation required:', e)

# pre-check permissions
# gcloud projects get-iam-policy PROJECT --format=json | grep dataflow

for attempt in range(3):
    try:
        result.cancel()
        break
    except DataflowRuntimeException:
        time.sleep(2 ** attempt)
else:
    print('Cancel via console/API manually')

When it happens

Trigger: Calling pipeline_result.cancel() while the Dataflow API accepts but does not apply the cancellation — e.g. the job already finished between the terminal-state check and the modify call, transient API errors, or insufficient permissions on the job.

Common situations: Race conditions where the job completes just as cancel() is called; IAM principals lacking dataflow.jobs.updateInstanceState permission; service hiccups.

Related errors

• cannot use option zone with workerRegion; prefer either…
• cannot use option zone with workerZone; prefer workerZone
• Could not upload to GCS path
• Dataflow pipeline failed. State
• experiment worker_region and option workerRegion are…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/0d6034af2bf4b885. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:865

  def cancel(self):
    if not self.has_job:
      raise IOError('Failed to get the Dataflow job id.')

    self._update_job()

    if self.is_in_terminal_state():
      _LOGGER.warning(
          'Cancel failed because job %s is already terminated in state %s.',
          self.job_id(),
          self.state)
    else:
      if not self._runner.dataflow_client.modify_job_state(
          self.job_id(), 'JOB_STATE_CANCELLED'):
        cancel_failed_message = (
            'Failed to cancel job %s, please go to the Developers Console to '
            'cancel it manually.') % self.job_id()
        _LOGGER.error(cancel_failed_message)
        raise DataflowRuntimeException(cancel_failed_message, self)

    return self.state

  def __str__(self):
    return '<%s %s %s>' % (self.__class__.__name__, self.job_id(), self.state)

  def __repr__(self):
    return '<%s %s at %s>' % (self.__class__.__name__, self._job, hex(id(self)))


class DataflowRuntimeException(Exception):
  """Indicates an error has occurred in running this pipeline."""
  def __init__(self, msg, result):
    super().__init__(msg)
    self.result = result

View on GitHub (pinned to 12126d8942)