apache/beam · error · ValueError

Can not query metrics. Job id is unknown.

Error message

Can not query metrics. Job id is unknown.

What it means

DataflowMetrics queries the Dataflow API by job id; before calling get_job_metrics it resolves the job id (argument or job_result.job_id()). If none is available it raises ValueError because metrics cannot be fetched without identifying the job.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py:224

      dist_min = int(metric.distribution['min'])
      dist_max = int(metric.distribution['max'])
      dist_sum = int(metric.distribution['sum'])
      return DistributionResult(
          DistributionData(dist_sum, dist_count, dist_min, dist_max))
      #TODO(https://github.com/apache/beam/issues/31788) support StringSet after
      #  re-generate apiclient
    else:
      return None

  def _get_metrics_from_dataflow(self, job_id=None):
    """Return cached metrics or query the dataflow service."""
    if not job_id:
      try:
        job_id = self.job_result.job_id()
      except AttributeError:
        job_id = None
    if not job_id:
      raise ValueError('Can not query metrics. Job id is unknown.')

    if self._cached_metrics:
      return self._cached_metrics

    job_metrics = self._dataflow_client.get_job_metrics(job_id)
    # If we cannot determine that the job has terminated,
    # then metrics will not change and we can cache them.
    if self.job_result and self.job_result.is_in_terminal_state():
      self._cached_metrics = job_metrics
    return job_metrics

  def all_metrics(self, job_id=None):
    """Return all user and system metrics from the dataflow service."""
    metric_results = []
    response = self._get_metrics_from_dataflow(job_id=job_id)
    self._populate_metrics(response, metric_results, user_metrics=True)
    self._populate_metrics(response, metric_results, user_metrics=False)
    return metric_results

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the pipeline run completed/submitted successfully and job_result.job_id() returns a real id before querying metrics.
2. Check the job launch result for errors; retry submission if the job never started.
3. Only construct/use DataflowMetrics after the job id is known (e.g. after wait_until_finish()).

Example fix

# before
result = pipeline.run()
print(DataflowMetrics(result, client).all_metrics())
# after
result = pipeline.run()
result.wait_until_finish()
if result.job_id():
    print(result.metrics().all_metrics())

job_id = getattr(job_result, 'job_id', lambda: None)()
if not job_id:
    raise RuntimeError('No Dataflow job id yet; run/wait for submission first')

def has_job_id(job_result) -> bool:
    try:
        return bool(job_result.job_id())
    except AttributeError:
        return False

try:
    m = metrics.all_metrics()
except ValueError as e:
    if 'Job id is unknown' in str(e):
        m = None  # job never submitted
    else:
        raise

When it happens

Trigger: Calling all_metrics() or query() on a DataflowMetrics whose job_result has no job_id yet — e.g. metrics object built before job submission completed, or a failed/dry-run result that never received an id.

Common situations: Accessing pipeline.result.metrics() when the job launch failed or when using a mock/custom job result without job_id(), or querying before the Dataflow job was actually submitted.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Could not find element
• Too many matches
• Could not translate the internal step name %r since job grap
• Could not translate the internal step name %r.
• NotImplementedError

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6963789c4e790f6d. Report an issue: GitHub.