apache/beam · error · LookupError

Found no metrics container for job

Error message

Found no metrics container for job {}

What it means

FlinkJobServerBeamJob.get_metrics() queries the Flink REST API for the job's user-task accumulators and looks for one named '__metricscontainers', which Beam pipelines use to carry metrics back from Flink workers. If no accumulator with that name exists, it means the job's tasks never published a metrics container (e.g. the job finished without running any Beam tasks successfully, or metrics were not reported), so a LookupError is raised.

Solutions

1. Wait until the job is running and at least one task has executed before calling get_metrics().
2. Verify the Flink job is healthy via the Flink REST endpoint (v1/jobs/<id>) and that it did not fail before reporting accumulators.
3. Ensure the Beam Flink runner jar on the job server matches the SDK version so '__metricscontainers' accumulators are emitted.
4. Wrap get_metrics() in try/except LookupError and return empty/unknown metrics if the container is absent.

Example fix

# before
metrics = job.get_metrics()
# after
try:
  metrics = job.get_metrics()
except LookupError:
  metrics = None  # job has not reported any metrics container yet

job_info = job.get('v1/jobs/%s' % job._flink_job_id)
if job_info.get('state') not in ('RUNNING', 'FINISHED'):
    raise RuntimeError('job not running; metrics may be unavailable')

def has_metrics_container(job):
    accs = job.get('v1/jobs/%s/accumulators' % job._flink_job_id).get('user-task-accumulators') or []
    return any(a.get('name') == '__metricscontainers' for a in accs)

try:
    metrics = job.get_metrics()
except LookupError:
    metrics = None  # or retry after the job has run longer

When it happens

Trigger: Calling get_metrics() on a FlinkJobServerBeamJob whose Flink job returned no 'user-task-accumulators' entry named '__metricscontainers' — e.g. querying metrics for a job that failed at startup, a job whose workers died before reporting, or a job id that resolved to a job with no Beam task accumulators.

Common situations: Polling metrics for an already-finished/failed Flink job; a Flink cluster whose classpath lacks the Beam Flink runner jars so accumulators are never published; querying too early before any task has executed; stale job ids after a job restart.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Can not query metrics. Job id is unknown.
• Could not find element
• Could not translate the internal step name %r.
• Could not translate the internal step name %r since job…
• 'Expected GaugeData metric type but received

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a76ec710ed3f3ec4. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py:252

        for ix, exc in enumerate(response['all-exceptions']):
          yield beam_job_api_pb2.JobMessage(
              message_id='message%d' % ix,
              time=str(exc['timestamp']),
              importance=beam_job_api_pb2.JobMessage.MessageImportance.
              JOB_MESSAGE_ERROR,
              message_text=exc['exception'])
        yield state, timestamp
        break
      else:
        yield state, timestamp

  def get_metrics(self):
    accumulators = self.get('v1/jobs/%s/accumulators' %
                            self._flink_job_id)['user-task-accumulators']
    for accumulator in accumulators:
      if accumulator['name'] == '__metricscontainers':
        return accumulator['value']
    raise LookupError(
        "Found no metrics container for job {}".format(self._flink_job_id))

View on GitHub (pinned to 12126d8942)