apache/beam · error · LookupError

Job {} does not exist

Error message

Job {} does not exist

What it means

FlinkUberJarJobServer.GetJobMetrics proxies metrics from the Flink REST API for jobs it launched. It raises LookupError when request.job_id is not in the server's in-memory _jobs dict — this job server process has no record of that job.

Source

Thrown at sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py:102

  def flink_version(self):
    full_version = requests.get(
        '%s/v1/config' % self._master_url, timeout=60).json()['flink-version']
    # Only return up to minor version.
    return '.'.join(full_version.split('.')[:2])

  def create_beam_job(self, job_id, job_name, pipeline, options):
    return FlinkBeamJob(
        self._master_url,
        self.executable_jar(),
        job_id,
        job_name,
        pipeline,
        options,
        artifact_port=self._artifact_port)

  def GetJobMetrics(self, request, context=None):
    if request.job_id not in self._jobs:
      raise LookupError("Job {} does not exist".format(request.job_id))
    metrics_text = self._jobs[request.job_id].get_metrics()
    response = beam_job_api_pb2.GetJobMetricsResponse()
    json_format.Parse(metrics_text, response)
    return response


class FlinkBeamJob(abstract_job_service.UberJarBeamJob):
  """Runs a single Beam job on Flink by staging all contents into a Jar
  and uploading it via the Flink Rest API."""
  def __init__(
      self,
      master_url,
      executable_jar,
      job_id,
      job_name,
      pipeline,
      options,
      artifact_port=0):

View on GitHub (pinned to 12126d8942)

Solutions

1. Request metrics with a job_id returned by this job server's create_beam_job/run call.
2. Re-launch the pipeline via the same job server if its process was restarted (in-memory registry lost).
3. Query the Flink REST API directly (jobs/<job_id>) for jobs not launched through this server.

Example fix

// before
stub.GetJobMetrics(beam_job_api_pb2.GetJobMetricsRequest(job_id=foreign_id))
// after
result = launcher.run()  # same FlinkUberJarJobServer instance
stub.GetJobMetrics(beam_job_api_pb2.GetJobMetricsRequest(job_id=result.job_id))

# request metrics only for jobs launched by this server instance
assert job_id in launched_job_ids_on_this_server

try:
    metrics = stub.GetJobMetrics(req)
except LookupError:
    metrics = fetch_metrics_from_flink_rest(job_id)  # fallback

When it happens

Trigger: Calling the BeamJobApi GetJobMetrics RPC with a job_id this job server did not create in its current process lifetime (server restarted, or id from a job run elsewhere).

Common situations: Fetching metrics after the job-server process exited and was relaunched; querying metrics for a job submitted directly to Flink rather than through this Beam job server; copying a job_id from another cluster's logs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Job {} does not exist
• Expected apache_beam.utils.timestamp.Timestamp, or google.pr
• Unable to parse jar URL "%s". If using a full URL, make sure
• Unable to parse jar URL "%s". If using a full URL, make sure
• Request to %s failed with status %d: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/413af00561289951. Report an issue: GitHub.