apache/beam · error · LookupError

Job {} does not exist

Error message

Job {} does not exist

What it means

GetStateStream yields job state transitions since stream start. It raises LookupError when request.job_id is not a key in the service's in-memory _jobs registry, i.e. this service instance has no record of that job.

Source

Thrown at sdks/python/apache_beam/runners/portability/abstract_job_service.py:151

        pipeline=self._jobs[request.job_id].get_pipeline())

  def Cancel(
      self,
      request: beam_job_api_pb2.CancelJobRequest,
      context=None,
      timeout=None) -> beam_job_api_pb2.CancelJobResponse:
    self._jobs[request.job_id].cancel()
    return beam_job_api_pb2.CancelJobResponse(
        state=self._jobs[request.job_id].get_state()[0])

  def GetStateStream(self,
                     request,
                     context=None,
                     timeout=None) -> Iterator[beam_job_api_pb2.JobStateEvent]:
    """Yields state transitions since the stream started.
      """
    if request.job_id not in self._jobs:
      raise LookupError("Job {} does not exist".format(request.job_id))

    job = self._jobs[request.job_id]
    for state, timestamp in job.get_state_stream():
      yield make_state_event(state, timestamp)

  def GetMessageStream(
      self,
      request,
      context=None,
      timeout=None) -> Iterator[beam_job_api_pb2.JobMessagesResponse]:
    """Yields messages since the stream started.
      """
    if request.job_id not in self._jobs:
      raise LookupError("Job {} does not exist".format(request.job_id))

    job = self._jobs[request.job_id]
    for msg in job.get_message_stream():
      if isinstance(msg, tuple):

View on GitHub (pinned to 12126d8942)

Solutions

1. Use the job_id returned by the submit/run call against the same, still-running service instance.
2. If the service was restarted, re-submit the pipeline since job state is not persisted.
3. Inspect the server's self._jobs keys to confirm which jobs exist.

Example fix

// before
stub.GetStateStream(beam_job_api_pb2.GetJobStateRequest(job_id=stale_id))
// after
resp = stub.Run(beam_job_api_pb2.RunJobRequest(...))
stub.GetStateStream(beam_job_api_pb2.GetJobStateRequest(job_id=resp.job_id))

# only call with a job_id returned by this service instance
assert job_id in known_job_ids_from_submit_responses

try:
    for ev in stub.GetStateStream(req):
        handle(ev)
except LookupError:
    job_id = resubmit_pipeline()  # registry is in-memory; server likely restarted

When it happens

Trigger: Calling the BeamJobApi GetStateStream RPC with a job_id never registered on this service instance, or with a job_id from before the service process restarted.

Common situations: Client connecting to a restarted job server (in-memory registry lost); stale or typo'd job_id; submitting to one service instance and polling on another.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Job {} does not exist
• Expected apache_beam.utils.timestamp.Timestamp, or google.pr
• no matching row found for row_key: %s with row_filter=%s
• Could not find coder for URN " + urn
• Unknown type of encoding context

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5ff4fb4f45ad9fb6. Report an issue: GitHub.