apache/beam · error · RuntimeError

Request to failed with status

Error message

Request to %s failed with status %d: %s

What it means

SparkUberJarJobServer.request wraps all REST calls (get/post/delete) to the Spark cluster and raises RuntimeError whenever the HTTP response status differs from the expected one (default 200), including the URL, actual status, and response body for diagnosis.

Solutions

1. Inspect the embedded response body for the Spark error detail (submission failures usually explain themselves).
2. Verify spark_rest_url points at the REST endpoint (standalone: http://master:6066 or :8080 cluster UI API), not the UI or driver port.
3. Handle 404 on status polling as 'job finished and purged' rather than a fatal error, or pass expected_status accordingly.
4. Check network/proxy/firewall between the driver and the Spark cluster; retry transient 5xx.
5. Ensure the uber jar and staged files are reachable from the cluster to avoid submission-time 500s.

Example fix

// before
status = job_server.get('v1/submissions/status/' + submission_id)  # may 404 after purge
// after
try:
    status = job_server.get('v1/submissions/status/' + submission_id)
except RuntimeError as e:
    if 'status 404' in str(e):
        status = {'submissionState': 'FINISHED'}  # treat purged app as finished
    else:
        raise

import requests
probe = requests.get(f'{rest_url.rstrip(chr(47))}/v1/applications', timeout=5)
probe.raise_for_status()  # confirm REST endpoint is healthy before submitting

try:
    status = server.get(f'v1/submissions/status/{submission_id}')
except RuntimeError as e:
    if 'status 404' in str(e):
        status = {'submissionState': 'FINISHED'}
    elif 'status 5' in str(e):
        retry_with_backoff()
    else:
        raise

When it happens

Trigger: Any REST interaction with the Spark cluster returning a non-expected status: submitting the job (bad jar path or master), polling status of a failed/killed app (404 once the app is purged from history), DELETE during cancellation, wrong expected_status passed by the caller.

Common situations: Spark REST service unreachable or behind a proxy returning 404/502; application already finished and removed so status polling 404s; wrong port (REST port vs UI port); cluster returned 500 due to submission failure (bad master URL, missing files).

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Request to failed with status
• could not complete request
• Method Not Allowed
• Option spark_rest_url must be set.
• Property "version" not found in…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3107c1072ba42752. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/spark_uber_jar_job_server.py:126

      pipeline,
      options,
      artifact_port=0):
    super().__init__(
        executable_jar,
        job_id,
        job_name,
        pipeline,
        options,
        artifact_port=artifact_port)
    self._rest_url = rest_url
    # Message history is a superset of state history.
    self._message_history = self._state_history[:]

  def request(self, method, path, expected_status=200, **kwargs):
    url = '%s/%s' % (self._rest_url, path)
    response = method(url, **kwargs)
    if response.status_code != expected_status:
      raise RuntimeError(
          "Request to %s failed with status %d: %s" %
          (url, response.status_code, response.text))
    if response.text:
      return response.json()

  def get(self, path, **kwargs):
    return self.request(requests.get, path, **kwargs)

  def post(self, path, **kwargs):
    return self.request(requests.post, path, **kwargs)

  def delete(self, path, **kwargs):
    return self.request(requests.delete, path, **kwargs)

  def _get_server_spark_version(self):
    # Spark REST API doesn't seem to offer a dedicated endpoint for getting the
    # version, but it does include the version in all responses, even errors.
    return self.get('', expected_status=400)['serverSparkVersion']

View on GitHub (pinned to 12126d8942)