apache/beam · error · RuntimeError

Shared Server Process crashed

Error message

Shared Server Process crashed:
{error_msg}

What it means

Raised in _create_server while _get_manager waits for the newly spawned shared-object server process to initialize. The server process wrote its traceback to an error file before dying, so Beam surfaces that message wrapped in this RuntimeError and removes the temp file.

Solutions

1. Read the embedded error_msg in the exception: fix the underlying exception raised during server initialization
2. Verify all imports/dependencies needed by the shared object are installed in the runtime environment
3. Ensure the initialization callable and its arguments are picklable and side-effect free
4. Test constructing the object in a fresh subprocess locally to reproduce the child's error

Example fix

// before
shared = multi_process_shared.MultiProcessShared(MyObj, name='x')  # MyObj ctor reads missing file
// after
path = ensure_file_exists()
shared = multi_process_shared.MultiProcessShared(lambda: MyObj(path), name='x')

import importlib, pickle
mod = importlib.import_module(init_callable.__module__)  # import must not fail
pickle.dumps(init_callable)  # must be picklable
init_callable()  # must construct locally without raising

try:
    handle = mps.acquire()
except RuntimeError as e:
    if str(e).startswith('Shared Server Process crashed'):
        logging.error('Server init failed: %s', e)
        # fix dependency/config, then retry
    else:
        raise

When it happens

Trigger: Starting a shared object via multi_process_shared when the server process's initialization (e.g. the entry function constructing the object, or the module import in the child) raises an exception before it begins serving.

Common situations: The callable that creates the shared object raises (bad config, missing resource); a module imported by the child fails; unpicklable/incorrect initialization args; missing dependencies in the worker environment (esp. on Dataflow/containers).

Related errors

• Shared Server Process died unexpectedly with exit code
• Entry was released.
• Unable to get the Filesystem
• A BigQuery table or a query must be specified
• A cluster_identifier should be Optional[Union[str…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4a8f32e231c7f543. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/utils/multi_process_shared.py:504

      atexit.register(cleanup_process)

      start_time = time.time()
      last_log = start_time
      while True:
        if os.path.exists(address_file):
          break

        if os.path.exists(error_file):
          with open(error_file, 'r') as f:
            error_msg = f.read()
          try:
            os.remove(error_file)
          except OSError:
            pass

          if p.is_alive(): p.terminate()
          raise RuntimeError(f"Shared Server Process crashed:\n{error_msg}")

        if not p.is_alive():
          exit_code = p.exitcode
          raise RuntimeError(
              "Shared Server Process died unexpectedly"
              f" with exit code {exit_code}")

        if time.time() - last_log > 300:
          logging.warning(
              "Still waiting for %s to initialize... %ss elapsed)",
              self._tag,
              int(time.time() - start_time))
          last_log = time.time()

        time.sleep(0.05)

      logging.info('External process successfully started for %s', self._tag)
    else:

View on GitHub (pinned to 12126d8942)