apache/beam · error · RuntimeError

--pid_file must be specified with --start

Error message

--pid_file must be specified with --start

What it means

The local_job_service_main.run() CLI entrypoint raises this RuntimeError when --background/--start daemonization is requested but no --pid_file was given. The pid file is required so the daemonized process can later be located and stopped via --stop.

Source

Thrown at sdks/python/apache_beam/runners/portability/local_job_service_main.py:97

    if not options.pid_file:
      raise RuntimeError('--pid_file must be specified with --stop')
    if os.path.exists(options.pid_file):
      with open(options.pid_file) as fin:
        pid = int(fin.read())
      print('Killing process at', pid)
      try:
        os.kill(pid, signal.SIGTERM)
      except Exception:
        print('Process', pid, 'already killed.')
      os.unlink(options.pid_file)
    else:
      print('Process id file', options.pid_file, 'already removed.')
    if not options.background:
      return

  if options.background:
    if not options.pid_file:
      raise RuntimeError('--pid_file must be specified with --start')
    if options.stop:
      argv.remove('--stop')
    argv.remove('--background')
    if not options.port_file:
      options.port_file = os.path.splitext(options.pid_file)[0] + '.port'
      argv.append('--port_file')
      argv.append(options.port_file)

    if not options.stdout_file:
      raise RuntimeError('--stdout_file must be specified with --background')
    os.makedirs(pathlib.PurePath(options.stdout_file).parent, exist_ok=True)
    stdout_dest = open(options.stdout_file, mode='w')

    if options.stderr_file:
      os.makedirs(pathlib.PurePath(options.stderr_file).parent, exist_ok=True)
      stderr_dest = open(options.stderr_file, mode='w')
    else:
      stderr_dest = subprocess.STDOUT

View on GitHub (pinned to 12126d8942)

Solutions

1. Add --pid_file /path/to/service.pid alongside --start/--background.
2. If you do not need daemonization, drop --start/--background and run in the foreground.
3. Reuse the same pid file path consistently between --start and --stop invocations.

Example fix

// before
python -m apache_beam.runners.portability.local_job_service_main --start --port 8099
// after
python -m apache_beam.runners.portability.local_job_service_main --start --port 8099 --pid_file /tmp/beam_job_service.pid

args = ['--start', '--port', '8099']
assert '--pid_file' in args or '--pid_file=...' in args, 'pass --pid_file with --start'

When it happens

Trigger: Running `python -m apache_beam.runners.portability.local_job_service_main --start` (or with --background) without passing --pid_file; options.pid_file is None when options.background is checked.

Common situations: Hand-writing a start command for a local Beam job server and forgetting the pid file; copying a stop command but omitting the flag pair used at start; scripts that set --stop without having recorded a pid file path.

Understand the failure class

Background: "--flag is required" and "must specify" CLI errors: how missing-required-flag validation works and how to fix it — this error's family across 20 libraries.

Related errors

• --stdout_file must be specified with --background
• endpoint not defined
• must supply dot_file argument
• Unrecognized value for stable unique names:
• Class %s missing a property named '%s'.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f78b69cba01621cd. Report an issue: GitHub.