apache/beam · error · RuntimeError

--stdout_file must be specified with --background

Error message

--stdout_file must be specified with --background

What it means

When the job service is launched with --background, run() requires --stdout_file so the daemon's output is redirected to a real file (the parent directory is created with os.makedirs). Without it there is nowhere to send the backgrounded process's stdout, so a RuntimeError is raised.

Source

Thrown at sdks/python/apache_beam/runners/portability/local_job_service_main.py:107

      os.unlink(options.pid_file)
    else:
      print('Process id file', options.pid_file, 'already removed.')
    if not options.background:
      return

  if options.background:
    if not options.pid_file:
      raise RuntimeError('--pid_file must be specified with --start')
    if options.stop:
      argv.remove('--stop')
    argv.remove('--background')
    if not options.port_file:
      options.port_file = os.path.splitext(options.pid_file)[0] + '.port'
      argv.append('--port_file')
      argv.append(options.port_file)

    if not options.stdout_file:
      raise RuntimeError('--stdout_file must be specified with --background')
    os.makedirs(pathlib.PurePath(options.stdout_file).parent, exist_ok=True)
    stdout_dest = open(options.stdout_file, mode='w')

    if options.stderr_file:
      os.makedirs(pathlib.PurePath(options.stderr_file).parent, exist_ok=True)
      stderr_dest = open(options.stderr_file, mode='w')
    else:
      stderr_dest = subprocess.STDOUT

    subprocess.Popen([
        sys.executable,
        '-m',
        'apache_beam.runners.portability.local_job_service_main'
    ] + argv,
                     stderr=stderr_dest,
                     stdout=stdout_dest)
    print('Waiting for server to start up...')
    while not os.path.exists(options.port_file):

View on GitHub (pinned to 12126d8942)

Solutions

1. Pass --stdout_file /path/to/service.log with --background.
2. Optionally also pass --stderr_file to capture stderr (it is optional but recommended).
3. Run without --background to keep output on the terminal.

Example fix

// before
python -m apache_beam.runners.portability.local_job_service_main --start --pid_file /tmp/svc.pid
// after
python -m apache_beam.runners.portability.local_job_service_main --start --pid_file /tmp/svc.pid --stdout_file /tmp/svc.log --stderr_file /tmp/svc.err

required = {'--pid_file', '--stdout_file'}
missing = [f for f in required if not any(a.startswith(f) for a in cli_args)]
assert not missing, f'missing flags for --background: {missing}'

When it happens

Trigger: Starting the local job service with --background (e.g. as part of --start) while omitting --stdout_file.

Common situations: Daemonizing the service for pytest/CI setups where only pid and port files were configured; forgetting that --background (implied by --start) also needs a stdout destination.

Understand the failure class

Background: "--flag is required" and "must specify" CLI errors: how missing-required-flag validation works and how to fix it — this error's family across 20 libraries.

Related errors

• --pid_file must be specified with --start
• endpoint not defined
• must supply dot_file argument
• Unrecognized value for stable unique names:
• Class %s missing a property named '%s'.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/82a19a4231f63204. Report an issue: GitHub.