apache/beam · error · RuntimeError

The file cannot be found. It was specified in the…

Error message

The file %s cannot be found. It was specified in the --requirements_file command line option.

What it means

Stager.create_job_resources stages a user's pip requirements file for worker environments. If setup_options.requirements_file is set but the path is not an existing file at staging time, Beam raises RuntimeError naming the file and pointing back to the --requirements_file option.

Solutions

Correct the --requirements_file path (use an absolute path) and ensure the file exists before launching.
cd to the expected directory or fix the relative path used by the submitting process.
In CI/containers, mount or copy the requirements file into the submitter's filesystem before running the pipeline.
Pre-validate with os.path.isfile(requirements_file) in your launcher script to fail fast with a clearer message.

Example fix

// before
--requirements_file=requirements.txt   # not in CWD
// after
--requirements_file=$(pwd)/requirements.txt

Defensive patterns

Strategy: validation

Validate before calling

import os
req = setup_options.requirements_file
if req is not None and not os.path.isfile(os.path.abspath(req)):
    raise SystemExit(f'--requirements_file not found: {os.path.abspath(req)}')

Try / catch

try:
    Stager.create_and_stage_job_resources(options, tmpdir)
except RuntimeError as e:
    if 'cannot be found' in str(e) and 'requirements_file' in str(e):
        print('Check --requirements_file path; use an absolute path')
        sys.exit(2)
    raise

Prevention

Pass absolute paths for --requirements_file
Run the submitter from a known CWD or resolve paths at launch
In CI, assert the requirements file exists before the pipeline step

When it happens

Trigger: --requirements_file=/path/reqs.txt passed on the CLI (or requirements_file set in SetupOptions) while the file does not exist relative to the working directory of the submitting process — checked with os.path.isfile before staging.

Common situations: Relative path resolved from a different CWD (CI vs local); file deleted between authoring and launch; typo in filename; container/remote submission where the file wasn't mounted or copied.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dae75c0937ec5cf9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/portability/stager.py:236

    # We can skip boot dependencies: apache beam sdk, python packages from
    # requirements.txt, python packages from extra_packages and workflow tarball
    # if we know we are using a dependency pre-installed sdk container image.
    if not skip_prestaged_dependencies:
      requirements_cache_path = (
          os.path.join(tempfile.gettempdir(), 'beam-requirements-cache') if
          (setup_options.requirements_cache
           is None) else setup_options.requirements_cache)
      if (setup_options.requirements_cache != SKIP_REQUIREMENTS_CACHE and
          not os.path.exists(requirements_cache_path)):
        os.makedirs(requirements_cache_path, exist_ok=True)

      # Track packages to stage for this specific run.
      packages_to_stage = set()
      # Stage a requirements file if present.
      if setup_options.requirements_file is not None:
        if not os.path.isfile(setup_options.requirements_file):
          raise RuntimeError(
              'The file %s cannot be found. It was specified in the '
              '--requirements_file command line option.' %
              setup_options.requirements_file)
        extra_packages, thinned_requirements_file = (
            Stager._extract_local_packages(setup_options.requirements_file))
        if extra_packages:
          setup_options.extra_packages = (
              setup_options.extra_packages or []) + extra_packages
        resources.append(
            Stager._create_file_stage_to_artifact(
                thinned_requirements_file, REQUIREMENTS_FILE))
        # Populate cache with packages from the requirement file option and
        # stage the files in the cache.
        if not use_beam_default_container:
          _LOGGER.warning(
              'When using a custom container image, prefer installing'
              ' additional PyPI dependencies directly into the image,'
              ' instead of specifying them via runtime options, '

View on GitHub (pinned to 12126d8942)