apache/beam · error · RuntimeError

Could not generate protos due to error

Error message

Could not generate protos due to error: %s

What it means

When protos must be generated, setup.py shells out to gen_protos.py with check=True; if that subprocess exits nonzero (CalledProcessError), setup.py re-raises as RuntimeError including the subprocess stderr. The real failure cause (protoc missing, syntax error in .proto, network fetch failure) is in err.stderr.

Solutions

1. Read err.stderr in the message for the root cause and fix it (usually a protoc or grpcio-tools issue)
2. Run `python gen_protos.py` manually to see full output and iterate
3. Pin/install compatible tools: pip install grpcio-tools mypy-protobuf types-protobuf
4. Ensure protoc is on PATH and matches the protobuf runtime version

Example fix

# before
$ pip install -e .
RuntimeError: Could not generate protos due to error: b'protoc not found...'
# after
$ pip install grpcio-tools
$ python gen_protos.py --no-force
$ pip install -e .

import shutil
if shutil.which('protoc') is None:
    raise RuntimeError('protoc is required to generate Beam protos')

try:
    generate_protos_first()
except RuntimeError as e:
    if 'Could not generate protos' in str(e):
        install_proto_toolchain()  # pip install grpcio-tools, protoc
        generate_protos_first()
    else:
        raise

When it happens

Trigger: Running a build/install that triggers generate_protos_first when gen_protos.py fails: protoc/grpcio-tools not installed or incompatible version, malformed proto definitions, or grpc plugin errors.

Common situations: Build environments lacking protoc; mismatched grpcio-tools versions after a Beam upgrade; proto generation flaky in restricted CI sandboxes without network access.

Related errors

• Could not generate external transform wrappers due to error
• protobuf files are not generated. Please generate pb2 files
• Skipping proto generation as they are already generated.
• Cannot convert from nanoseconds to microseconds because…
• cannot convert to micro seconds

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/871072a2bdc63faa. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/setup.py:259

      # make sure we already generated protos
      pb2_files = list(
          find_by_ext(
              os.path.join(cwd, 'apache_beam', 'portability', 'api'),
              '_pb2.py'))
      if not pb2_files:
        raise RuntimeError(
            'protobuf files are not generated. '
            'Please generate pb2 files')

      warnings.warn('Skipping proto generation as they are already generated.')
      return
    out = subprocess.run(
        [sys.executable, os.path.join(cwd, 'gen_protos.py'), '--no-force'],
        capture_output=True,
        check=True)
    print(out.stdout)
  except subprocess.CalledProcessError as err:
    raise RuntimeError('Could not generate protos due to error: %s', err.stderr)


def copy_tests_from_docs():
  python_root = os.path.abspath(os.path.dirname(__file__))
  docs_src = os.path.normpath(
      os.path.join(
          python_root, '../../website/www/site/content/en/documentation/sdks'))
  docs_dest = os.path.normpath(
      os.path.join(python_root, 'apache_beam/yaml/docs'))
  if os.path.exists(docs_src):
    shutil.rmtree(docs_dest, ignore_errors=True)
    os.mkdir(docs_dest)
    for path in glob.glob(os.path.join(docs_src, 'yaml*.md')):
      shutil.copy(path, docs_dest)
  else:
    warnings.warn(
        f'Could not locate yaml docs source directory {docs_src}. '
        f'Skipping copying tests from docs.')

View on GitHub (pinned to 12126d8942)