error creating new virtual environment for python expansion…

Error message

error creating new virtual environment for python expansion service

What it means

Raised in expansionx.SetUpPythonEnvironment when running `python -m venv <venvDir>` fails while preparing a local Python expansion service. It means a virtualenv could not be created, so no expansion service environment exists. The wrapped error comes from exec.Command(...).Run().

Solutions

Install venv support for the interpreter (e.g. apt install python3-venv / python3-pip)
Delete any stale/corrupt venv directory in the expansionx cache path and retry
Verify `python -m venv /tmp/testenv` works manually with the same interpreter
Check disk space and write permissions for the directory holding the venv

Example fix

// before
// error creating new virtual environment...
// after (fix host env, not code)
sudo apt-get install -y python3-venv python3-pip
rm -rf ~/.cache/beam/expansion_service_venv

Defensive patterns

Strategy: validation

Validate before calling

if err := exec.Command(py, "-m", "venv", "/tmp/beam_venv_probe").Run(); err != nil {
    return fmt.Errorf("python %q cannot create venvs: %w (install python3-venv?)", py, err)
}
os.RemoveAll("/tmp/beam_venv_probe")

Try / catch

python, err := expansionx.SetUpPythonEnvironment(ctx, py, pkg, extras)
if err != nil {
    return fmt.Errorf("venv creation failed; check python3-venv, disk space, and permissions: %w", err)
}

Prevention

Install python3-venv/ensurepip for the interpreter you pass
Pre-create a writable cache directory and keep free disk space
Delete stale venv directories from failed runs

When it happens

Trigger: startPythonExpansionService -> SetUpPythonEnvironment when the chosen python interpreter lacks the venv module, the venv target path is unwritable/exists corrupt, or the python binary itself is broken.

Common situations: System python installed without python3-venv (common on Debian/Ubuntu); pointing at a python without ensurepip; disk full or permission denied in the cache directory; corrupted venv left behind by a previous failed run.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6a25b3f7ec5b20ee. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go:463

	}
	extraPackages := []string{}
	if len(extraPackage) > 0 {
		extraPackages = strings.Split(extraPackage, " ")
	}

	// create python virtual environment
	sort.Strings(extraPackages)
	beamPackage := fmt.Sprintf("apache_beam[gcp,aws,azure,dataframe]==%s", core.SdkVersion)
	venvDir := filepath.Join(
		cacheDir, "venvs",
		fmt.Sprintf("py-%s-beam-%s-%s", py, core.SdkVersion, strings.Join(extraPackages, ";")),
	)
	venvPython := filepath.Join(venvDir, "bin", "python")

	if _, err := os.Stat(venvPython); err != nil {
		err := exec.Command(py, "-m", "venv", venvDir).Run()
		if err != nil {
			return "", errors.Wrap(err, "error creating new virtual environment for python expansion service")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", "--upgrade", "pip").Run()
		if err != nil {
			return "", errors.Wrap(err, "error upgrading pip")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", "--upgrade", "setuptools").Run()
		if err != nil {
			return "", errors.Wrap(err, "error upgrading setuptools")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", beamPackage, "pyparsing==2.4.2").Run()
		if err != nil {
			return "", errors.Wrap(err, fmt.Sprintf("error installing beam package: %v", beamPackage))
		}
		if len(extraPackages) > 0 {
			cmd := []string{"-m", "pip", "install"}
			cmd = append(cmd, extraPackages...)
			err = exec.Command(venvPython, cmd...).Run()
			if err != nil {

View on GitHub (pinned to 12126d8942)