apache/beam · error

error upgrading pip

Error message

error upgrading pip

What it means

Raised in expansionx.SetUpPythonEnvironment when `pip install --upgrade pip` fails inside the freshly created virtualenv for the Python expansion service. The venv itself was created, but bootstrap of the package tooling failed.

Solutions

1. Ensure outbound network/HTTPS access to pypi.org from the machine (configure HTTPS_PROXY if needed)
2. Create the venv with pip support (ensure ensurepip is installed for the interpreter)
3. Retry after cleaning the cache; run the same pip upgrade command manually to see the real error
4. Use a newer Python interpreter (3.7+) whose bundled pip supports current PyPI TLS

Example fix

// before
// pip install --upgrade pip fails behind proxy
// after
export HTTPS_PROXY=http://proxy.corp:8080
./venv/bin/python -m pip install --upgrade pip  # verify manually

if err := exec.Command("curl", "-sSf", "https://pypi.org/simple/", "-o", "/dev/null").Run(); err != nil {
    return fmt.Errorf("PyPI unreachable; configure proxy before starting expansion service")
}

python, err := expansionx.SetUpPythonEnvironment(ctx, py, pkg, extras)
if err != nil && strings.Contains(err.Error(), "error upgrading pip") {
    return fmt.Errorf("set HTTPS_PROXY/HTTPS_PROXY and ensure ensurepip is installed: %w", err)
}

When it happens

Trigger: startPythonExpansionService -> SetUpPythonEnvironment where the venv's pip upgrade step returns non-zero: no network access to PyPI, missing ensurepip in the venv, or SSL/proxy issues.

Common situations: Corporate proxies/firewalls blocking pypi.org; venv created without pip (python -m venv --without-pip or missing ensurepip); old Python whose bundled pip can't negotiate TLS.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

• error upgrading setuptools
• apache beam installation failed in virtualenv
• Artifact not found at
• error creating new virtual environment for python expansion…
• error installing beam package

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d2d16c4e41883f2a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go:467

	}

	// create python virtual environment
	sort.Strings(extraPackages)
	beamPackage := fmt.Sprintf("apache_beam[gcp,aws,azure,dataframe]==%s", core.SdkVersion)
	venvDir := filepath.Join(
		cacheDir, "venvs",
		fmt.Sprintf("py-%s-beam-%s-%s", py, core.SdkVersion, strings.Join(extraPackages, ";")),
	)
	venvPython := filepath.Join(venvDir, "bin", "python")

	if _, err := os.Stat(venvPython); err != nil {
		err := exec.Command(py, "-m", "venv", venvDir).Run()
		if err != nil {
			return "", errors.Wrap(err, "error creating new virtual environment for python expansion service")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", "--upgrade", "pip").Run()
		if err != nil {
			return "", errors.Wrap(err, "error upgrading pip")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", "--upgrade", "setuptools").Run()
		if err != nil {
			return "", errors.Wrap(err, "error upgrading setuptools")
		}
		err = exec.Command(venvPython, "-m", "pip", "install", beamPackage, "pyparsing==2.4.2").Run()
		if err != nil {
			return "", errors.Wrap(err, fmt.Sprintf("error installing beam package: %v", beamPackage))
		}
		if len(extraPackages) > 0 {
			cmd := []string{"-m", "pip", "install"}
			cmd = append(cmd, extraPackages...)
			err = exec.Command(venvPython, cmd...).Run()
			if err != nil {
				return "", errors.Wrap(err, "error installing dependencies")
			}
		}
		err = exec.Command(venvPython, "-c", "import apache_beam").Run()

View on GitHub (pinned to 12126d8942)