istio/istio · error

failure running port forward process: %v

Error message

failure running port forward process: %v

What it means

The port-forwarder object was built, but fw.Start() failed while establishing the SPDY port-forward stream to the Prometheus pod. The wrapped %v is usually a kubelet/upgrade error: pod not ready, node unreachable, or the POST /portforward request rejected.

Source

Thrown at istioctl/pkg/metrics/metrics.go:129

	pl, err := client.PodsForSelector(context.TODO(), ctx.IstioNamespace(), "app.kubernetes.io/name=prometheus")
	if err != nil {
		return fmt.Errorf("not able to locate Prometheus pod: %v", err)
	}

	if len(pl.Items) < 1 {
		return errors.New("no Prometheus pods found")
	}

	// only use the first pod in the list
	promPod := pl.Items[0]
	fw, err := client.NewPortForwarder(promPod.Name, ctx.IstioNamespace(), "", 0, 9090)
	if err != nil {
		return fmt.Errorf("could not build port forwarder for prometheus: %v", err)
	}

	if err = fw.Start(); err != nil {
		return fmt.Errorf("failure running port forward process: %v", err)
	}

	// Close the forwarder either when we exit or when this process is interrupted.
	defer fw.Close()
	go dashboard.ClosePortForwarderOnInterrupt(fw)

	log.Debugf("port-forward to prometheus pod ready")

	promAPI, err := prometheusAPI(fmt.Sprintf("http://%s", fw.Address()))
	if err != nil {
		return fmt.Errorf("failure running port forward process: %v", err)
	}

	printHeader(c.OutOrStdout())

	workloads := args
	for _, workload := range workloads {
		sm, err := metrics(promAPI, workload, metricsDuration)

View on GitHub (pinned to 8dc789c5cf)

Solutions

1. Wait for readiness: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n istio-system
2. Confirm the container really listens on 9090: kubectl get -n istio-system pod <p> -o jsonpath='{.spec.containers[0].ports}'
3. Retry the command once the pod is Running
4. If upgrades are stripped by a proxy, port-forward manually with kubectl and query Prometheus directly

Example fix

# before
istioctl experimental metrics productpage-v1   # forward start fails, pod not ready
# after
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n istio-system --timeout=120s
istioctl experimental metrics productpage-v1

kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus \
  -n istio-system --timeout=120s || exit 1
istioctl experimental metrics productpage-v1 || istioctl experimental metrics productpage-v1

for attempt in 1 2 3; do
  out=$(istioctl experimental metrics productpage-v1 2>&1) && break
  case "$out" in *'port forward'*) sleep 5;; *) echo "$out"; exit 1;; esac
done

When it happens

Trigger: fw.Start() errors: Prometheus pod not Running/Ready yet, kubelet refusing the port (container port mismatch with 9090), SPDY upgrade denied by proxy, or the pod terminating concurrently.

Common situations: Running metrics immediately after install before Prometheus is ready; Prometheus configured on a non-9090 port; API-server fronting that drops connection upgrades.

Related errors

• could not build port forwarder for prometheus: %v
• not able to locate Prometheus pod: %v
• no Prometheus pods found
• bad metric value type returned for query
• failed to create k8s client: %v

AI-assisted analysis of istio/istio@8dc789c5cf (2026-08-15). Data as JSON: /api/errors/25b6433f704e6a0a. Report an issue: GitHub.