{"record":{"id":"25b6433f704e6a0a","repo":"istio/istio","slug":"failure-running-port-forward-process-v-25b643","errorCode":null,"errorMessage":"failure running port forward process: %v","messagePattern":"failure running port forward process: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"istioctl/pkg/metrics/metrics.go","lineNumber":129,"sourceCode":"\n\tpl, err := client.PodsForSelector(context.TODO(), ctx.IstioNamespace(), \"app.kubernetes.io/name=prometheus\")\n\tif err != nil {\n\t\treturn fmt.Errorf(\"not able to locate Prometheus pod: %v\", err)\n\t}\n\n\tif len(pl.Items) < 1 {\n\t\treturn errors.New(\"no Prometheus pods found\")\n\t}\n\n\t// only use the first pod in the list\n\tpromPod := pl.Items[0]\n\tfw, err := client.NewPortForwarder(promPod.Name, ctx.IstioNamespace(), \"\", 0, 9090)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"could not build port forwarder for prometheus: %v\", err)\n\t}\n\n\tif err = fw.Start(); err != nil {\n\t\treturn fmt.Errorf(\"failure running port forward process: %v\", err)\n\t}\n\n\t// Close the forwarder either when we exit or when this process is interrupted.\n\tdefer fw.Close()\n\tgo dashboard.ClosePortForwarderOnInterrupt(fw)\n\n\tlog.Debugf(\"port-forward to prometheus pod ready\")\n\n\tpromAPI, err := prometheusAPI(fmt.Sprintf(\"http://%s\", fw.Address()))\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failure running port forward process: %v\", err)\n\t}\n\n\tprintHeader(c.OutOrStdout())\n\n\tworkloads := args\n\tfor _, workload := range workloads {\n\t\tsm, err := metrics(promAPI, workload, metricsDuration)","sourceCodeStart":111,"sourceCodeEnd":147,"githubUrl":"https://github.com/istio/istio/blob/8dc789c5cf17517c64e3c36cb3288230f149dfae/istioctl/pkg/metrics/metrics.go#L111-L147","documentation":"The port-forwarder object was built, but fw.Start() failed while establishing the SPDY port-forward stream to the Prometheus pod. The wrapped %v is usually a kubelet/upgrade error: pod not ready, node unreachable, or the POST /portforward request rejected.","triggerScenarios":"fw.Start() errors: Prometheus pod not Running/Ready yet, kubelet refusing the port (container port mismatch with 9090), SPDY upgrade denied by proxy, or the pod terminating concurrently.","commonSituations":"Running metrics immediately after install before Prometheus is ready; Prometheus configured on a non-9090 port; API-server fronting that drops connection upgrades.","solutions":["Wait for readiness: kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n istio-system","Confirm the container really listens on 9090: kubectl get -n istio-system pod <p> -o jsonpath='{.spec.containers[0].ports}'","Retry the command once the pod is Running","If upgrades are stripped by a proxy, port-forward manually with kubectl and query Prometheus directly"],"exampleFix":"# before\nistioctl experimental metrics productpage-v1   # forward start fails, pod not ready\n# after\nkubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus -n istio-system --timeout=120s\nistioctl experimental metrics productpage-v1","handlingStrategy":"retry","validationCode":"kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=prometheus \\\n  -n istio-system --timeout=120s || exit 1\nistioctl experimental metrics productpage-v1 || istioctl experimental metrics productpage-v1","typeGuard":null,"tryCatchPattern":"for attempt in 1 2 3; do\n  out=$(istioctl experimental metrics productpage-v1 2>&1) && break\n  case \"$out\" in *'port forward'*) sleep 5;; *) echo \"$out\"; exit 1;; esac\ndone","preventionTips":["Wait for Prometheus readiness before scraping (kubectl wait)","Verify the container actually exposes 9090 so the forward target is valid"],"tags":["kubernetes","port-forward","prometheus","istioctl","metrics","timing"],"backgroundTag":null,"analyzedSha":"8dc789c5cf17517c64e3c36cb3288230f149dfae","analyzedAt":"2026-08-15T15:16:55.434Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}