GoogleContainerTools/skaffold · error

attempting to watch verify pods in cluster

Error message

attempting to watch verify pods in cluster

What it means

After watching the Job, watchJob also watches the Pods labeled job-name=<job> to capture pod logs/status. If that pod Watch API call fails, the test is marked failed with 'attempting to watch verify pods in cluster' wrapping the kube error.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:269

	return execErr
}

func (v *Verifier) watchJob(ctx context.Context, clientset k8sclient.Interface, job *batchv1.Job, tc latest.VerifyTestCase) error {
	w, err := clientset.BatchV1().Jobs(job.Namespace).Watch(ctx,
		metav1.ListOptions{FieldSelector: fmt.Sprintf("metadata.name=%s", job.Name)})
	if err != nil {
		eventV2.VerifyFailed(tc.Name, err)
		return errors.Wrap(err, "attempting to watch verify job in cluster")
	}
	defer w.Stop()

	w, err = clientset.CoreV1().Pods(job.Namespace).Watch(ctx,
		metav1.ListOptions{
			LabelSelector: labels.Set(map[string]string{"job-name": job.Name}).String(),
		})
	if err != nil {
		eventV2.VerifyFailed(tc.Name, err)
		return errors.Wrap(err, "attempting to watch verify pods in cluster")
	}
	defer w.Stop()

	var podErr error
	for event := range w.ResultChan() {
		pod, ok := event.Object.(*corev1.Pod)
		if ok {
			if pod.Status.Phase == corev1.PodSucceeded {
				// TODO(aaron-prindle) add support for jobs w/ multiple pods in the future
				break
			}
			if pod.Status.Phase == corev1.PodFailed {
				failReason := pod.Status.Reason
				if failReason == "" {
					failReason = "<empty>"
				}

				failMessage := pod.Status.Message

View on GitHub (pinned to a1189de023)

Solutions

1. Verify RBAC: `kubectl auth can-i watch pods -n <ns>` and add pods get/list/watch to the role
2. Confirm pods with label job-name=<job> exist in the namespace
3. Check API server connectivity/kubeconfig between job and pod watch steps
4. Rerun `skaffold verify` if the failure was transient

Example fix

// before
verbs: ["create"]
// after (include pod watch perms)
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

// Pre-check watch permission on pods
if err := exec.Command("kubectl", "auth", "can-i", "watch", "pods", "-n", ns).Run(); err != nil {
    return fmt.Errorf("serviceaccount cannot watch pods in namespace %s", ns)
}

if err := verifier.Verify(ctx, out, tc); err != nil {
    if strings.Contains(err.Error(), "attempting to watch verify pods in cluster") {
        return fmt.Errorf("pod watch failed (add pods/watch RBAC): %w", err)
    }
    return err
}

When it happens

Trigger: clientset.CoreV1().Pods(ns).Watch(...) with LabelSelector job-name=<job> returns an error — RBAC denies watching pods, namespace missing, or API server errors, immediately after the job watch succeeded.

Common situations: ServiceAccount lacking 'watch pods' permission; pod label selector namespace mismatch; cluster connectivity drops between the two watch calls; restricted environments blocking pod watches.

Related errors

• attempting to watch verify job in cluster
• STATUSCHECK_STANDALONE_PODS_FETCH_ERR
• could not fetch standalone pods: %w
• initializing deployment watcher: %s
• initializing pod watcher for %q: %w

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/1f45cb87b3837bcf. Report an issue: GitHub.