GoogleContainerTools/skaffold · error

verify test failed

Error message

verify test failed

What it means

The pod watch loop observed a failure for the verify job's pod (podErr set from pod status/termination), so the verify test case is marked failed with 'verify test failed' wrapping the pod error. This means the job's container itself reported failure, not a cluster/API problem.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:308

				}

				podErr = fmt.Errorf(
					"%q running job %q errored during run: reason=%q, message=%q",
					tc.Name, job.Name, failReason, failMessage,
				)
				break
			}

			if err := k8sjobutil.CheckIfPullImgErr(pod, job.Name); err != nil {
				v.logger.CancelJobLogger(job.Name)
				return err
			}
		}
	}

	if podErr != nil {
		eventV2.VerifyFailed(tc.Name, podErr)
		return errors.Wrap(podErr, "verify test failed")
	}
	eventV2.VerifySucceeded(tc.Name)
	return nil
}

// Cleanup deletes what was verified by calling Verify.
func (v *Verifier) Cleanup(ctx context.Context, out io.Writer, dryRun bool) error {
	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
		"VerifierType": "kubernetesCluster",
	})

	clientset, err := kubernetesclient.Client(v.kubectl.KubeContext)
	if err != nil {
		return fmt.Errorf("getting Kubernetes client: %w", err)
	}

	for _, job := range v.tracker.DeployedJobs() {
		// assumes the job namespace is set and not "" which is the case as createJob

View on GitHub (pinned to a1189de023)

Solutions

1. Get pod logs: `kubectl logs job/<job-name> -n <ns>` to see the actual test failure
2. Check pod events/status (`kubectl describe pod`) for OOMKilled or ImagePullBackOff
3. Fix the failing test or its in-cluster dependencies (env vars, service endpoints)
4. Ensure the verify image is reachable by the cluster (push to a registry the cluster can pull from)

Example fix

// before: verify image only local, cluster can't pull
container:
  image: local-only-image
// after: push and reference registry image
container:
  image: registry.example.com/verify-image:tag

// Ensure verify image is pullable from the cluster
if err := exec.Command("kubectl", "run", "pull-check", "--image", verifyImage, "--restart=Never", "--", "true").Run(); err != nil {
    return fmt.Errorf("cluster cannot pull verify image %s", verifyImage)
}

if err := verifier.Verify(ctx, out, tc); err != nil {
    if strings.Contains(err.Error(), "verify test failed") {
        logs, _ := exec.Command("kubectl", "logs", "job/"+jobName, "-n", ns).CombinedOutput()
        return fmt.Errorf("verify failed, pod logs: %s: %w", logs, err)
    }
    return err
}

When it happens

Trigger: During the w.ResultChan() loop, a pod event indicates the verify container failed (non-zero exit code, terminated with failure reason), setting podErr; after the watch ends, watchJob wraps and returns it.

Common situations: Test command in the verify pod exits non-zero; pod OOMKilled; image pull backoff for the verify image; test fails due to missing in-cluster service dependencies.

Related errors

• %q running job %q errored during run: reason=%q, message=%q
• %q running k8s job timed out after : %v
• error in %v job execution, job failed
• starting logger: %w
• verify test failed

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/13ccde008f390a6b. Report an issue: GitHub.