GoogleContainerTools/skaffold · error

%q running job %q errored during run: reason=%q, message=%q

Error message

%q running job %q errored during run: reason=%q, message=%q

What it means

While watching a Kubernetes Job's pods, the verifier saw a pod reach a failed state and reports the test case name, job name, and the pod's failure reason/message. This is not a Skaffold bug — the containerized verify test itself failed inside the cluster, and Skaffold surfaces the pod's own status.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:292

	for event := range w.ResultChan() {
		pod, ok := event.Object.(*corev1.Pod)
		if ok {
			if pod.Status.Phase == corev1.PodSucceeded {
				// TODO(aaron-prindle) add support for jobs w/ multiple pods in the future
				break
			}
			if pod.Status.Phase == corev1.PodFailed {
				failReason := pod.Status.Reason
				if failReason == "" {
					failReason = "<empty>"
				}

				failMessage := pod.Status.Message
				if failMessage == "" {
					failMessage = "<empty>"
				}

				podErr = fmt.Errorf(
					"%q running job %q errored during run: reason=%q, message=%q",
					tc.Name, job.Name, failReason, failMessage,
				)
				break
			}

			if err := k8sjobutil.CheckIfPullImgErr(pod, job.Name); err != nil {
				v.logger.CancelJobLogger(job.Name)
				return err
			}
		}
	}

	if podErr != nil {
		eventV2.VerifyFailed(tc.Name, podErr)
		return errors.Wrap(podErr, "verify test failed")
	}
	eventV2.VerifySucceeded(tc.Name)

View on GitHub (pinned to a1189de023)

Solutions

1. Read the embedded reason/message; get details with `kubectl describe pod` and `kubectl logs` on the failing pod
2. If the test itself failed, fix the test/container so it exits 0
3. For ImagePullBackOff, fix the image name/tag and ensure the cluster can pull it (registry auth)
4. For OOMKilled, raise memory limits in the job spec/container config
5. If using a custom JobManifestPath, validate the manifest's resources, command, and restartPolicy

Example fix

// before: test container crashes with exit 1
reason=Error message="command terminated with non-zero exit code"
// after: fix the test so it exits 0, or bump resources:
resources:
  limits:
    memory: 512Mi

// before running the job, ensure image exists & test exits 0 locally
docker manifest inspect <image:tag>
# and run the test container locally with the same command

err := verifier.Verify(ctx, out)
var podFail = regexp.MustCompile(`reason="?([^,"]+)`)
if m := podFail.FindStringSubmatch(err.Error()); m != nil {
    // m[1] = e.g. Error / OOMKilled / ImagePullBackOff -> branch remediation
}

When it happens

Trigger: watchJob observes a pod for the job with a failure condition (e.g. Container terminated with non-zero exit code, ImagePullBackOff, OOMKilled, CrashLoopBackOff); failReason comes from pod status and failMessage from pod.Status.Message (shown as <empty> if blank).

Common situations: The verify test container exits non-zero; wrong/untagged image causing pull failures; test exceeding memory limits (OOMKilled); bad job manifest (invalid command/args); cluster issues like insufficient resources.

Related errors

• %q running k8s job timed out after : %v
• verify test failed
• error in %v job execution, job failed
• starting logger: %w
• creating verify job in cluster

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/ef2feb994c6f21a9. Report an issue: GitHub.