{"record":{"id":"ef2feb994c6f21a9","repo":"GoogleContainerTools/skaffold","slug":"q-running-job-q-errored-during-run-reason-q-m","errorCode":null,"errorMessage":"%q running job %q errored during run: reason=%q, message=%q","messagePattern":"%q running job %q errored during run: reason=%q, message=%q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/skaffold/verify/k8sjob/verify.go","lineNumber":292,"sourceCode":"\tfor event := range w.ResultChan() {\n\t\tpod, ok := event.Object.(*corev1.Pod)\n\t\tif ok {\n\t\t\tif pod.Status.Phase == corev1.PodSucceeded {\n\t\t\t\t// TODO(aaron-prindle) add support for jobs w/ multiple pods in the future\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif pod.Status.Phase == corev1.PodFailed {\n\t\t\t\tfailReason := pod.Status.Reason\n\t\t\t\tif failReason == \"\" {\n\t\t\t\t\tfailReason = \"<empty>\"\n\t\t\t\t}\n\n\t\t\t\tfailMessage := pod.Status.Message\n\t\t\t\tif failMessage == \"\" {\n\t\t\t\t\tfailMessage = \"<empty>\"\n\t\t\t\t}\n\n\t\t\t\tpodErr = fmt.Errorf(\n\t\t\t\t\t\"%q running job %q errored during run: reason=%q, message=%q\",\n\t\t\t\t\ttc.Name, job.Name, failReason, failMessage,\n\t\t\t\t)\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tif err := k8sjobutil.CheckIfPullImgErr(pod, job.Name); err != nil {\n\t\t\t\tv.logger.CancelJobLogger(job.Name)\n\t\t\t\treturn err\n\t\t\t}\n\t\t}\n\t}\n\n\tif podErr != nil {\n\t\teventV2.VerifyFailed(tc.Name, podErr)\n\t\treturn errors.Wrap(podErr, \"verify test failed\")\n\t}\n\teventV2.VerifySucceeded(tc.Name)","sourceCodeStart":274,"sourceCodeEnd":310,"githubUrl":"https://github.com/GoogleContainerTools/skaffold/blob/a1189de023efc32d4b8e11f395acc678aa555011/pkg/skaffold/verify/k8sjob/verify.go#L274-L310","documentation":"While watching a Kubernetes Job's pods, the verifier saw a pod reach a failed state and reports the test case name, job name, and the pod's failure reason/message. This is not a Skaffold bug — the containerized verify test itself failed inside the cluster, and Skaffold surfaces the pod's own status.","triggerScenarios":"watchJob observes a pod for the job with a failure condition (e.g. Container terminated with non-zero exit code, ImagePullBackOff, OOMKilled, CrashLoopBackOff); failReason comes from pod status and failMessage from pod.Status.Message (shown as <empty> if blank).","commonSituations":"The verify test container exits non-zero; wrong/untagged image causing pull failures; test exceeding memory limits (OOMKilled); bad job manifest (invalid command/args); cluster issues like insufficient resources.","solutions":["Read the embedded reason/message; get details with `kubectl describe pod` and `kubectl logs` on the failing pod","If the test itself failed, fix the test/container so it exits 0","For ImagePullBackOff, fix the image name/tag and ensure the cluster can pull it (registry auth)","For OOMKilled, raise memory limits in the job spec/container config","If using a custom JobManifestPath, validate the manifest's resources, command, and restartPolicy"],"exampleFix":"// before: test container crashes with exit 1\nreason=Error message=\"command terminated with non-zero exit code\"\n// after: fix the test so it exits 0, or bump resources:\nresources:\n  limits:\n    memory: 512Mi","handlingStrategy":"try-catch","validationCode":"// before running the job, ensure image exists & test exits 0 locally\ndocker manifest inspect <image:tag>\n# and run the test container locally with the same command","typeGuard":null,"tryCatchPattern":"err := verifier.Verify(ctx, out)\nvar podFail = regexp.MustCompile(`reason=\"?([^,\"]+)`)\nif m := podFail.FindStringSubmatch(err.Error()); m != nil {\n    // m[1] = e.g. Error / OOMKilled / ImagePullBackOff -> branch remediation\n}","preventionTips":["Test the verify container locally before running it in-cluster","Set sane resource limits to avoid OOMKilled","Use fully-qualified image tags reachable from the cluster","Inspect `kubectl describe pod` output for the underlying failure"],"tags":["kubernetes","job","pod-failure","verify"],"backgroundTag":"kubernetes-job-pod-failed","analyzedSha":"a1189de023efc32d4b8e11f395acc678aa555011","analyzedAt":"2026-09-05T12:09:27.064Z","contentChangedAt":"2026-09-05T12:09:27.064Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}