{"record":{"id":"63432a4d600c1ae7","repo":"GoogleContainerTools/skaffold","slug":"q-running-k8s-job-timed-out-after-v","errorCode":null,"errorMessage":"%q running k8s job timed out after : %v","messagePattern":"%q running k8s job timed out after : (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/skaffold/verify/k8sjob/verify.go","lineNumber":243,"sourceCode":"\t\t}\n\t}\n\n\tvar timeoutDuration *time.Duration = nil\n\tif tc.Config.Timeout != nil {\n\t\ttimeoutDuration = util.Ptr(time.Second * time.Duration(*tc.Config.Timeout))\n\t}\n\n\tvar execErr error\n\texecCh := make(chan error)\n\tgo func() {\n\t\texecCh <- v.watchJob(ctx, clientset, job, tc)\n\t\tclose(execCh)\n\t}()\n\n\tselect {\n\tcase execErr = <-execCh:\n\tcase <-v.timeout(timeoutDuration):\n\t\texecErr = errors.New(fmt.Sprintf(\"%q running k8s job timed out after : %v\", tc.Name, *timeoutDuration))\n\t\tv.logger.CancelJobLogger(job.Name)\n\t\tif err := k8sjobutil.ForceJobDelete(ctx, job.Name, clientset.BatchV1().Jobs(job.Namespace), &v.kubectl); err != nil {\n\t\t\texecErr = errors.Wrap(execErr, err.Error())\n\t\t}\n\t\teventV2.VerifyFailed(tc.Name, execErr)\n\t}\n\n\treturn execErr\n}\n\nfunc (v *Verifier) watchJob(ctx context.Context, clientset k8sclient.Interface, job *batchv1.Job, tc latest.VerifyTestCase) error {\n\tw, err := clientset.BatchV1().Jobs(job.Namespace).Watch(ctx,\n\t\tmetav1.ListOptions{FieldSelector: fmt.Sprintf(\"metadata.name=%s\", job.Name)})\n\tif err != nil {\n\t\teventV2.VerifyFailed(tc.Name, err)\n\t\treturn errors.Wrap(err, \"attempting to watch verify job in cluster\")\n\t}\n\tdefer w.Stop()","sourceCodeStart":225,"sourceCodeEnd":261,"githubUrl":"https://github.com/GoogleContainerTools/skaffold/blob/a1189de023efc32d4b8e11f395acc678aa555011/pkg/skaffold/verify/k8sjob/verify.go#L225-L261","documentation":"In the Kubernetes Job verify runner (pkg/skaffold/verify/k8sjob/verify.go:243), createAndRunJob waits for the job's execution result with a timeout. If the timeout fires first, this error names the test case and duration, the job logger is cancelled and the Job is force-deleted (ForceJobDelete) so no orphaned job remains.","triggerScenarios":"A `skaffold verify` k8s job test whose pod does not complete before timeoutDuration; select's `case <-v.timeout(...)` branch fires; ForceJobDelete on BatchV1().Jobs(job.Namespace) failing wraps the delete error into execErr via errors.Wrap.","commonSituations":"Job pod stuck Pending due to insufficient cluster resources or unschedulable node; image pull errors/ImagePullBackOff in the cluster; test running far longer than the configured timeout; RBAC or quota issues preventing the pod from starting.","solutions":["Increase the verify test timeout in skaffold.yaml","Check pod status with `kubectl describe pod` / `kubectl get jobs` for Pending, ImagePullBackOff, or CrashLoopBackOff causes","Free up cluster resources or raise quotas so the job pod can be scheduled","Ensure the job's image is reachable from the cluster (push to a registry the cluster can pull from)"],"exampleFix":"// before: job stuck Pending, timeout 120s\n// timeout: 120\n// after: larger timeout + pullable image\n// timeout: 900\n// image: registry.example.com/my-test:latest","handlingStrategy":"validation","validationCode":"// Before running verify, ensure the cluster can schedule the job\n// kubectl get nodes -o jsonpath='{.items[*].status.allocatable.cpu}'\n// kubectl describe resourcequota  # confirm quota headroom","typeGuard":"func isK8sJobTimeout(err error) bool {\n    return strings.Contains(err.Error(), \"running k8s job timed out after\")\n}","tryCatchPattern":"if err := skaffold.Verify(ctx, opts); err != nil {\n    if isK8sJobTimeout(err) {\n        log.Println(\"k8s verify job timed out; check `kubectl describe pod` for Pending/ImagePullBackOff\")\n    } else {\n        return err\n    }\n}","preventionTips":["Set verify timeouts from measured job runtimes with headroom","Confirm cluster capacity, quotas, and RBAC before running verify jobs","Push verify images to a registry the cluster can pull from","Alert on pods stuck Pending to catch scheduling problems early"],"tags":["kubernetes","verify","timeout","job"],"backgroundTag":"operation-timed-out","analyzedSha":"a1189de023efc32d4b8e11f395acc678aa555011","analyzedAt":"2026-09-05T12:09:27.064Z","contentChangedAt":"2026-09-05T12:09:27.064Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}