GoogleContainerTools/skaffold · error

%q running k8s job timed out after : %v

Error message

%q running k8s job timed out after : %v

What it means

In the Kubernetes Job verify runner (pkg/skaffold/verify/k8sjob/verify.go:243), createAndRunJob waits for the job's execution result with a timeout. If the timeout fires first, this error names the test case and duration, the job logger is cancelled and the Job is force-deleted (ForceJobDelete) so no orphaned job remains.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:243

		}
	}

	var timeoutDuration *time.Duration = nil
	if tc.Config.Timeout != nil {
		timeoutDuration = util.Ptr(time.Second * time.Duration(*tc.Config.Timeout))
	}

	var execErr error
	execCh := make(chan error)
	go func() {
		execCh <- v.watchJob(ctx, clientset, job, tc)
		close(execCh)
	}()

	select {
	case execErr = <-execCh:
	case <-v.timeout(timeoutDuration):
		execErr = errors.New(fmt.Sprintf("%q running k8s job timed out after : %v", tc.Name, *timeoutDuration))
		v.logger.CancelJobLogger(job.Name)
		if err := k8sjobutil.ForceJobDelete(ctx, job.Name, clientset.BatchV1().Jobs(job.Namespace), &v.kubectl); err != nil {
			execErr = errors.Wrap(execErr, err.Error())
		}
		eventV2.VerifyFailed(tc.Name, execErr)
	}

	return execErr
}

func (v *Verifier) watchJob(ctx context.Context, clientset k8sclient.Interface, job *batchv1.Job, tc latest.VerifyTestCase) error {
	w, err := clientset.BatchV1().Jobs(job.Namespace).Watch(ctx,
		metav1.ListOptions{FieldSelector: fmt.Sprintf("metadata.name=%s", job.Name)})
	if err != nil {
		eventV2.VerifyFailed(tc.Name, err)
		return errors.Wrap(err, "attempting to watch verify job in cluster")
	}
	defer w.Stop()

View on GitHub (pinned to a1189de023)

Solutions

1. Increase the verify test timeout in skaffold.yaml
2. Check pod status with `kubectl describe pod` / `kubectl get jobs` for Pending, ImagePullBackOff, or CrashLoopBackOff causes
3. Free up cluster resources or raise quotas so the job pod can be scheduled
4. Ensure the job's image is reachable from the cluster (push to a registry the cluster can pull from)

Example fix

// before: job stuck Pending, timeout 120s
// timeout: 120
// after: larger timeout + pullable image
// timeout: 900
// image: registry.example.com/my-test:latest

// Before running verify, ensure the cluster can schedule the job
// kubectl get nodes -o jsonpath='{.items[*].status.allocatable.cpu}'
// kubectl describe resourcequota  # confirm quota headroom

func isK8sJobTimeout(err error) bool {
    return strings.Contains(err.Error(), "running k8s job timed out after")
}

if err := skaffold.Verify(ctx, opts); err != nil {
    if isK8sJobTimeout(err) {
        log.Println("k8s verify job timed out; check `kubectl describe pod` for Pending/ImagePullBackOff")
    } else {
        return err
    }
}

When it happens

Trigger: A `skaffold verify` k8s job test whose pod does not complete before timeoutDuration; select's `case <-v.timeout(...)` branch fires; ForceJobDelete on BatchV1().Jobs(job.Namespace) failing wraps the delete error into execErr via errors.Wrap.

Common situations: Job pod stuck Pending due to insufficient cluster resources or unschedulable node; image pull errors/ImagePullBackOff in the cluster; test running far longer than the configured timeout; RBAC or quota issues preventing the pod from starting.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• %q running job %q errored during run: reason=%q, message=%q
• %q running container image %q timed out after : %v
• %d resources failed to complete their deletion before a new
• timeout waiting for event from pod of kubernetes job: %s
• starting logger: %w

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/63432a4d600c1ae7. Report an issue: GitHub.