GoogleContainerTools/skaffold · error

creating verify job in cluster

Error message

creating verify job in cluster

What it means

The k8sjob verify runner failed while waiting for the verify Job to be created in the cluster. On wait errors (other than context cancellation), the test is marked failed and the error 'creating verify job in cluster' wraps the underlying kube API error.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:224

	v.TrackContainerAndJobFromBuild(graph.Artifact{
		ImageName: tc.Container.Name,
		Tag:       tc.Name,
	}, tracker.Job{Name: tc.Container.Name, ID: job.Name}, job)

	// This retrying is added as when attempting to kickoff multiple jobs simultaneously
	// This is because the k8s API server can be unresponsive when hit with a large
	// intitial set of Job CREATE requests
	if waitErr := wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) {
		_, err = clientset.BatchV1().Jobs(job.Namespace).Create(ctx, job, metav1.CreateOptions{})
		if err != nil {
			return false, nil
		}
		return true, nil
	}); waitErr != nil {
		if ctx.Err() != context.Canceled {
			eventV2.VerifyFailed(tc.Name, err)
			return errors.Wrap(err, "creating verify job in cluster")
		}
	}

	var timeoutDuration *time.Duration = nil
	if tc.Config.Timeout != nil {
		timeoutDuration = util.Ptr(time.Second * time.Duration(*tc.Config.Timeout))
	}

	var execErr error
	execCh := make(chan error)
	go func() {
		execCh <- v.watchJob(ctx, clientset, job, tc)
		close(execCh)
	}()

	select {
	case execErr = <-execCh:
	case <-v.timeout(timeoutDuration):

View on GitHub (pinned to a1189de023)

Solutions

1. Check RBAC: user/serviceaccount needs 'create jobs' permission (`kubectl auth can-i create jobs -n <ns>`)
2. Confirm the namespace exists and cluster is reachable (`kubectl get ns`)
3. Inspect the wrapped kube error for validation messages on the job spec
4. Increase timeouts or fix connectivity, then rerun `skaffold verify`

Example fix

// before: verify runs against namespace without job permissions
verify: [...]
// after: run in a namespace with RBAC
skaffold verify --namespace verify-ns

// Pre-check job creation RBAC
if err := exec.Command("kubectl", "auth", "can-i", "create", "jobs", "-n", ns).Run(); err != nil {
    return fmt.Errorf("serviceaccount cannot create jobs in namespace %s", ns)
}

if err := verifier.Verify(ctx, out, tc); err != nil {
    if strings.Contains(err.Error(), "creating verify job in cluster") {
        return fmt.Errorf("could not create verify job (check RBAC/namespace): %w", err)
    }
    return err
}

When it happens

Trigger: createAndRunJob's wait.PollImmediate (via waitExponentialBackoff) returns waitErr and ctx.Err() != context.Canceled — job creation API call fails or the job never appears within the backoff window.

Common situations: Insufficient RBAC to create batch/v1 Jobs in the namespace; invalid job spec rejected by the API server; namespace missing; cluster unreachable; context deadline hit before job creation.

Related errors

• attempting to watch verify job in cluster
• attempting to watch verify pods in cluster
• %q running k8s job timed out after : %v
• patching resource %s/%q: %w
• patching resource %q: %w

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/6a10dbb86d9f8c94. Report an issue: GitHub.