GoogleContainerTools/skaffold · error

cleaning up deployed job

Error message

cleaning up deployed job

What it means

Skaffold's verify Cleanup wraps an error returned while force-deleting a Kubernetes Job it had deployed for verification. Cleanup iterates over the tracker's deployed jobs and calls ForceJobDelete in each job's namespace; any kubectl delete failure is wrapped as "cleaning up deployed job". It means a verification Job object could not be removed from the cluster, often because the cluster is unreachable or the job/namespace is gone.

Source

Thrown at pkg/skaffold/verify/k8sjob/verify.go:331

// Cleanup deletes what was verified by calling Verify.
func (v *Verifier) Cleanup(ctx context.Context, out io.Writer, dryRun bool) error {
	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
		"VerifierType": "kubernetesCluster",
	})

	clientset, err := kubernetesclient.Client(v.kubectl.KubeContext)
	if err != nil {
		return fmt.Errorf("getting Kubernetes client: %w", err)
	}

	for _, job := range v.tracker.DeployedJobs() {
		// assumes the job namespace is set and not "" which is the case as createJob
		// & createJobFromManifestPath set the namespace in the created Job
		namespace := job.Namespace
		if err := k8sjobutil.ForceJobDelete(ctx, job.Name, clientset.BatchV1().Jobs(namespace), &v.kubectl); err != nil {
			// TODO(aaron-prindle): replace with actionable error
			return errors.Wrap(err, "cleaning up deployed job")
		}
	}
	return nil
}

// Dependencies lists all the files that describe what needs to be verified.
func (v *Verifier) Dependencies() ([]string, error) {
	return []string{}, nil
}

// TrackContainerAndJobFromBuild adds an artifact and its newly-associated container
// to the container tracker.
func (v *Verifier) TrackContainerAndJobFromBuild(artifact graph.Artifact, container tracker.Job, job *batchv1.Job) {
	v.tracker.Add(artifact, container, job.Namespace)
	v.tracker.AddJob(job)
	v.logger.RegisterJob(job.Name)
}

View on GitHub (pinned to a1189de023)

Solutions

1. Check cluster connectivity and current kubectl context (kubectl cluster-info) and retry cleanup
2. Verify the job's namespace still exists and the Job name is valid; delete manually with kubectl -n <ns> delete job <name> --force --grace-period=0
3. Confirm RBAC allows delete on batch/jobs in that namespace
4. If the job is already gone (NotFound), treat cleanup as done — this is the actionable-error TODO noted in the code

Example fix

// before: wrapped opaque error
cleanup failed: cleaning up deployed job: job.batch "verify-abc" not found
// after: ignore NotFound during cleanup
if apierrors.IsNotFound(err) { continue }
return errors.Wrap(err, "cleaning up deployed job")

if err := exec.Command("kubectl", "cluster-info").Run(); err != nil { return fmt.Errorf("cluster unreachable: %w", err) }
if _, err := clientset.BatchV1().Jobs(ns).Get(ctx, jobName, metav1.GetOptions{}); err != nil { /* pre-check job exists */ }

func isNotFoundErr(err error) bool { return apierrors.IsNotFound(err) }

func safeCleanup(v *Verifier) (err error) {
  defer func() { if r := recover(); r != nil { err = fmt.Errorf("cleanup panic: %v", r) } }()
  return v.Cleanup(context.Background())
}

When it happens

Trigger: Calling Cleanup on a k8sjob verifier when k8sjobutil.ForceJobDelete fails — e.g. the kubectl client cannot reach the cluster, the Job's namespace no longer exists, the Job already disappeared with a NotFound error treated as fatal, or RBAC denies deleting batchv1 Jobs.

Common situations: Cluster was torn down or context switched between the verify run and cleanup; namespace was deleted out from under the job; stale kubeconfig/credentials; network partition to the API server during teardown.

Related errors

• strings.Join(errMsgs, "\n") (joined helm cleanup error messa
• %q running k8s job timed out after : %v
• kubectl delete: %w
• starting logger: %w
• %q running job %q errored during run: reason=%q, message=%q

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/6bb16af177f6d29f. Report an issue: GitHub.