What causes "error in %v job execution, event type: %v" in GoogleContainerTools/skaffold?

Check kubectl get jobs -n and cluster events (kubectl get events) to see who deleted the Job or why the watch errored.

error in %v job execution, event type: %v

Error message

error in %v job execution, event type: %v

What it means

watchJob streams Job events from the API server; if the watched Job is Deleted or the watch emits an Error event, the task records 'error in <job> job execution, event type: <type>' and stops. This indicates the job object disappeared or the watch errored before the job could succeed.

Source

Thrown at pkg/skaffold/actions/k8sjob/task.go:213

	})

	return g.Wait()
}

func (t Task) watchJob(ctx context.Context, jobManifest batchv1.Job, jobsManager typesbatchv1.JobInterface) error {
	watcher, err := jobsManager.Watch(ctx, v1.ListOptions{
		FieldSelector: fmt.Sprintf("metadata.name=%v", jobManifest.Name),
	})
	if err != nil {
		return err
	}

	defer watcher.Stop()
	var jobErr error
	// The ctx is used by the watcher, so if the ctx is canceled, the channel will close finishing the loop.
	for event := range watcher.ResultChan() {
		if event.Type == apiwatch.Deleted || event.Type == apiwatch.Error {
			jobErr = fmt.Errorf("error in %v job execution, event type: %v", jobManifest.Name, event.Type)
			break
		}

		jobState, ok := event.Object.(*batchv1.Job)
		if ok && jobState.Status.Failed > 0 {
			jobErr = fmt.Errorf("error in %v job execution, job failed", jobManifest.Name)
			break
		}

		if ok && jobState.Status.Succeeded > 0 {
			break
		}
	}

	// We need this condition to check when the ctx was cancelled due to a timeout. In that case, the previous
	// watcher.ResultChan stops without reporting an error.
	if ctx.Err() != nil && jobErr == nil {
		jobErr = ctx.Err()

View on GitHub (pinned to a1189de023)

Solutions

Check kubectl get jobs -n <ns> and cluster events (kubectl get events) to see who deleted the Job or why the watch errored.
Re-run the action once cluster conditions stabilize; transient watch errors often disappear on retry.
Ensure nothing concurrently deletes Jobs in the namespace (CI cleanup scripts, TTL controllers ttlSecondsAfterFinished, GC policies).

Example fix

// before (job spec)
spec:
  ttlSecondsAfterFinished: 0  # job deleted immediately -> watch sees Deleted
// after
spec:
  ttlSecondsAfterFinished: 3600

Defensive patterns

Strategy: retry

Validate before calling

// Ensure no concurrent Job deletions: check TTL and cleanup policies
kubectl get jobs -n <ns> -o jsonpath='{.items[*].spec.ttlSecondsAfterFinished}'

Try / catch

try {
  await skaffold.verify(...);
} catch (e) {
  if (/event type: (Deleted|Error)/.test(String(e))) {
    await new Promise(r => setTimeout(r, 5000)); // transient watch/delete storms
    await skaffold.verify(...);
    return;
  }
  throw e;
}

Prevention

Do not run kubectl delete job / cleanup scripts while skaffold actions run
Set ttlSecondsAfterFinished high enough to outlive the watch
Keep the job's namespace free of concurrent teardown (CI stage isolation)

When it happens

Trigger: During a k8sjob action run: someone/something deletes the Job (external kubectl delete, namespace teardown, TTL-based GC, garbage collector removing the Job's owner) or the watch connection yields an Error event.

Common situations: Manual kubectl delete job during a long test; cluster autoscaler draining the node and GC'ing resources; RBAC/event issues; job manifest path pointing into a namespace being torn down concurrently.

Related errors

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/5c319b435d76d000. Report an issue: GitHub.