What causes "error in %v job execution, job failed" in GoogleContainerTools/skaffold?

Inspect the failed pod logs: kubectl logs job/ -n (or kubectl describe job) to find the container exit reason, then fix the action's command/args.

GoogleContainerTools/skaffold · error

error in %v job execution, job failed

Error message

error in %v job execution, job failed

What it means

watchJob inspects each Job event object's status; if jobState.Status.Failed > 0, the Job's pod(s) failed and the task records 'error in <job> job execution, job failed'. This is the normal signal that the action's container exited non-zero or its pod was unschedulable/evicted.

Source

Thrown at pkg/skaffold/actions/k8sjob/task.go:219

	watcher, err := jobsManager.Watch(ctx, v1.ListOptions{
		FieldSelector: fmt.Sprintf("metadata.name=%v", jobManifest.Name),
	})
	if err != nil {
		return err
	}

	defer watcher.Stop()
	var jobErr error
	// The ctx is used by the watcher, so if the ctx is canceled, the channel will close finishing the loop.
	for event := range watcher.ResultChan() {
		if event.Type == apiwatch.Deleted || event.Type == apiwatch.Error {
			jobErr = fmt.Errorf("error in %v job execution, event type: %v", jobManifest.Name, event.Type)
			break
		}

		jobState, ok := event.Object.(*batchv1.Job)
		if ok && jobState.Status.Failed > 0 {
			jobErr = fmt.Errorf("error in %v job execution, job failed", jobManifest.Name)
			break
		}

		if ok && jobState.Status.Succeeded > 0 {
			break
		}
	}

	// We need this condition to check when the ctx was cancelled due to a timeout. In that case, the previous
	// watcher.ResultChan stops without reporting an error.
	if ctx.Err() != nil && jobErr == nil {
		jobErr = ctx.Err()
		// Sometimes the timeout reports an error before the job result channel, even though the job already failed.
		// This is to do a last check and assign the appropriate error.
		if t.isJobErr(context.TODO(), jobManifest.Name, jobsManager) {
			jobErr = fmt.Errorf("error in %v job execution, job failed", jobManifest.Name)
		}
	}

View on GitHub (pinned to a1189de023)

Solutions

Inspect the failed pod logs: kubectl logs job/<job> -n <ns> (or kubectl describe job) to find the container exit reason, then fix the action's command/args.
Raise resources (limits/requests) or fix OOM/eviction causes in the job manifest's container spec.
Check image availability and pull secrets for the job's namespace if the failure stems from ImagePullBackOff inside the job pod.

Example fix

// before (job container)
args: ["run-tests", "--suite", "integration"]  # exit 1 -> job failed
// after
args: ["run-tests", "--suite", "unit"]  # or fix the failing test, verify locally first

Defensive patterns

Strategy: try-catch

Validate before calling

// Smoke-test the action command locally before running as a Job
docker run --rm <action-image> <args...> || echo 'action command fails locally too'

Try / catch

try {
  await skaffold.verify(...);
} catch (e) {
  if (/job failed/.test(String(e))) {
    const job = String(e).match(/error in (\S+) job execution/)?.[1];
    console.error(`Inspect logs: kubectl logs job/${job} -n <ns>`);
  }
  throw e;
}

Prevention

Run the action command in a local container first to validate exit code
Set sane resource requests/limits to avoid OOMKill and evictions
Check imagePullSecrets are present in the target namespace

When it happens

Trigger: A k8sjob action's Job reaches Status.Failed > 0: the action container exits non-zero, hits backOffLimit restarts, or its pod is evicted/OOMKilled during the skaffold run.

Common situations: Test command inside the job fails (non-zero exit); insufficient resources so pods get evicted; image pull errors inside the job pod; wrong env/args causing the entrypoint to crash; backoffLimit exhausted after transient node failures.

Related errors

AI-assisted analysis of GoogleContainerTools/skaffold@a1189de023 (2026-09-05). Data as JSON: /api/errors/cd4f825d6f69365b. Report an issue: GitHub.