argoproj/argo-workflows · error

failed to unmarshall kill command annotation %q: %w

Error message

failed to unmarshall kill command annotation %q: %w

What it means

argoexec determines how to kill a container's process by reading the pod's kill-command annotation (common.AnnotationKeyKillCmd), which stores a JSON-encoded argv. killCommand parses it with json.Unmarshal; if the annotation value is not valid JSON (or the wrong shape), the error wraps the raw value and the underlying unmarshal error, and the container cannot be signalled.

Source

Thrown at workflow/signal/signal.go:69

			for _, m := range c.VolumeMounts {
				switch m.MountPath {
				case common.VarRunArgoPath:
					hasVarRunArgo = true
				case common.ArgoExecBinMountPath:
					hasArgoExecBin = true
				}
			}
			switch {
			case hasArgoExecBin:
				command = []string{common.ArgoExecBinPath, "kill", "%d", "1"}
			case hasVarRunArgo:
				command = []string{common.LegacyArgoExecBinPath, "kill", "%d", "1"}
			}
		}
	}
	if v, ok := pod.Annotations[common.AnnotationKeyKillCmd(container)]; ok {
		if err := json.Unmarshal([]byte(v), &command); err != nil {
			return nil, fmt.Errorf("failed to unmarshall kill command annotation %q: %w", v, err)
		}
	}

	for i, v := range command {
		if strings.Contains(v, "%d") {
			command[i] = fmt.Sprintf(v, s)
		}
	}
	return command, nil
}

func ExecPodContainerAndGetOutput(ctx context.Context, restConfig *rest.Config, namespace string, pod string, container string, command ...string) error {
	x, err := common.ExecPodContainer(ctx, restConfig, namespace, pod, container, true, true, command...)
	if err != nil {
		return err
	}
	// workaround for when exec does not properly return: https://github.com/kubernetes/kubernetes/pull/103177
	ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)

View on GitHub (pinned to 35bff19146)

Solutions

1. Delete/recreate the pod so the controller rewrites the annotation correctly
2. Inspect the annotation value (kubectl get pod <pod> -o jsonpath='{.metadata.annotations...}') and fix it to valid JSON, e.g. ["argoexec","kill","%d","1"]
3. Check for mutation webhooks or tooling rewriting pod annotations and exclude Argo-managed annotations
4. Align workflow-controller and argoexec versions so the annotation format matches
5. If the workflow is stuck, terminate it and resubmit; report a bug if the controller itself writes malformed JSON

Example fix

# before: corrupted annotation
annotations:
  workflows.argoproj.io/kill-cmd-main: argoexec kill %d 1
# after: valid JSON argv
annotations:
  workflows.argoproj.io/kill-cmd-main: '["argoexec", "kill", "%d", "1"]'

// verify the annotation is valid JSON argv before relying on kill behavior
var cmd []string
if v, ok := pod.Annotations[annotationKey]; ok {
	if err := json.Unmarshal([]byte(v), &cmd); err != nil {
		return fmt.Errorf("pod %s has corrupt kill-cmd annotation: %w", pod.Name, err)
	}
}

func hasValidKillCmd(pod *corev1.Pod, container string) bool {
	v, ok := pod.Annotations[fmt.Sprintf("workflows.argoproj.io/kill-cmd-%s", container)]
	if !ok { return false }
	var cmd []string
	return json.Unmarshal([]byte(v), &cmd) == nil && len(cmd) > 0
}

if err != nil && strings.Contains(err.Error(), "failed to unmarshall kill command annotation") {
	// recreate the pod or fall back to the default kill command
}

When it happens

Trigger: Container kill path (process/pause) reads pod.Annotations[AnnotationKeyKillCmd] whose value is malformed JSON — e.g. the annotation was hand-edited, truncated by another tool, written by an incompatible controller/executor version, or corrupted by mutation webhooks.

Common situations: Pod objects modified by external controllers/webhooks stripping quoting; mismatch between controller version writing the annotation and executor version reading it; users debugging pods and manually editing annotations; manifests copied between clusters with escaped-quote corruption.

Related errors

• failed to unmarshal container args: %w
• failed to unmarshal template: %w
• invalid TTL
• error converting %s: %w
• failed to parse TypeMeta: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a9fa9ae8b68138d6. Report an issue: GitHub.