{"record":{"id":"a9fa9ae8b68138d6","repo":"argoproj/argo-workflows","slug":"failed-to-unmarshall-kill-command-annotation-q","errorCode":null,"errorMessage":"failed to unmarshall kill command annotation %q: %w","messagePattern":"failed to unmarshall kill command annotation %q: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/signal/signal.go","lineNumber":69,"sourceCode":"\t\t\tfor _, m := range c.VolumeMounts {\n\t\t\t\tswitch m.MountPath {\n\t\t\t\tcase common.VarRunArgoPath:\n\t\t\t\t\thasVarRunArgo = true\n\t\t\t\tcase common.ArgoExecBinMountPath:\n\t\t\t\t\thasArgoExecBin = true\n\t\t\t\t}\n\t\t\t}\n\t\t\tswitch {\n\t\t\tcase hasArgoExecBin:\n\t\t\t\tcommand = []string{common.ArgoExecBinPath, \"kill\", \"%d\", \"1\"}\n\t\t\tcase hasVarRunArgo:\n\t\t\t\tcommand = []string{common.LegacyArgoExecBinPath, \"kill\", \"%d\", \"1\"}\n\t\t\t}\n\t\t}\n\t}\n\tif v, ok := pod.Annotations[common.AnnotationKeyKillCmd(container)]; ok {\n\t\tif err := json.Unmarshal([]byte(v), &command); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to unmarshall kill command annotation %q: %w\", v, err)\n\t\t}\n\t}\n\n\tfor i, v := range command {\n\t\tif strings.Contains(v, \"%d\") {\n\t\t\tcommand[i] = fmt.Sprintf(v, s)\n\t\t}\n\t}\n\treturn command, nil\n}\n\nfunc ExecPodContainerAndGetOutput(ctx context.Context, restConfig *rest.Config, namespace string, pod string, container string, command ...string) error {\n\tx, err := common.ExecPodContainer(ctx, restConfig, namespace, pod, container, true, true, command...)\n\tif err != nil {\n\t\treturn err\n\t}\n\t// workaround for when exec does not properly return: https://github.com/kubernetes/kubernetes/pull/103177\n\tctx, cancel := context.WithTimeout(ctx, 2*time.Minute)","sourceCodeStart":51,"sourceCodeEnd":87,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/signal/signal.go#L51-L87","documentation":"argoexec determines how to kill a container's process by reading the pod's kill-command annotation (common.AnnotationKeyKillCmd), which stores a JSON-encoded argv. killCommand parses it with json.Unmarshal; if the annotation value is not valid JSON (or the wrong shape), the error wraps the raw value and the underlying unmarshal error, and the container cannot be signalled.","triggerScenarios":"Container kill path (process/pause) reads pod.Annotations[AnnotationKeyKillCmd] whose value is malformed JSON — e.g. the annotation was hand-edited, truncated by another tool, written by an incompatible controller/executor version, or corrupted by mutation webhooks.","commonSituations":"Pod objects modified by external controllers/webhooks stripping quoting; mismatch between controller version writing the annotation and executor version reading it; users debugging pods and manually editing annotations; manifests copied between clusters with escaped-quote corruption.","solutions":["Delete/recreate the pod so the controller rewrites the annotation correctly","Inspect the annotation value (kubectl get pod <pod> -o jsonpath='{.metadata.annotations...}') and fix it to valid JSON, e.g. [\"argoexec\",\"kill\",\"%d\",\"1\"]","Check for mutation webhooks or tooling rewriting pod annotations and exclude Argo-managed annotations","Align workflow-controller and argoexec versions so the annotation format matches","If the workflow is stuck, terminate it and resubmit; report a bug if the controller itself writes malformed JSON"],"exampleFix":"# before: corrupted annotation\nannotations:\n  workflows.argoproj.io/kill-cmd-main: argoexec kill %d 1\n# after: valid JSON argv\nannotations:\n  workflows.argoproj.io/kill-cmd-main: '[\"argoexec\", \"kill\", \"%d\", \"1\"]'","handlingStrategy":"validation","validationCode":"// verify the annotation is valid JSON argv before relying on kill behavior\nvar cmd []string\nif v, ok := pod.Annotations[annotationKey]; ok {\n\tif err := json.Unmarshal([]byte(v), &cmd); err != nil {\n\t\treturn fmt.Errorf(\"pod %s has corrupt kill-cmd annotation: %w\", pod.Name, err)\n\t}\n}","typeGuard":"func hasValidKillCmd(pod *corev1.Pod, container string) bool {\n\tv, ok := pod.Annotations[fmt.Sprintf(\"workflows.argoproj.io/kill-cmd-%s\", container)]\n\tif !ok { return false }\n\tvar cmd []string\n\treturn json.Unmarshal([]byte(v), &cmd) == nil && len(cmd) > 0\n}","tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"failed to unmarshall kill command annotation\") {\n\t// recreate the pod or fall back to the default kill command\n}","preventionTips":["Never hand-edit Argo pod annotations","Audit mutation webhooks for rewriting workflows.argoproj.io/* annotations","Keep workflow-controller and argoexec versions aligned","Recreate stuck pods rather than patching annotations manually"],"tags":["executor","pod-annotation","json","kill-signal"],"backgroundTag":"annotation-json-unmarshal-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}