argoproj/argo-workflows · error

failed to unmarshal webhook client "%s": %w

Error message

failed to unmarshal webhook client "%s": %w

What it means

During webhook request authentication, the interceptor iterates over the `webhook-clients` ConfigMap data entries and YAML-unmarshals each entry into a webhookClient struct. If an entry is not valid YAML or does not match the expected schema (type, secret fields), the loop aborts and wraps the parse error with the offending service account key name. This prevents authenticating the incoming webhook at all, so the request fails authorization.

Source

Thrown at server/auth/webhook/interceptor.go:95

	// we need to read the request body to check the signature, but we still need it for the GRPC request,
	// so read it all now, and then reinstate when we are done.
	// Limit to 2MB to prevent denial-of-service via oversized webhook payloads.
	const maxWebhookSize = 2 * 1024 * 1024 // 2MB
	buf, err2 := io.ReadAll(io.LimitReader(r.Body, maxWebhookSize+1))
	if err2 != nil {
		return fmt.Errorf("failed to read webhook request body: %w", err2)
	}
	if len(buf) > maxWebhookSize {
		return fmt.Errorf("webhook request body exceeds maximum size of 2MB")
	}
	defer func() { r.Body = io.NopCloser(bytes.NewBuffer(buf)) }()
	serviceAccountInterface := kube.CoreV1().ServiceAccounts(namespace)
	for serviceAccountName, data := range webhookClients.Data {
		r.Body = io.NopCloser(bytes.NewBuffer(buf))
		client := &webhookClient{}
		err := yaml.Unmarshal(data, client)
		if err != nil {
			return fmt.Errorf("failed to unmarshal webhook client \"%s\": %w", serviceAccountName, err)
		}
		i.logger.WithFields(logging.Fields{"serviceAccountName": serviceAccountName, "webhookType": client.Type}).Debug(r.Context(), "Attempting to match webhook request")
		ok := webhookParsers[client.Type](client.Secret, r)
		if ok {
			i.logger.WithField("serviceAccountName", serviceAccountName).Debug(r.Context(), "Matched webhook request")
			serviceAccount, err := serviceAccountInterface.Get(ctx, serviceAccountName, metav1.GetOptions{})
			if err != nil {
				return fmt.Errorf("failed to get service account \"%s\": %w", serviceAccountName, err)
			}
			tokenSecret, err := secretsInterface.Get(ctx, secrets.TokenNameForServiceAccount(serviceAccount), metav1.GetOptions{})
			if err != nil {
				return fmt.Errorf("failed to get token secret \"%s\": %w", tokenSecret, err)
			}
			r.Header["Authorization"] = []string{"Bearer " + string(tokenSecret.Data["token"])}
			return nil
		}
	}
	return nil

View on GitHub (pinned to 35bff19146)

Solutions

1. Run `kubectl get configmap webhook-clients -n <ns> -o yaml` and validate every data key parses as YAML (e.g. pipe each entry through `yamllint` or a local yaml.Unmarshal test).
2. Check each client entry has the expected schema: `type: <github|gitlab|bitbucket|bitbucketserver>` and a `secret:` string.
3. Quote secret values that contain YAML special characters (`:`, `#`, leading/trailing spaces).
4. Use tabs-free indentation (YAML forbids tabs); re-apply the ConfigMap after fixing.
5. Restart/retry the webhook call — the config is re-read per request, no restart needed.

Example fix

# before (broken: unquoted colon, wrong key)
my-github:
  typ: github
  secret: foo:bar
# after
my-github:
  type: github
  secret: "foo:bar"

// Validate every webhook-clients entry before applying
cm, _ := clientset.CoreV1().ConfigMaps(ns).Get(ctx, "webhook-clients", metav1.GetOptions{})
for name, data := range cm.Data {
	var c struct{ Type, Secret string }
	if err := yaml.Unmarshal([]byte(data), &c); err != nil {
		return fmt.Errorf("entry %q invalid: %w", name, err)
	}
	if c.Type == "" { return fmt.Errorf("entry %q missing type", name) }
}

func validWebhookClient(raw []byte) (*webhookClient, bool) {
	var c webhookClient
	if err := yaml.Unmarshal(raw, &c); err != nil || c.Type == "" || c.Secret == "" { return nil, false }
	return &c, true
}

When it happens

Trigger: An incoming webhook (GitHub/Bitbucket/GitLab etc.) hits the Argo server's /api/v1/events/{namespace}/{discriminator} endpoint while some key in the `webhook-clients` ConfigMap contains malformed YAML, wrong indentation, tabs, or unknown/misspelled fields (e.g. `typ: github`), or non-string values where strings are expected.

Common situations: Hand-editing the ConfigMap with kubectl edit and breaking indentation; applying YAML parsed through a templating tool that emitted invalid output; pasting a secret containing special characters (e.g. `#`, `:`) without quoting; upgrading Argo with a ConfigMap written for an older schema.

Related errors

• failed to get token secret "%s": %w
• failed to unmarshall config map key %q for artifact reposito
• --client-certificate and --client-key must be provided toget
• no workflow found in given files
• memoization configmap doesn't have %s label, refusing to use

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/91e3e436513b8cab. Report an issue: GitHub.