argoproj/argo-workflows · error

failed to get webhook clients: %w

Error message

failed to get webhook clients: %w

What it means

The webhook auth interceptor fetches the `argo-workflows-webhook-clients` Secret in the request's namespace to look up HMAC credentials; any failure reading it (NotFound, RBAC denial, API server error) is wrapped in this error and the webhook request is rejected.

Source

Thrown at server/auth/webhook/interceptor.go:75

	}
}

func (i *Interceptor) addWebhookAuthorization(r *http.Request, kube kubernetes.Interface) error {
	// try and exit quickly before we do anything API calls
	if r.Method != http.MethodPost || len(r.Header["Authorization"]) > 0 || !strings.HasPrefix(r.URL.Path, pathPrefix) {
		return nil
	}
	parts := strings.SplitN(strings.TrimPrefix(r.URL.Path, pathPrefix), "/", 2)
	if len(parts) != 2 {
		return nil
	}
	namespace := parts[0]
	secretsInterface := kube.CoreV1().Secrets(namespace)
	ctx := r.Context()

	webhookClients, err := secretsInterface.Get(ctx, "argo-workflows-webhook-clients", metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("failed to get webhook clients: %w", err)
	}
	// we need to read the request body to check the signature, but we still need it for the GRPC request,
	// so read it all now, and then reinstate when we are done.
	// Limit to 2MB to prevent denial-of-service via oversized webhook payloads.
	const maxWebhookSize = 2 * 1024 * 1024 // 2MB
	buf, err2 := io.ReadAll(io.LimitReader(r.Body, maxWebhookSize+1))
	if err2 != nil {
		return fmt.Errorf("failed to read webhook request body: %w", err2)
	}
	if len(buf) > maxWebhookSize {
		return fmt.Errorf("webhook request body exceeds maximum size of 2MB")
	}
	defer func() { r.Body = io.NopCloser(bytes.NewBuffer(buf)) }()
	serviceAccountInterface := kube.CoreV1().ServiceAccounts(namespace)
	for serviceAccountName, data := range webhookClients.Data {
		r.Body = io.NopCloser(bytes.NewBuffer(buf))
		client := &webhookClient{}
		err := yaml.Unmarshal(data, client)

View on GitHub (pinned to 35bff19146)

Solutions

1. Create the Secret `argo-workflows-webhook-clients` in the target namespace with the client credentials
2. Verify the webhook URL namespace segment is correct
3. Grant the argo-server service account RBAC get on secrets in that namespace
4. Check k8s API health if errors are cluster-wide

Example fix

kubectl create secret generic argo-workflows-webhook-clients \
  -n <namespace> \
  --from-literal=bitbucket=shh... \
  --from-literal=github=shh...

_, err := kube.CoreV1().Secrets(ns).Get(ctx, "argo-workflows-webhook-clients", metav1.GetOptions{})
if apierrors.IsNotFound(err) { /* create secret or skip webhook auth */ }

if err := webhook.Verify(r); err != nil {
    if apierrors.IsNotFound(errors.Unwrap(err)) {
        // secret missing: provision it or return 503 with setup hint
    }
    return status.Error(codes.Unauthenticated, "webhook auth failed")
}

When it happens

Trigger: An HTTP request hits a webhook endpoint path and the argo-server's service account cannot GET the Secret `argo-workflows-webhook-clients` in the target namespace — secret not created, wrong namespace in the URL, or RBAC not granting secrets get.

Common situations: Upgrading to a version that requires the webhook secret without creating it; webhook URL namespace typo; ClusterRole/Role lacking `secrets` `get` for the argo-server service account; Kubernetes API outage.

Related errors

• failed to get service account "%s": %w
• failed to get existing cluster workflow template %q to updat
• failed to list SSO RBAC service accounts: %w
• failed to get service account secret: %w
• failed to create secret: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/4719b152fcbbfa14. Report an issue: GitHub.