argoproj/argo-workflows · warning

failed to read webhook request body: %w

Error message

failed to read webhook request body: %w

What it means

The interceptor must buffer the entire webhook request body to verify HMAC signatures and then reinstate it for the gRPC handler. If io.ReadAll fails (client disconnect, network error mid-body, timeout) the request is rejected with this wrapped error.

Source

Thrown at server/auth/webhook/interceptor.go:83

	parts := strings.SplitN(strings.TrimPrefix(r.URL.Path, pathPrefix), "/", 2)
	if len(parts) != 2 {
		return nil
	}
	namespace := parts[0]
	secretsInterface := kube.CoreV1().Secrets(namespace)
	ctx := r.Context()

	webhookClients, err := secretsInterface.Get(ctx, "argo-workflows-webhook-clients", metav1.GetOptions{})
	if err != nil {
		return fmt.Errorf("failed to get webhook clients: %w", err)
	}
	// we need to read the request body to check the signature, but we still need it for the GRPC request,
	// so read it all now, and then reinstate when we are done.
	// Limit to 2MB to prevent denial-of-service via oversized webhook payloads.
	const maxWebhookSize = 2 * 1024 * 1024 // 2MB
	buf, err2 := io.ReadAll(io.LimitReader(r.Body, maxWebhookSize+1))
	if err2 != nil {
		return fmt.Errorf("failed to read webhook request body: %w", err2)
	}
	if len(buf) > maxWebhookSize {
		return fmt.Errorf("webhook request body exceeds maximum size of 2MB")
	}
	defer func() { r.Body = io.NopCloser(bytes.NewBuffer(buf)) }()
	serviceAccountInterface := kube.CoreV1().ServiceAccounts(namespace)
	for serviceAccountName, data := range webhookClients.Data {
		r.Body = io.NopCloser(bytes.NewBuffer(buf))
		client := &webhookClient{}
		err := yaml.Unmarshal(data, client)
		if err != nil {
			return fmt.Errorf("failed to unmarshal webhook client \"%s\": %w", serviceAccountName, err)
		}
		i.logger.WithFields(logging.Fields{"serviceAccountName": serviceAccountName, "webhookType": client.Type}).Debug(r.Context(), "Attempting to match webhook request")
		ok := webhookParsers[client.Type](client.Secret, r)
		if ok {
			i.logger.WithField("serviceAccountName", serviceAccountName).Debug(r.Context(), "Matched webhook request")
			serviceAccount, err := serviceAccountInterface.Get(ctx, serviceAccountName, metav1.GetOptions{})

View on GitHub (pinned to 35bff19146)

Solutions

1. Retry the webhook delivery from the source (GitHub/Bitbucket redeliver)
2. Check ingress/proxy timeouts and raise them for the argo-server webhook route
3. Verify TLS configuration between the sender and argo-server isn't causing mid-stream resets
4. Look at argo-server logs for repeated cancellations to identify the network hop failing

if r.ContentLength > maxWebhookSize { /* reject early instead of failing mid-read */ }

buf, err := io.ReadAll(io.LimitReader(r.Body, maxWebhookSize+1))
if err != nil {
    if errors.Is(err, context.Canceled) || errors.Is(err, io.ErrUnexpectedEOF) {
        // client aborted: log at debug, let provider redeliver
    }
    return err
}

When it happens

Trigger: addWebhookAuthorization reads the request body and the underlying connection errors: client aborted the POST, proxy dropped the connection, TLS/network interruption, or request context canceled before the body completed.

Common situations: Flaky networks between Git host (GitHub/Bitbucket) and argo-server; aggressive ingress/LLM proxies with short idle timeouts; clients canceling large webhook deliveries.

Related errors

• io copy: %w
• failed to stream artifact: %v
• CodeNotFound
• failed to get workflow: %w
• failed to get and store artifact data: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/ccc1b5795d86e3b5. Report an issue: GitHub.