dapr/dapr · error

error receiving message: %w

Error message

error receiving message: %w

What it means

While draining the response stream, stream.RecvMsg(chunk) returned an error other than io.EOF (EOF is the normal end marker that breaks the loop). The goroutine wraps that error with %w and closes the response pipe with it, so the caller's body read fails. The root cause is always the underlying gRPC transport: connection reset, deadline exceeded, unavailable target, TLS failure, or message-size limits.

Source

Thrown at pkg/messaging/direct_messaging.go:552

					pw.CloseWithError(readErr)
					return
				}

				// Check if the sequence number is greater than the previous
				if readSeq != expectSeq {
					pw.CloseWithError(fmt.Errorf("invalid sequence number received: %d (expected: %d)", readSeq, expectSeq))
					return
				}
				expectSeq++
			}

			// Read the next chunk
			readErr = stream.RecvMsg(chunk)
			if errors.Is(readErr, io.EOF) {
				// Receiving an io.EOF signifies that the client has stopped sending data over the pipe, so we can stop reading
				break
			} else if readErr != nil {
				pw.CloseWithError(fmt.Errorf("error receiving message: %w", readErr))
				return
			}

			if chunk.GetResponse().GetStatus() != nil || chunk.GetResponse().GetHeaders() != nil || chunk.GetResponse().GetMessage() != nil {
				pw.CloseWithError(errors.New("response metadata found in non-leading chunk"))
				return
			}
		}

		pw.Close()
	}()

	return res, nil
}

func (d *directMessaging) addDestinationAppIDHeaderToMetadata(appID string, req *invokev1.InvokeMethodRequest) {
	req.Metadata()[invokev1.DestinationIDHeader] = &internalv1pb.ListStringValue{
		Values: []string{appID},

View on GitHub (pinned to 74ad417027)

Solutions

1. Read the wrapped error: extract the gRPC status code - UNAVAILABLE/DEADLINE_EXCEEDED are transient and safe to retry; RESOURCE_EXHAUSTED means payload too large
2. Add a Resiliency policy with timeout plus retries for service invocation to that app
3. Check target app health and logs for crashes at the time of the call
4. For RESOURCE_EXHAUSTED, raise dapr.io/max-request-size / use streaming-compatible clients so payloads travel as chunks

Example fix

cat <<'EOF' | kubectl apply -f -
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
  name: net-retry
spec:
  policies:
    timeouts:
      general: 30s
    retries:
      transient:
        policy: exponential
        maxInterval: 5s
        maxRetries: 4
    circuitBreakers:
      netCB:
        maxRequests: 1
        trip: consecutiveFailures > 5
  targets:
    apps:
      myapp:
        timeout: general
        retry: transient
        circuitBreaker: netCB
EOF

func transientRecvError(err error) bool {
	s, ok := status.FromError(errors.Unwrap(err))
	if !ok {
		return false
	}
	switch s.Code() {
	case codes.Unavailable, codes.DeadlineExceeded:
		return true
	}
	return false
}

if _, err := io.ReadAll(resp.Body); err != nil {
	if transientRecvError(err) {
		// network-level drop mid-stream: retry with backoff
		return retryInvoke(ctx, req, 3, 500*time.Millisecond)
	}
	return fmt.Errorf("non-retryable stream failure: %w", err)
}

When it happens

Trigger: Network interruption between sidecars mid-response; target app or sidecar crashing while the response streams; gRPC context deadline exceeded; keepalive/GOAWAY from a proxy; payload exceeding receiver message-size limits arriving as a status error.

Common situations: Target pod restarting during a long or large invocation; cross-node/cluster network drops; oversized payloads when streaming is not used; mTLS certificate rotation mid-stream.

Related errors

• response does not contain the required fields in the leading
• response metadata found in non-leading chunk
• target app '%s' is running a version of Dapr that does not s
• error watching scheduler jobs: %w
• server is closed

AI-assisted analysis of dapr/dapr@74ad417027 (2026-08-16). Data as JSON: /api/errors/dc3ba150f3fade52. Report an issue: GitHub.