{"record":{"id":"dc3ba150f3fade52","repo":"dapr/dapr","slug":"error-receiving-message-w-dc3ba1","errorCode":null,"errorMessage":"error receiving message: %w","messagePattern":"error receiving message: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/messaging/direct_messaging.go","lineNumber":552,"sourceCode":"\t\t\t\t\tpw.CloseWithError(readErr)\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\t// Check if the sequence number is greater than the previous\n\t\t\t\tif readSeq != expectSeq {\n\t\t\t\t\tpw.CloseWithError(fmt.Errorf(\"invalid sequence number received: %d (expected: %d)\", readSeq, expectSeq))\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\texpectSeq++\n\t\t\t}\n\n\t\t\t// Read the next chunk\n\t\t\treadErr = stream.RecvMsg(chunk)\n\t\t\tif errors.Is(readErr, io.EOF) {\n\t\t\t\t// Receiving an io.EOF signifies that the client has stopped sending data over the pipe, so we can stop reading\n\t\t\t\tbreak\n\t\t\t} else if readErr != nil {\n\t\t\t\tpw.CloseWithError(fmt.Errorf(\"error receiving message: %w\", readErr))\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tif chunk.GetResponse().GetStatus() != nil || chunk.GetResponse().GetHeaders() != nil || chunk.GetResponse().GetMessage() != nil {\n\t\t\t\tpw.CloseWithError(errors.New(\"response metadata found in non-leading chunk\"))\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\n\t\tpw.Close()\n\t}()\n\n\treturn res, nil\n}\n\nfunc (d *directMessaging) addDestinationAppIDHeaderToMetadata(appID string, req *invokev1.InvokeMethodRequest) {\n\treq.Metadata()[invokev1.DestinationIDHeader] = &internalv1pb.ListStringValue{\n\t\tValues: []string{appID},","sourceCodeStart":534,"sourceCodeEnd":570,"githubUrl":"https://github.com/dapr/dapr/blob/74ad41702745709bb15fe2114ff693b8c59bc3cc/pkg/messaging/direct_messaging.go#L534-L570","documentation":"While draining the response stream, stream.RecvMsg(chunk) returned an error other than io.EOF (EOF is the normal end marker that breaks the loop). The goroutine wraps that error with %w and closes the response pipe with it, so the caller's body read fails. The root cause is always the underlying gRPC transport: connection reset, deadline exceeded, unavailable target, TLS failure, or message-size limits.","triggerScenarios":"Network interruption between sidecars mid-response; target app or sidecar crashing while the response streams; gRPC context deadline exceeded; keepalive/GOAWAY from a proxy; payload exceeding receiver message-size limits arriving as a status error.","commonSituations":"Target pod restarting during a long or large invocation; cross-node/cluster network drops; oversized payloads when streaming is not used; mTLS certificate rotation mid-stream.","solutions":["Read the wrapped error: extract the gRPC status code - UNAVAILABLE/DEADLINE_EXCEEDED are transient and safe to retry; RESOURCE_EXHAUSTED means payload too large","Add a Resiliency policy with timeout plus retries for service invocation to that app","Check target app health and logs for crashes at the time of the call","For RESOURCE_EXHAUSTED, raise dapr.io/max-request-size / use streaming-compatible clients so payloads travel as chunks"],"exampleFix":"cat <<'EOF' | kubectl apply -f -\napiVersion: dapr.io/v1alpha1\nkind: Resiliency\nmetadata:\n  name: net-retry\nspec:\n  policies:\n    timeouts:\n      general: 30s\n    retries:\n      transient:\n        policy: exponential\n        maxInterval: 5s\n        maxRetries: 4\n    circuitBreakers:\n      netCB:\n        maxRequests: 1\n        trip: consecutiveFailures > 5\n  targets:\n    apps:\n      myapp:\n        timeout: general\n        retry: transient\n        circuitBreaker: netCB\nEOF","handlingStrategy":"retry","validationCode":null,"typeGuard":"func transientRecvError(err error) bool {\n\ts, ok := status.FromError(errors.Unwrap(err))\n\tif !ok {\n\t\treturn false\n\t}\n\tswitch s.Code() {\n\tcase codes.Unavailable, codes.DeadlineExceeded:\n\t\treturn true\n\t}\n\treturn false\n}","tryCatchPattern":"if _, err := io.ReadAll(resp.Body); err != nil {\n\tif transientRecvError(err) {\n\t\t// network-level drop mid-stream: retry with backoff\n\t\treturn retryInvoke(ctx, req, 3, 500*time.Millisecond)\n\t}\n\treturn fmt.Errorf(\"non-retryable stream failure: %w\", err)\n}","preventionTips":["Set explicit Resiliency timeouts and retries for cross-app invocation","Monitor target app health so mid-response crashes are caught before traffic hits them","Keep gRPC max message sizes consistent between caller and callee for non-streamed payloads","Prefer streaming APIs for large payloads to stay clear of message-size aborts mid-response"],"tags":["network","grpc","streaming","service-invocation"],"backgroundTag":null,"analyzedSha":"74ad41702745709bb15fe2114ff693b8c59bc3cc","analyzedAt":"2026-08-16T04:22:26.543Z","schemaVersion":2},"datasetVersion":"2026-08-16T08:17:34.114Z"}