dgraph-io/dgraph · error

recv final downstream(%d): %w

Error message

recv final downstream(%d): %w

What it means

While draining the downstream for the Finish frame, out.Recv() failed with a non-EOF error. The relay cannot learn the import outcome and aborts with this wrapped error identifying the group. Unlike EOF (816), this indicates an active transport/RPC failure rather than a clean close.

Source

Thrown at worker/import.go:430

		if req.Pkt.Done {
			// Forward Done, half-close downstream send.
			if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil && !errors.Is(err, io.EOF) {
				return fmt.Errorf("send done downstream(%d): %w", groupId, err)
			}
			_ = out.CloseSend()

			// Drain downstream and relay upstream until Finish=true.
			for {
				if err := ctx.Err(); err != nil {
					return err
				}
				resp, err := out.Recv()
				if errors.Is(err, io.EOF) {
					return fmt.Errorf("downstream(%d) closed before Finish=true", groupId)
				}
				if err != nil {
					return fmt.Errorf("recv final downstream(%d): %w", groupId, err)
				}
				if err := in.Send(resp); err != nil {
					return fmt.Errorf("relay final upstream: %w", err)
				}
				if resp.Finish {
					glog.Infof("[import] [forward %d -> %d] finish", currentGroup, groupId)
					return nil
				}
			}
		}

		// Normal data chunk: send -> wait ack -> send upstream ack.
		if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil {
			return fmt.Errorf("send data downstream(%d): %w", groupId, err)
		}
		if _, err := out.Recv(); err != nil {
			return fmt.Errorf("ack data downstream(%d): %w", groupId, err)
		}

View on GitHub (pinned to 759e242be6)

Solutions

1. Inspect the wrapped status code: Unavailable/transport => retry the import; Canceled => find what cancelled the context
2. Raise context deadlines to cover slow finalization on large snapshots
3. Verify leader stability and connectivity, then re-run the import
4. Enable gRPC keepalives to detect and survive idle periods gracefully

ctx, cancel := context.WithTimeout(context.Background(), generousDeadlineForSnapshot)
defer cancel()

func isTransientRecvFailure(err error) bool {
    c := status.Code(err)
    return c == codes.Unavailable || c == codes.DeadlineExceeded
}

if err != nil && strings.Contains(err.Error(), "recv final downstream") {
    switch status.Code(errors.Unwrap(err)) {
    case codes.Unavailable, codes.DeadlineExceeded: /* retry import */
    case codes.Canceled: /* find the cancelling caller */
    }
}

When it happens

Trigger: out.Recv() returns codes.Canceled (context cancelled), codes.Unavailable (connection lost), DeadlineExceeded, or a transport error while waiting for the leader's final frames.

Common situations: Network partition during the final phase; leader restarted; overall import context deadline hit during a slow final flush; keepalive probe failure killing the connection.

Related errors

• while calling MovePredicate
• failed to establish stream with leader: %v
• failed to send forward request: %v
• ack data downstream(%d): %w
• send ack upstream: %w

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/8f0a7ad043fdfed6. Report an issue: GitHub.