dgraph-io/dgraph · error

relay final upstream: %w

Error message

relay final upstream: %w

What it means

After receiving a response from the leader during the final drain, the relay must echo it back to the import client via in.Send(resp). If that send fails, the upstream client can no longer be informed of progress and piping aborts with this wrapped error.

Source

Thrown at worker/import.go:433

			if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil && !errors.Is(err, io.EOF) {
				return fmt.Errorf("send done downstream(%d): %w", groupId, err)
			}
			_ = out.CloseSend()

			// Drain downstream and relay upstream until Finish=true.
			for {
				if err := ctx.Err(); err != nil {
					return err
				}
				resp, err := out.Recv()
				if errors.Is(err, io.EOF) {
					return fmt.Errorf("downstream(%d) closed before Finish=true", groupId)
				}
				if err != nil {
					return fmt.Errorf("recv final downstream(%d): %w", groupId, err)
				}
				if err := in.Send(resp); err != nil {
					return fmt.Errorf("relay final upstream: %w", err)
				}
				if resp.Finish {
					glog.Infof("[import] [forward %d -> %d] finish", currentGroup, groupId)
					return nil
				}
			}
		}

		// Normal data chunk: send -> wait ack -> send upstream ack.
		if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil {
			return fmt.Errorf("send data downstream(%d): %w", groupId, err)
		}
		if _, err := out.Recv(); err != nil {
			return fmt.Errorf("ack data downstream(%d): %w", groupId, err)
		}
		if err := in.Send(&api.StreamExtSnapshotResponse{}); err != nil {
			return fmt.Errorf("send ack upstream: %w", err)
		}

View on GitHub (pinned to 759e242be6)

Solutions

1. Check the wrapped error: Canceled indicates the client gave up — increase client timeout for large snapshots
2. Re-run the import with a stable client connection
3. Remove idle-timeout proxies on the client-to-alpha path
4. Keep the client alive until Finish=true is received

// client must stay connected for the whole final phase
ctx, cancel := context.WithTimeout(context.Background(), generousDeadlineForSnapshot)
defer cancel()

func clientCancelled(err error) bool { return status.Code(err) == codes.Canceled }

if err != nil && strings.Contains(err.Error(), "relay final upstream") {
    if clientCancelled(errors.Unwrap(err)) {
        // client gave up: increase client timeout and re-import
    }
}

When it happens

Trigger: in.Send(resp) errors because the import client disconnected, cancelled its context, or its stream was torn down by a transport error while the leader was still responding.

Common situations: Client timed out waiting during the leader's slow final flush and cancelled; client process died near the end of a long import; proxy between client and alpha closed the stream.

Related errors

• recv upstream(%d): %w
• send done downstream(%d): %w
• send data downstream(%d): %w
• while calling MovePredicate
• failed to establish stream with leader: %v

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/d4bf2defabe2f2a3. Report an issue: GitHub.