dgraph-io/dgraph · error

send done downstream(%d): %w

Error message

send done downstream(%d): %w

What it means

When the upstream sends the Done packet, pipeTwoStream forwards it downstream and half-closes its send side, then waits for the leader to finish. If that out.Send of the Done packet fails (and it is not a benign io.EOF), the relay aborts with this wrapped error naming the downstream group.

Source

Thrown at worker/import.go:416

		if err := ctx.Err(); err != nil {
			return err
		}

		req, err := in.Recv()
		if errors.Is(err, io.EOF) {
			return nil
		}
		if err != nil {
			return fmt.Errorf("recv upstream(%d): %w", currentGroup, err)
		}
		if req.Pkt == nil {
			return fmt.Errorf("unexpected empty request")
		}

		if req.Pkt.Done {
			// Forward Done, half-close downstream send.
			if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil && !errors.Is(err, io.EOF) {
				return fmt.Errorf("send done downstream(%d): %w", groupId, err)
			}
			_ = out.CloseSend()

			// Drain downstream and relay upstream until Finish=true.
			for {
				if err := ctx.Err(); err != nil {
					return err
				}
				resp, err := out.Recv()
				if errors.Is(err, io.EOF) {
					return fmt.Errorf("downstream(%d) closed before Finish=true", groupId)
				}
				if err != nil {
					return fmt.Errorf("recv final downstream(%d): %w", groupId, err)
				}
				if err := in.Send(resp); err != nil {
					return fmt.Errorf("relay final upstream: %w", err)
				}

View on GitHub (pinned to 759e242be6)

Solutions

1. Check the wrapped error and the leader's logs for the server-side failure that closed the stream
2. Retry the import after confirming leader health
3. Keep the whole import under any proxy idle/timeout limits or disable them for internal gRPC traffic
4. Treat an io.EOF here as benign — the code deliberately ignores it

// precheck leader disk/memory headroom before finalizing a snapshot
dfEnoughDisk(leaderHost, requiredBytes); memEnough(leaderHost, requiredBytes)

func isBenignEOF(err error) bool { return errors.Is(err, io.EOF) } // code already treats this as benign

if err != nil && strings.Contains(err.Error(), "send done downstream") {
    // leader stream broke during half-close; check leader logs, retry import
}

When it happens

Trigger: out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}) errors because the gRPC client stream to the leader was already closed/broken, the leader errored and tore the stream down, or the shared context was cancelled.

Common situations: Leader crashed or was restarted just as the snapshot transfer finished; long import outlived an intermediary's connection timeout; leader returned an error earlier and closed the stream, making the final Done send fail.

Related errors

• recv upstream(%d): %w
• relay final upstream: %w
• send data downstream(%d): %w
• while calling MovePredicate
• failed to establish stream with leader: %v

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/a004152c929aee6a. Report an issue: GitHub.