What causes "recv upstream(%d): %w" in dgraph-io/dgraph?

Inspect the wrapped error: Canceled means the caller aborted, Unavailable/EOF means transport loss

recv upstream(%d): %w

Error message

recv upstream(%d): %w

What it means

pipeTwoStream relays the upstream import stream (from the client) to the downstream leader. Reading from the upstream with in.Recv() failed with an error other than io.EOF, so piping aborts with this wrapped error identifying the group whose upstream read failed. It signals the import client stopped sending unexpectedly.

Source

Thrown at worker/import.go:407

	return pipeTwoStream(stream, alphaStream, groupId)
}

func pipeTwoStream(in api.Dgraph_StreamExtSnapshotServer, out pb.Worker_StreamExtSnapshotClient, groupId uint32) error {
	currentGroup := groups().Node.gid
	ctx := in.Context()

	for {
		if err := ctx.Err(); err != nil {
			return err
		}

		req, err := in.Recv()
		if errors.Is(err, io.EOF) {
			return nil
		}
		if err != nil {
			return fmt.Errorf("recv upstream(%d): %w", currentGroup, err)
		}
		if req.Pkt == nil {
			return fmt.Errorf("unexpected empty request")
		}

		if req.Pkt.Done {
			// Forward Done, half-close downstream send.
			if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil && !errors.Is(err, io.EOF) {
				return fmt.Errorf("send done downstream(%d): %w", groupId, err)
			}
			_ = out.CloseSend()

			// Drain downstream and relay upstream until Finish=true.
			for {
				if err := ctx.Err(); err != nil {
					return err
				}
				resp, err := out.Recv()

View on GitHub (pinned to 759e242be6)

Solutions

Inspect the wrapped error: Canceled means the caller aborted, Unavailable/EOF means transport loss
Re-run the import from the client once the network is stable
Increase client-side keepalive and any intermediary idle timeouts for long streaming imports
Check client host resources (OOM killer) if the client vanished mid-stream

Defensive patterns

Strategy: try-catch

Validate before calling

// before a long import, ensure the client can hold a long-lived stream
ctx, cancel := context.WithTimeout(context.Background(), maxImportDuration)
defer cancel() // sized generously for the snapshot size

Type guard

func isUpstreamDisconnect(err error) bool {
    return status.Code(err) == codes.Canceled || status.Code(err) == codes.Unavailable || errors.Is(err, io.EOF)
}

Try / catch

if err != nil && strings.Contains(err.Error(), "recv upstream") {
    if isUpstreamDisconnect(errors.Unwrap(err)) {
        // client dropped; restart the import from the client
    }
}

Prevention

Run import clients with sufficient memory to avoid OOM kills
Keep the client process alive for the whole transfer
Raise intermediary idle timeouts for long streams
Add gRPC keepalive on the client

When it happens

Trigger: in.Recv() returns a non-EOF error: the importing client (dgraph live/dgraph incremental importer) disconnected, the client context was cancelled, or a transport error occurred on the client-to-alpha stream.

Common situations: Client process killed or OOMed mid-snapshot; network drop between client and alpha; client hit its own timeout and cancelled the context; load balancer idle timeout closing an apparently quiet stream during slow disk flushes.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/354b39cc0c542758. Report an issue: GitHub.