dgraph-io/dgraph · error

failed to send forward request: %v

Error message

failed to send forward request: %v

What it means

InStream must first tell the downstream leader that it is forwarding an external snapshot by sending api.StreamExtSnapshotRequest{Forward: true}. If alphaStream.Send fails, the handshake cannot complete and this error is returned with the underlying cause. It almost always means the gRPC stream to the leader died right after being opened.

Source

Thrown at worker/import.go:387

		glog.Errorf("[import] unable to connect to the leader of group [%v]", groupId)
		return fmt.Errorf("unable to connect to the leader of group [%v] : %v", groupId, conn.ErrNoConnection)
	}

	con := pl.Get()
	c := pb.NewWorkerClient(con)
	alphaStream, err := c.StreamExtSnapshot(stream.Context())
	if err != nil {
		glog.Errorf("[import] failed to establish stream with leader: %v", err)
		return fmt.Errorf("failed to establish stream with leader: %v", err)
	}
	glog.Infof("[import] [forward %d -> %d] start", groups().Node.gid, groupId)
	glog.Infof("[import] [forward %v -> %d] start", groups().Node.MyAddr, groups().Leader(groupId).Addr)

	glog.Infof("[import] sending forward true to leader of group [%v]", groupId)
	forwardReq := &api.StreamExtSnapshotRequest{Forward: true}
	if err := alphaStream.Send(forwardReq); err != nil {
		glog.Errorf("[import] failed to send forward request: %v", err)
		return fmt.Errorf("failed to send forward request: %v", err)
	}

	return pipeTwoStream(stream, alphaStream, groupId)
}

func pipeTwoStream(in api.Dgraph_StreamExtSnapshotServer, out pb.Worker_StreamExtSnapshotClient, groupId uint32) error {
	currentGroup := groups().Node.gid
	ctx := in.Context()

	for {
		if err := ctx.Err(); err != nil {
			return err
		}

		req, err := in.Recv()
		if errors.Is(err, io.EOF) {
			return nil
		}

View on GitHub (pinned to 759e242be6)

Solutions

1. Read the wrapped err — io.EOF usually means the leader closed the stream; check the leader's logs for the server-side reason
2. Verify the leader process is still alive and re-establish connectivity, then retry the import
3. Ensure both sides use compatible StreamExtSnapshot message versions
4. Check for network devices between nodes that kill long-lived gRPC streams (idle timeouts)

Example fix

// before
if err := alphaStream.Send(forwardReq); err != nil {
    return fmt.Errorf("failed to send forward request: %v", err)
}
// after: log the leader address alongside the cause for faster diagnosis
if err := alphaStream.Send(forwardReq); err != nil {
    glog.Errorf("[import] forward handshake to %s failed: %v", pl.Addr, err)
    return fmt.Errorf("failed to send forward request: %w", err)
}

// confirm leader stability immediately before import
if err := pingLeader(leaderAddr); err != nil {
    return fmt.Errorf("leader %s unhealthy before import: %w", leaderAddr, err)
}

if err != nil && strings.Contains(err.Error(), "failed to send forward request") {
    // stream died at handshake; check leader liveness, then retry with backoff
    return retryImport(ctx, groupID)
}

When it happens

Trigger: alphaStream.Send(forwardReq) returns err because the connection to the leader dropped, the leader cancelled the stream's context, flow-control backpressure timed out, or the server closed the stream immediately (e.g. it errored on its own startup).

Common situations: Leader crashed between RPC setup and first Send; network interruption mid-import startup; server-side import rejected the request and closed the stream before reading; gRPC max message/frame limits misconfigured.

Related errors

• failed to establish stream with leader: %v
• recv final downstream(%d): %w
• ack data downstream(%d): %w
• send ack upstream: %w
• failed to connect to endpoint [%s]: %w

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/60b92750c6299700. Report an issue: GitHub.