dgraph-io/dgraph · error

ack data downstream(%d): %w

Error message

ack data downstream(%d): %w

What it means

During external snapshot streaming, pipeTwoStream relays each data chunk from the upstream (proxy) stream to the downstream group-leader stream. After sending a chunk downstream it waits for the downstream ack via out.Recv(); if that receive fails, the chunk is not acknowledged downstream and the pipe aborts with this wrapped error.

Source

Thrown at worker/import.go:447

				if err != nil {
					return fmt.Errorf("recv final downstream(%d): %w", groupId, err)
				}
				if err := in.Send(resp); err != nil {
					return fmt.Errorf("relay final upstream: %w", err)
				}
				if resp.Finish {
					glog.Infof("[import] [forward %d -> %d] finish", currentGroup, groupId)
					return nil
				}
			}
		}

		// Normal data chunk: send -> wait ack -> send upstream ack.
		if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil {
			return fmt.Errorf("send data downstream(%d): %w", groupId, err)
		}
		if _, err := out.Recv(); err != nil {
			return fmt.Errorf("ack data downstream(%d): %w", groupId, err)
		}
		if err := in.Send(&api.StreamExtSnapshotResponse{}); err != nil {
			return fmt.Errorf("send ack upstream: %w", err)
		}

	}
}

func (w *grpcWorker) UpdateExtSnapshotStreamingState(ctx context.Context,
	req *api.UpdateExtSnapshotStreamingStateRequest) (*pb.Status, error) {
	if req == nil {
		return nil, errors.New("UpdateExtSnapshotStreamingStateRequest must not be nil")
	}

	if req.Start && req.Finish {
		return nil, errors.New("UpdateExtSnapshotStreamingStateRequest cannot have both Start and Finish set to true")
	}

View on GitHub (pinned to 759e242be6)

Solutions

1. Check connectivity and health of the downstream alpha that should ack the stream, then retry the snapshot import.
2. Increase gRPC keepalive/deadline settings so long idles between chunks are not terminated.
3. Ensure the downstream group leader is up and serving the target group (check membership state).
4. Retry the whole external snapshot streaming operation; the pipe aborts the stream, so a partial retry is not possible.

Example fix

// before (no resilience around the stream)
client.StreamExtSnapshot(ctx)
// after (bound the stream and retry on failure)
for attempt := 0; attempt < 3; attempt++ {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
    err := streamSnapshot(ctx)
    cancel()
    if err == nil { break }
    time.Sleep(5 * time.Second)
}

// before starting the import, check the downstream alpha is reachable
conn, err := grpc.DialContext(ctx, addr, grpc.WithBlock(), grpc.WithTimeout(10*time.Second))
if err != nil { return fmt.Errorf("downstream %s unreachable: %w", addr, err) }

err := streamSnapshot(ctx)
var retriable = isTransportErr(err) // match 'ack data downstream' wrapper
if retriable { backoff-and-retry whole snapshot import }

When it happens

Trigger: The downstream gRPC StreamExtSnapshot stream breaks (peer alpha restarted, network drop, stream deadline, context cancellation) between out.Send of a data packet and its ack Recv, or the downstream peer returns a non-EOF error on its ack response.

Common situations: Follower/leader alpha restarted mid-import; network partition or load balancer idle timeout between alphas; import run canceled or context deadline exceeded; downstream node crashed while applying the snapshot.

Related errors

• failed to establish stream with leader: %v
• failed to send forward request: %v
• recv final downstream(%d): %w
• send ack upstream: %w
• failed to run in group streaming: %v

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/897a8d1cb1be65e7. Report an issue: GitHub.