dgraph-io/dgraph · error

send ack upstream: %w

Error message

send ack upstream: %w

What it means

After a data chunk is acked by the downstream peer, pipeTwoStream sends the ack back upstream on the proxy's stream (in.Send). If that send fails, the upstream caller never receives the ack and the relay aborts with this wrapped error.

Source

Thrown at worker/import.go:450

				if err := in.Send(resp); err != nil {
					return fmt.Errorf("relay final upstream: %w", err)
				}
				if resp.Finish {
					glog.Infof("[import] [forward %d -> %d] finish", currentGroup, groupId)
					return nil
				}
			}
		}

		// Normal data chunk: send -> wait ack -> send upstream ack.
		if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: req.Pkt}); err != nil {
			return fmt.Errorf("send data downstream(%d): %w", groupId, err)
		}
		if _, err := out.Recv(); err != nil {
			return fmt.Errorf("ack data downstream(%d): %w", groupId, err)
		}
		if err := in.Send(&api.StreamExtSnapshotResponse{}); err != nil {
			return fmt.Errorf("send ack upstream: %w", err)
		}

	}
}

func (w *grpcWorker) UpdateExtSnapshotStreamingState(ctx context.Context,
	req *api.UpdateExtSnapshotStreamingStateRequest) (*pb.Status, error) {
	if req == nil {
		return nil, errors.New("UpdateExtSnapshotStreamingStateRequest must not be nil")
	}

	if req.Start && req.Finish {
		return nil, errors.New("UpdateExtSnapshotStreamingStateRequest cannot have both Start and Finish set to true")
	}

	glog.Infof("[import] Applying import mode proposal: %+v", req)
	err := groups().Node.proposeAndWait(ctx, &pb.Proposal{ExtSnapshotState: req})

View on GitHub (pinned to 759e242be6)

Solutions

1. Check the client that opened the stream: it likely disconnected or timed out; increase its deadline and retry.
2. Verify network stability between this alpha and the upstream caller.
3. Retry the snapshot streaming operation end-to-end after fixing the connection.
4. Make sure no middleware/proxy (LB) kills long-lived gRPC streams; enable keepalives.

Example fix

// before: client ctx with short timeout
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
// after: generous timeout for bulk streaming
ctx, cancel := context.WithTimeout(ctx, 2*time.Hour)

if err := runImport(ctx); err != nil {
    if strings.Contains(err.Error(), "send ack upstream") || status.Code(errors.Unwrap(err)) == codes.Unavailable {
        // client disconnected: fix client and retry
    }
}

When it happens

Trigger: The upstream gRPC stream from the calling alpha has been closed, reset, or its context canceled while pipeTwoStream tries to send the StreamExtSnapshotResponse ack — e.g. client disconnected, deadline exceeded, or transport broken.

Common situations: Proxy alpha (or the dgraph live/backup client driving it) dropped its connection mid-import; client-side timeout shorter than streaming duration; client canceled the import.

Related errors

• failed to establish stream with leader: %v
• failed to send forward request: %v
• recv final downstream(%d): %w
• ack data downstream(%d): %w
• failed to connect to endpoint [%s]: %w

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/64593ee2732ea2d4. Report an issue: GitHub.