What causes "failed to send data chunk: %w" in dgraph-io/dgraph?

Check the wrapped gRPC status for Unavailable/Canceled to confirm a transport break.

failed to send data chunk: %w

Error message

failed to send data chunk: %w

What it means

Produced inside the badger stream.Send callback when out.Send of a StreamPacket fails with a non-EOF error. Each packet must be sent and ACKed; a send failure aborts Orchestrate with this wrapped error. EOF is deliberately tolerated because the server may close its send side.

Source

Thrown at dgraph/cmd/dgraphimport/import_client.go:222

	// Configure and start the BadgerDB stream
	glog.Infof("[import] Starting BadgerDB stream for group [%v]", groupId)
	if err := streamBadger(ctx, ps, out, groupId); err != nil {
		return fmt.Errorf("badger streaming failed for group [%v]: %v", groupId, err)
	}
	return nil
}

// streamBadger runs a BadgerDB stream to send key-value pairs to the specified group.
// It creates a new stream at the maximum sequence number and sends the data to the specified group.
// It also sends a final 'done' signal to mark completion.
func streamBadger(ctx context.Context, ps *badger.DB, out api.Dgraph_StreamExtSnapshotClient, groupId uint32) error {
	stream := ps.NewStreamAt(math.MaxUint64)
	stream.LogPrefix = "[import] Sending external snapshot to group [" + fmt.Sprintf("%d", groupId) + "]"
	stream.KeyToList = nil
	stream.Send = func(buf *z.Buffer) error {
		p := &api.StreamPacket{Data: buf.Bytes()}
		if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: p}); err != nil && !errors.Is(err, io.EOF) {
			return fmt.Errorf("failed to send data chunk: %w", err)
		}
		if _, err := out.Recv(); err != nil {
			return fmt.Errorf("failed to receive response for group ID [%v] from the server: %w", groupId, err)
		}
		glog.Infof("[import] Group [%v]: Received ACK for sending data chunk", groupId)

		return nil
	}

	// Execute the stream process
	if err := stream.Orchestrate(ctx); err != nil {
		return fmt.Errorf("stream orchestration failed for group [%v]: %w, badger path: %s", groupId, err, ps.Opts().Dir)
	}

	// Send the final 'done' signal to mark completion
	glog.Infof("[import] Sending completion signal for group [%d]", groupId)
	done := &api.StreamPacket{Done: true}

View on GitHub (pinned to 759e242be6)

Solutions

Check the wrapped gRPC status for Unavailable/Canceled to confirm a transport break.
Verify server health (pod restarts, OOM kills) and network path stability.
Raise keepalive/timeout settings on client and LB so long streams survive.
Re-run the import; streams cannot resume mid-transfer.
Reduce packet pressure only if server logs show backpressure/OOM at that time.

Example fix

// before
if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: p}); err != nil && !errors.Is(err, io.EOF) {
	return fmt.Errorf("failed to send data chunk: %w", err)
}
// after
if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: p}); err != nil && !errors.Is(err, io.EOF) {
	if s, ok := status.FromError(err); ok && s.Code() == codes.Unavailable {
		return fmt.Errorf("connection lost while sending chunk for group %d: %w", groupId, err)
	}
	return fmt.Errorf("failed to send data chunk: %w", err)
}

Defensive patterns

Strategy: retry

Validate before calling

if ctx.Err() != nil {
	return fmt.Errorf("cancelled before streaming; fix deadline or cancellation: %w", ctx.Err())
}

Type guard

func isTransportBreak(err error) bool {
	s, ok := status.FromError(err)
	return ok && (s.Code() == codes.Unavailable || s.Code() == codes.Canceled || s.Code() == codes.Internal)
}

Try / catch

if err := out.Send(&api.StreamExtSnapshotRequest{Pkt: p}); err != nil && !errors.Is(err, io.EOF) {
	if isTransportBreak(err) {
		return errRetryWholeStream // stream not resumable; caller restarts group
	}
	return fmt.Errorf("failed to send data chunk: %w", err)
}

Prevention

Enable gRPC keepalive on both ends of the connection.
Bypass or reconfigure LB idle timeouts for the import stream.
Pin imports to a stable Alpha (avoid pod eviction windows).
Test with a small pdir first to validate network stability.

When it happens

Trigger: gRPC stream broken mid-transfer (connection reset, server crash, LB timeout); context cancelled while blocking on Send; server closed the stream after an earlier error so subsequent Sends fail.

Common situations: Multi-GB snapshot transfer interrupted by network hiccup; k8s service/ingress killing long-lived streams; Alpha pod evicted mid-import.

Related errors

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/ac37118ae7a34e09. Report an issue: GitHub.