dgraph-io/dgraph · error

failed to disable drain mode: %v

Error message

failed to disable drain mode: %v

What it means

This error is returned by streamSnapshot when the final UpdateExtSnapshotStreamingState call (Start:false, Finish:true) fails to turn off the server's drain mode after an external snapshot import. It wraps the underlying gRPC/server error with fmt.Errorf, and the import client logs it via glog before returning. The server may be left in drain mode, so follow-up traffic could still be affected.

Source

Thrown at dgraph/cmd/dgraphimport/import_client.go:158

			DropData: true,
		}
		if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
			return fmt.Errorf("failed to turn off drain mode: %v", err)
		}

		glog.Info("[import] successfully disabled drain mode")
		return err
	}

	glog.Info("[import] Completed streaming external snapshot")
	req := &api.UpdateExtSnapshotStreamingStateRequest{
		Start:    false,
		Finish:   true,
		DropData: false,
	}
	if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
		glog.Errorf("[import] failed to disable drain mode: %v", err)
		return fmt.Errorf("failed to disable drain mode: %v", err)
	}
	glog.Info("[import] successfully disable drain mode")
	return nil
}

// streamSnapshotForGroup handles the actual data streaming process for a single group.
// It opens the BadgerDB at the specified directory and streams all data to the server.
func streamSnapshotForGroup(ctx context.Context, dc api.DgraphClient, pdir string, groupId uint32) error {
	glog.Infof("Opening stream for group %d from directory %s", groupId, pdir)

	// Initialize stream with the server
	out, err := dc.StreamExtSnapshot(ctx)
	if err != nil {
		return fmt.Errorf("failed to start external snapshot stream for group %d: %w", groupId, err)
	}
	defer func() {
		_ = out.CloseSend()
	}()

View on GitHub (pinned to 759e242be6)

Solutions

1. Check the wrapped cause (%v) to distinguish connectivity failures from server-side rejections.
2. Verify the target Alpha(s) are healthy and reachable, then re-run the import step; disable-drain is idempotent to retry.
3. Ensure the gRPC context/deadline allows enough time for the finish call on large imports.
4. Confirm server version supports the UpdateExtSnapshotStreamingState API (mirrors client version).
5. If the server remains in drain mode, call the snapshot streaming state API again with Start:false/Finish:true or restart the Alpha.

Example fix

// before
ctx := context.Background()
if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
	return fmt.Errorf("failed to disable drain mode: %v", err)
}
// after
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
	if st, ok := status.FromError(err); ok && st.Code() == codes.Unavailable {
		// retry once after reconnect
		if _, rerr := dc.UpdateExtSnapshotStreamingState(ctx, req); rerr != nil {
			return fmt.Errorf("failed to disable drain mode: %v", rerr)
		}
	} else {
		return fmt.Errorf("failed to disable drain mode: %v", err)
	}
}

if err := dcHealthCheck(dc); err != nil {
	return fmt.Errorf("alpha unreachable, fix before import: %w", err)
}

for attempt := 0; attempt < 3; attempt++ {
	_, err := dc.UpdateExtSnapshotStreamingState(ctx, req)
	if err == nil {
		break
	}
	if s, ok := status.FromError(err); ok && s.Code() == codes.Unavailable {
		time.Sleep(time.Duration(1<<attempt) * time.Second)
		continue
	}
	return err
}

When it happens

Trigger: dc.UpdateExtSnapshotStreamingState(ctx, req) with Finish:true returns an error: the Alpha serving the request is unreachable, the RPC times out or the context is cancelled, or the server rejects the state update (e.g. snapshot streaming session already torn down, or server-side error while exiting drain mode).

Common situations: Network blip or Alpha restart right at the end of a long snapshot stream; context deadline exceeded on slow imports; calling the import against a server version that does not implement UpdateExtSnapshotStreamingState; server already removed the streaming session because the stream failed earlier.

Related errors

• failed to start external snapshot stream for group %d: %w
• failed to send request for group ID [%v] to the server: %w
• failed to receive response for group ID [%v] from the server
• failed to send data chunk: %w
• failed to send 'done' signal for group [%d]: %w

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/5d5eb2b639e5c98c. Report an issue: GitHub.