dgraph-io/dgraph · critical

failed to turn off drain mode: %v

Error message

failed to turn off drain mode: %v

What it means

After a streaming failure, streamSnapshot attempts to exit drain mode destructively (Finish=true, DropData=true) to restore the cluster to a clean slate. If that recovery call to UpdateExtSnapshotStreamingState also fails, the error is wrapped as 'failed to turn off drain mode'. The cluster may remain in drain mode, rejecting traffic.

Source

Thrown at dgraph/cmd/dgraphimport/import_client.go:143

				return err
			}

			return nil
		})
	}

	if err := errG.Wait(); err != nil {
		glog.Errorf("[import] failed to stream external snapshot: %v", err)
		// If errors occurs during streaming of the external snapshot, we drop all the data and
		// go back to ensure a clean slate and the cluster remains in working state.
		glog.Info("[import] dropping all the data and going back to clean slate")
		req := &api.UpdateExtSnapshotStreamingStateRequest{
			Start:    false,
			Finish:   true,
			DropData: true,
		}
		if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
			return fmt.Errorf("failed to turn off drain mode: %v", err)
		}

		glog.Info("[import] successfully disabled drain mode")
		return err
	}

	glog.Info("[import] Completed streaming external snapshot")
	req := &api.UpdateExtSnapshotStreamingStateRequest{
		Start:    false,
		Finish:   true,
		DropData: false,
	}
	if _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {
		glog.Errorf("[import] failed to disable drain mode: %v", err)
		return fmt.Errorf("failed to disable drain mode: %v", err)
	}
	glog.Info("[import] successfully disable drain mode")
	return nil

View on GitHub (pinned to 759e242be6)

Solutions

1. Inspect the wrapped cause (%v); fix connectivity/leader issues first, then re-issue UpdateExtSnapshotStreamingState{Start:false, Finish:true, DropData:true} manually via dgo to exit drain mode.
2. Verify cluster health (Zero leaders, Alpha state) and wait for quorum before retrying the recovery call.
3. Check ACL credentials if the error indicates permission denied.
4. Re-run the import from a clean state once drain mode is confirmed off (cluster state / metrics).

Example fix

// before
// import failed; drain mode still ON, recovery RPC failed silently in logs
// after
req := &api.UpdateExtSnapshotStreamingStateRequest{Start: false, Finish: true, DropData: true}
_, err := dg.UpdateExtSnapshotStreamingState(ctx, req) // manual recovery after restoring connectivity

resp, err := http.Get("http://zero1:6080/state")
if err != nil || resp.StatusCode != http.StatusOK {
    return fmt.Errorf("zero not reachable; fix cluster before import/cleanup")
}

if err := dgraphimport.Import(ctx, addr, outDir); err != nil {
    if strings.Contains(err.Error(), "failed to turn off drain mode") {
        // cluster may be stuck in drain mode: restore connectivity and
        // re-issue Finish=true, DropData=true via dgo before retrying
        glog.Errorf("CRITICAL: cluster possibly still in drain mode: %v", err)
    }
    return err
}

When it happens

Trigger: errG.Wait() returned a streaming error AND the subsequent destructive UpdateExtSnapshotStreamingState call fails — Alpha unreachable at that moment, leader change mid-recovery, context canceled, or ACL/permission rejection.

Common situations: Network partition or Alpha restart during a failed import, Zero leadership churn while wiping data, import run canceled (ctx done) leaving drain mode set, then recovery also times out.

Related errors

• failed to initiate external snapshot stream: %v
• unable to connect to the leader of group [%v] : %v
• connection string cannot be empty
• failed to connect to endpoint [%s]: %w
• failed to disable drain mode: %v

AI-assisted analysis of dgraph-io/dgraph@759e242be6 (2026-09-01). Data as JSON: /api/errors/f551e596ec6bd038. Report an issue: GitHub.