{"record":{"id":"f551e596ec6bd038","repo":"dgraph-io/dgraph","slug":"failed-to-turn-off-drain-mode-v","errorCode":null,"errorMessage":"failed to turn off drain mode: %v","messagePattern":"failed to turn off drain mode: (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"dgraph/cmd/dgraphimport/import_client.go","lineNumber":143,"sourceCode":"\t\t\t\treturn err\n\t\t\t}\n\n\t\t\treturn nil\n\t\t})\n\t}\n\n\tif err := errG.Wait(); err != nil {\n\t\tglog.Errorf(\"[import] failed to stream external snapshot: %v\", err)\n\t\t// If errors occurs during streaming of the external snapshot, we drop all the data and\n\t\t// go back to ensure a clean slate and the cluster remains in working state.\n\t\tglog.Info(\"[import] dropping all the data and going back to clean slate\")\n\t\treq := &api.UpdateExtSnapshotStreamingStateRequest{\n\t\t\tStart:    false,\n\t\t\tFinish:   true,\n\t\t\tDropData: true,\n\t\t}\n\t\tif _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to turn off drain mode: %v\", err)\n\t\t}\n\n\t\tglog.Info(\"[import] successfully disabled drain mode\")\n\t\treturn err\n\t}\n\n\tglog.Info(\"[import] Completed streaming external snapshot\")\n\treq := &api.UpdateExtSnapshotStreamingStateRequest{\n\t\tStart:    false,\n\t\tFinish:   true,\n\t\tDropData: false,\n\t}\n\tif _, err := dc.UpdateExtSnapshotStreamingState(ctx, req); err != nil {\n\t\tglog.Errorf(\"[import] failed to disable drain mode: %v\", err)\n\t\treturn fmt.Errorf(\"failed to disable drain mode: %v\", err)\n\t}\n\tglog.Info(\"[import] successfully disable drain mode\")\n\treturn nil","sourceCodeStart":125,"sourceCodeEnd":161,"githubUrl":"https://github.com/dgraph-io/dgraph/blob/759e242be62c91f8d084da06ad0c8d21256d9c07/dgraph/cmd/dgraphimport/import_client.go#L125-L161","documentation":"After a streaming failure, streamSnapshot attempts to exit drain mode destructively (Finish=true, DropData=true) to restore the cluster to a clean slate. If that recovery call to UpdateExtSnapshotStreamingState also fails, the error is wrapped as 'failed to turn off drain mode'. The cluster may remain in drain mode, rejecting traffic.","triggerScenarios":"errG.Wait() returned a streaming error AND the subsequent destructive UpdateExtSnapshotStreamingState call fails — Alpha unreachable at that moment, leader change mid-recovery, context canceled, or ACL/permission rejection.","commonSituations":"Network partition or Alpha restart during a failed import, Zero leadership churn while wiping data, import run canceled (ctx done) leaving drain mode set, then recovery also times out.","solutions":["Inspect the wrapped cause (%v); fix connectivity/leader issues first, then re-issue UpdateExtSnapshotStreamingState{Start:false, Finish:true, DropData:true} manually via dgo to exit drain mode.","Verify cluster health (Zero leaders, Alpha state) and wait for quorum before retrying the recovery call.","Check ACL credentials if the error indicates permission denied.","Re-run the import from a clean state once drain mode is confirmed off (cluster state / metrics)."],"exampleFix":"// before\n// import failed; drain mode still ON, recovery RPC failed silently in logs\n// after\nreq := &api.UpdateExtSnapshotStreamingStateRequest{Start: false, Finish: true, DropData: true}\n_, err := dg.UpdateExtSnapshotStreamingState(ctx, req) // manual recovery after restoring connectivity","handlingStrategy":"try-catch","validationCode":"resp, err := http.Get(\"http://zero1:6080/state\")\nif err != nil || resp.StatusCode != http.StatusOK {\n    return fmt.Errorf(\"zero not reachable; fix cluster before import/cleanup\")\n}","typeGuard":null,"tryCatchPattern":"if err := dgraphimport.Import(ctx, addr, outDir); err != nil {\n    if strings.Contains(err.Error(), \"failed to turn off drain mode\") {\n        // cluster may be stuck in drain mode: restore connectivity and\n        // re-issue Finish=true, DropData=true via dgo before retrying\n        glog.Errorf(\"CRITICAL: cluster possibly still in drain mode: %v\", err)\n    }\n    return err\n}","preventionTips":["Keep a stable network path to Alphas for the whole import; avoid node restarts mid-stream.","Use a context that isn't canceled mid-import (no short CLI timeouts).","After any failed import, always confirm drain mode is off before serving traffic.","Maintain valid ACL credentials through the entire operation including cleanup."],"tags":["grpc","raft","drain-mode","recovery","import"],"backgroundTag":"drain-mode-exit-failed","analyzedSha":"759e242be62c91f8d084da06ad0c8d21256d9c07","analyzedAt":"2026-09-01T14:42:12.034Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}