{"record":{"id":"ada484f6d7749420","repo":"vitessio/vitess","slug":"gave-up-waiting-for-mysqld-to-stop","errorCode":null,"errorMessage":"gave up waiting for mysqld to stop","messagePattern":"gave up waiting for mysqld to stop","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/mysqlctl/mysqld.go","lineNumber":1241,"sourceCode":"// message, the match returns false and Shutdown reports the mysqladmin\n// error exactly as it did before this check existed. The endtoend mysqlctl\n// suite exercises this match against the real mysqladmin binary.\nfunc mysqladminAbortedWaiting(output string) bool {\n\treturn strings.Contains(output, \"Aborted waiting on pid file\")\n}\n\n// waitForMysqldExit polls until both socketFile and pidFile have been removed,\n// which signals that the mysqld process has fully exited.\nfunc waitForMysqldExit(ctx context.Context, socketFile, pidFile string) error {\n\tfor {\n\t\t_, socketErr := os.Stat(socketFile)\n\t\t_, pidErr := os.Stat(pidFile)\n\t\tif os.IsNotExist(socketErr) && os.IsNotExist(pidErr) {\n\t\t\treturn nil\n\t\t}\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn errors.New(\"gave up waiting for mysqld to stop\")\n\t\tcase <-time.After(mysqldExitPollInterval):\n\t\t}\n\t}\n}\n\n// execCmd searches the PATH for a command and runs it, logging the output.\n// If input is not nil, pipe it to the command's stdin. It runs without a\n// deadline; use execCmdWithContext to bound the command by a context.\nfunc execCmd(name string, args, env []string, dir string, input io.Reader) (cmd *exec.Cmd, output string, err error) {\n\treturn execCmdWithContext(context.Background(), name, args, env, dir, input)\n}\n\n// execCmdWithContext searches the PATH for a command and runs it, logging the\n// output. If input is not nil, pipe it to the command's stdin. If ctx is\n// cancelled or its deadline passes, the command is killed and the call returns\n// promptly rather than blocking on a stalled process.\nfunc execCmdWithContext(ctx context.Context, name string, args, env []string, dir string, input io.Reader) (cmd *exec.Cmd, output string, err error) {\n\tcmdPath, _ := exec.LookPath(name)","sourceCodeStart":1223,"sourceCodeEnd":1259,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/mysqlctl/mysqld.go#L1223-L1259","documentation":"waitForMysqldExit polls the socket and pid files while shutting mysqld down; shutdown succeeds when both disappear. If the context is cancelled or its deadline expires while mysqld is still running (files still present), this error reports that mysqld refused to exit in time. It signals a hung or wedged shutdown rather than a failed command invocation.","triggerScenarios":"Calling Shutdown (via executeShutdown) or StartAfterExit with a context that expires before the mysqld process writes no more and removes its socket/pid files; mysqld stuck in a long rollback or waiting on a slow shutdown.","commonSituations":"Large InnoDB buffers making a clean shutdown slow; mysqld ignoring SIGTERM and requiring a longer grace period; shutdown context deadlines set too aggressively in tests or CI.","solutions":["Increase the shutdown context deadline to accommodate slow mysqld shutdowns","Check mysqld's error log to see what is blocking shutdown (long transactions, slow flush)","Use a forced/immediate shutdown mode if clean shutdown is not required","Ensure no client connections or long-running queries keep mysqld alive during shutdown"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(ctx, 3*time.Second)\nerr := mysqld.Shutdown(ctx, cnf, true) // gave up waiting\n// after\nctx, cancel := context.WithTimeout(ctx, 120*time.Second)\nerr := mysqld.Shutdown(ctx, cnf, true)","handlingStrategy":"retry","validationCode":"// before shutdown, ensure no blockers\nif pid, err := os.ReadFile(mycnf.PidFile); err == nil {\n    log.Info(\"shutting down mysqld\", slog.String(\"pid\", strings.TrimSpace(string(pid))))\n}","typeGuard":null,"tryCatchPattern":"ctx, cancel := context.WithTimeout(ctx, 120*time.Second)\ndefer cancel()\nif err := mysqld.Shutdown(ctx, mycnf, false); err != nil {\n    if strings.Contains(err.Error(), \"gave up waiting for mysqld to stop\") {\n        log.Error(\"mysqld shutdown timeout; check error log and consider forced shutdown\", slog.Any(\"error\", err))\n    }\n    return err\n}","preventionTips":["Allow ample time for shutdown on hosts with large buffer pools","Kill long-running queries/connections before shutdown","Use forced shutdown as a fallback when a clean stop times out","Monitor the pid/socket files to confirm the process actually exits"],"tags":["go","mysqlctl","mysqld-shutdown","timeout"],"backgroundTag":"mysqld-shutdown-timeout","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}