{"record":{"id":"8dc6d268a33a9fe7","repo":"vitessio/vitess","slug":"mysqld-shutdown-hook-failed-v","errorCode":null,"errorMessage":"mysqld_shutdown hook failed: %v","messagePattern":"mysqld_shutdown hook failed: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/mysqlctl/mysqld.go","lineNumber":1177,"sourceCode":"\t\t\t// The error is only suppressed when the wait below will run:\n\t\t\t// for waitForMysqld=false callers a nil return would claim a\n\t\t\t// shutdown nothing verified, so they get the error as before.\n\t\t\tif !mysqladminAbortedWaiting(output) {\n\t\t\t\t// The SHUTDOWN command was never delivered.\n\t\t\t\treturn false, err\n\t\t\t}\n\t\t\tif !waitForMysqld {\n\t\t\t\t// Delivered, but nothing below will verify the outcome.\n\t\t\t\treturn true, err\n\t\t\t}\n\t\t\tlog.Warn(\"mysqladmin gave up waiting for mysqld to stop, waiting on pid/socket files instead\", slog.Any(\"error\", err))\n\t\t\tvar cancel context.CancelFunc\n\t\t\tctx, cancel = boundShutdownWaitContext(ctx)\n\t\t\tdefer cancel()\n\t\t}\n\tdefault:\n\t\t// hook failed, we report error\n\t\treturn false, fmt.Errorf(\"mysqld_shutdown hook failed: %v\", hr.String())\n\t}\n\n\t// Wait for mysqld to really stop. Use the socket and pid files as a\n\t// proxy for that since we can't call wait() in a process we\n\t// didn't start.\n\tif waitForMysqld {\n\t\tlog.Info(fmt.Sprintf(\"Mysqld.Shutdown: waiting for socket file (%v) and pid file (%v) to disappear\", cnf.SocketFile, cnf.PidFile))\n\t\tif err := waitForMysqldExit(ctx, cnf.SocketFile, cnf.PidFile); err != nil {\n\t\t\treturn true, err\n\t\t}\n\t}\n\treturn true, nil\n}\n\n// StartAfterExit waits for a mysqld process that shut itself down (e.g. after a\n// CLONE operation) to fully exit, then starts a new one. It polls for the\n// disappearance of the socket and pid files before calling Start.\nfunc (mysqld *Mysqld) StartAfterExit(ctx context.Context, cnf *Mycnf) error {","sourceCodeStart":1159,"sourceCodeEnd":1195,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/mysqlctl/mysqld.go#L1159-L1195","documentation":"During executeShutdown, Vitess runs the optional 'mysqld_shutdown' hook to customize how mysqld is stopped. If the hook returns any status other than success or does-not-exist, shutdown is aborted immediately and this error is returned. Unlike the preflight hook, this one fires mid-shutdown, so mysqld may already be in the middle of stopping when it fails.","triggerScenarios":"Mysqld.executeShutdown (via shutdownWithReplicaCrashSafety) invokes the mysqld_shutdown hook; the hook script exits with a non-success status (other than HOOK_SUCCESS / HOOK_DOES_NOT_EXIST).","commonSituations":"A custom shutdown script (e.g. using mysqladmin shutdown with special flags) fails because of wrong credentials or mysqld already gone; the hook times out; the script's environment (PATH, config) differs from what it expects under Vitess.","solutions":["Read the hook's exit status and output embedded in the error (hr.String()) and fix the custom shutdown script accordingly.","Test the hook script manually on the host with the same user and environment Vitess runs under.","Ensure the script handles the already-stopped-mysqld case gracefully (idempotent exit 0).","Remove the hook if it is no longer needed so the default shutdown path is used."],"exampleFix":"// before\nmysqladmin -p wrongpass shutdown || exit 1\n// after\nmysqladmin shutdown 2>/dev/null || exit 0  # already stopped is OK","handlingStrategy":"try-catch","validationCode":"hr := hook.NewHook(\"mysqld_shutdown\").ExecuteContext(ctx)\nif hr.ExitStatus != hook.HOOK_SUCCESS && hr.ExitStatus != hook.HOOK_DOES_NOT_EXIST {\n    return fmt.Errorf(\"shutdown hook not ready: %s\", hr.String())\n}","typeGuard":"func shutdownHookOK(hr *hook.HookResult) bool {\n    return hr.ExitStatus == hook.HOOK_SUCCESS || hr.ExitStatus == hook.HOOK_DOES_NOT_EXIST\n}","tryCatchPattern":"err := mysqld.Shutdown(ctx, cnf, true, timeout)\nif err != nil && strings.Contains(err.Error(), \"mysqld_shutdown hook failed\") {\n    // parse hr from message; if mysqld is already partially stopping, poll socket file removal\n}","preventionTips":["Make shutdown hook scripts idempotent (succeed when mysqld is already stopped).","Test hooks under the exact user/environment Vitess runs with.","Avoid long-running logic in shutdown hooks; keep them under the shutdown timeout.","Remove obsolete hooks from the hook directory rather than leaving broken scripts."],"tags":["mysql","shutdown","hooks","mysqld","extension-point"],"backgroundTag":"hook-failed","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}