{"record":{"id":"317aec53b76d3e49","repo":"vitessio/vitess","slug":"shutdown-didn-t-kill-process-v","errorCode":null,"errorMessage":"Shutdown didn't kill process %v","messagePattern":"Shutdown didn't kill process (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"go/vt/zkctl/zkctl.go","lineNumber":153,"sourceCode":"\tif err != nil {\n\t\treturn err\n\t}\n\tpid, err := strconv.Atoi(string(bytes.TrimSpace(pidData)))\n\tif err != nil {\n\t\treturn err\n\t}\n\terr = syscallutil.Kill(pid, syscall.SIGKILL)\n\tif err != nil && err != syscall.ESRCH {\n\t\treturn err\n\t}\n\ttimeout := time.Now().Add(shutdownWaitTime)\n\tfor time.Now().Before(timeout) {\n\t\tif syscallutil.Kill(pid, syscall.SIGKILL) == syscall.ESRCH {\n\t\t\treturn nil\n\t\t}\n\t\ttime.Sleep(time.Second)\n\t}\n\treturn fmt.Errorf(\"Shutdown didn't kill process %v\", pid)\n}\n\nfunc (zkd *Zkd) makeCfg() (string, error) {\n\troot, err := env.VtRoot()\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\tcnfTemplatePaths := []string{path.Join(root, \"config/zkcfg/zoo.cfg\")}\n\treturn MakeZooCfg(cnfTemplatePaths, zkd.config, \"# generated by vt\")\n}\n\n// Init generates a new config and then starts ZooKeeper.\nfunc (zkd *Zkd) Init() error {\n\tif zkd.Inited() {\n\t\treturn errors.New(\"zk already inited\")\n\t}\n\n\tlog.Info(\"zkd.Init\")","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/vitessio/vitess/blob/01a25a7d176f94613b8d59d799f438380a8760e4/go/vt/zkctl/zkctl.go#L135-L171","documentation":"Zkd.Shutdown sends SIGTERM and waits for the pid to die; if after the timeout window a SIGKILL check still finds the process alive (Kill does not return ESRCH), this error is returned. It means zookeeper refused to die within the allotted time.","triggerScenarios":"Calling zkd.Shutdown() (via zkctl Teardown) when the zk process ignores/kills slowly — kill signal delivered but process still present past the timeout deadline.","commonSituations":"Zookeeper stuck in uninterruptible I/O on a slow/hung disk; process in a stopped (SIGSTOP) state; very slow container under resource pressure; wrong pid recorded in the pid file.","solutions":["Kill the pid manually: `kill -9 <pid>` and remove the stale pid file","Check process state with `ps -o stat -p <pid>` (D/S/T states explain the hang)","If disk I/O is stuck, fix the underlying storage or restart the host"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if syscallutil.Kill(pid, 0) == nil {\n    log.Infof(\"zk pid %d still running before shutdown attempt\", pid)\n}","typeGuard":null,"tryCatchPattern":"if err := zkd.Shutdown(); err != nil {\n    if strings.Contains(err.Error(), \"didn't kill process\") {\n        syscall.Kill(pid, syscall.SIGKILL) // escalate\n    }\n}","preventionTips":["Avoid stopping the zk process with SIGSTOP externally","Ensure disk I/O to the zk data dir is healthy","Monitor for hung processes and escalate TERM→KILL with a bounded wait"],"tags":["zookeeper","shutdown","process","timeout"],"backgroundTag":"process-kill-timeout","analyzedSha":"01a25a7d176f94613b8d59d799f438380a8760e4","analyzedAt":"2026-09-01T17:28:30.605Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}