{"record":{"id":"2dfd98047c2baac4","repo":"hyperledger/fabric","slug":"failed-to-stop-instance-s","errorCode":null,"errorMessage":"failed to stop instance '%s'","messagePattern":"failed to stop instance '(.+?)'","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"core/container/externalbuilder/instance.go","lineNumber":176,"sourceCode":"func (i *Instance) Stop() error {\n\tif i.Session == nil {\n\t\treturn errors.Errorf(\"instance has not been started\")\n\t}\n\n\tdone := make(chan struct{})\n\tgo func() { i.Wait(); close(done) }()\n\n\ti.Session.Signal(syscall.SIGTERM)\n\tselect {\n\tcase <-time.After(i.TermTimeout):\n\t\ti.Session.Signal(syscall.SIGKILL)\n\tcase <-done:\n\t\treturn nil\n\t}\n\n\tselect {\n\tcase <-time.After(5 * time.Second):\n\t\treturn errors.Errorf(\"failed to stop instance '%s'\", i.PackageID)\n\tcase <-done:\n\t\treturn nil\n\t}\n}\n\nfunc (i *Instance) Wait() (int, error) {\n\tif i.Session == nil {\n\t\treturn -1, errors.Errorf(\"instance was not successfully started\")\n\t}\n\n\terr := i.Session.Wait()\n\terr = errors.Wrapf(err, \"builder '%s' run failed\", i.Builder.Name)\n\tif exitErr, ok := errors.Cause(err).(*exec.ExitError); ok {\n\t\treturn exitErr.ExitCode(), err\n\t}\n\treturn 0, err\n}\n","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/hyperledger/fabric/blob/2736b63f8fd5932511d56fe68b7039d15977f7f6/core/container/externalbuilder/instance.go#L158-L194","documentation":"After sending SIGTERM, Stop waits 5 seconds for the process to exit; if the Wait goroutine has not completed within that grace period it gives up and returns this error (the process is effectively left to SIGKILL semantics/timed out). It identifies the stuck instance by PackageID.","triggerScenarios":"A builder process that ignores or is blocked on SIGTERM so it doesn't exit within the hard-coded 5s window after Stop() signals it; a hung builder blocked on I/O, a child process, or a stuck network call.","commonSituations":"Builder binaries that spawn children and don't forward signals; NFS/network filesystem stalls during build; deadlocked builder processes; stopping instances under heavy load where the builder needs more than 5s to unwind.","solutions":["Ensure builder processes handle SIGTERM and exit promptly (and forward signals to child processes).","Investigate the builder for hangs: check for open file locks, blocked network calls, or uninterruptible I/O.","Retry Stop or escalate with SIGKILL manually (kill -9 on the process) if the builder is stuck.","Patch the 5-second grace period in instance.go if your builders legitimately need longer to terminate."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := inst.Stop(); err != nil && strings.Contains(err.Error(), \"failed to stop instance\") {\n    logger.Warningf(\"builder %s did not exit within grace period: %s\", pkgID, err)\n    // escalate: locate and SIGKILL the process\n}","preventionTips":["Make builder binaries handle SIGTERM and reap their children.","Avoid blocking syscalls/NFS mounts inside builders.","Monitor build durations; investigate builders approaching the 5s unwind window.","Test builder shutdown behavior under load before production."],"tags":["process-management","timeout","sigterm"],"backgroundTag":"process-termination-timeout","analyzedSha":"2736b63f8fd5932511d56fe68b7039d15977f7f6","analyzedAt":"2026-09-04T08:52:36.465Z","contentChangedAt":"2026-09-04T08:52:36.465Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}