What causes "failed to stop instance '%s'" in hyperledger/fabric?

Ensure builder processes handle SIGTERM and exit promptly (and forward signals to child processes).

failed to stop instance '%s'

Error message

failed to stop instance '%s'

What it means

After sending SIGTERM, Stop waits 5 seconds for the process to exit; if the Wait goroutine has not completed within that grace period it gives up and returns this error (the process is effectively left to SIGKILL semantics/timed out). It identifies the stuck instance by PackageID.

Source

Thrown at core/container/externalbuilder/instance.go:176

func (i *Instance) Stop() error {
	if i.Session == nil {
		return errors.Errorf("instance has not been started")
	}

	done := make(chan struct{})
	go func() { i.Wait(); close(done) }()

	i.Session.Signal(syscall.SIGTERM)
	select {
	case <-time.After(i.TermTimeout):
		i.Session.Signal(syscall.SIGKILL)
	case <-done:
		return nil
	}

	select {
	case <-time.After(5 * time.Second):
		return errors.Errorf("failed to stop instance '%s'", i.PackageID)
	case <-done:
		return nil
	}
}

func (i *Instance) Wait() (int, error) {
	if i.Session == nil {
		return -1, errors.Errorf("instance was not successfully started")
	}

	err := i.Session.Wait()
	err = errors.Wrapf(err, "builder '%s' run failed", i.Builder.Name)
	if exitErr, ok := errors.Cause(err).(*exec.ExitError); ok {
		return exitErr.ExitCode(), err
	}
	return 0, err
}

View on GitHub (pinned to 2736b63f8f)

Solutions

Ensure builder processes handle SIGTERM and exit promptly (and forward signals to child processes).
Investigate the builder for hangs: check for open file locks, blocked network calls, or uninterruptible I/O.
Retry Stop or escalate with SIGKILL manually (kill -9 on the process) if the builder is stuck.
Patch the 5-second grace period in instance.go if your builders legitimately need longer to terminate.

Defensive patterns

Strategy: retry

Try / catch

if err := inst.Stop(); err != nil && strings.Contains(err.Error(), "failed to stop instance") {
    logger.Warningf("builder %s did not exit within grace period: %s", pkgID, err)
    // escalate: locate and SIGKILL the process
}

Prevention

Make builder binaries handle SIGTERM and reap their children.
Avoid blocking syscalls/NFS mounts inside builders.
Monitor build durations; investigate builders approaching the 5s unwind window.
Test builder shutdown behavior under load before production.

When it happens

Trigger: A builder process that ignores or is blocked on SIGTERM so it doesn't exit within the hard-coded 5s window after Stop() signals it; a hung builder blocked on I/O, a child process, or a stuck network call.

Common situations: Builder binaries that spawn children and don't forward signals; NFS/network filesystem stalls during build; deadlocked builder processes; stopping instances under heavy load where the builder needs more than 5s to unwind.

Related errors

AI-assisted analysis of hyperledger/fabric@2736b63f8f (2026-09-04). Data as JSON: /api/errors/2dfd98047c2baac4. Report an issue: GitHub.