gastownhall/beads · error

procid: process %d still matches token after fatal signal an

Error message

procid: process %d still matches token after fatal signal and %s re-check

What it means

This error is returned by confirmFatalSignal in the Linux procid implementation after sending a fatal signal and re-checking whether the process still matches the captured identity token (PID + start time). It means the target process refused to die (or the signal did not take effect) within the fallbackSignalConfirmTimeout deadline, so procid cannot confirm the kill and refuses to report success to avoid PID-reuse ambiguity.

Source

Thrown at internal/procid/procid_linux.go:178

	}
	if !match {
		return fmt.Errorf("procid: process %d no longer matches token", h.pid)
	}
	return nil
}

func (h *Handle) confirmFatalSignal() error {
	deadline := time.Now().Add(fallbackSignalConfirmTimeout)
	for {
		match, err := Verify(h.pid, h.token)
		if err != nil {
			return err
		}
		if !match {
			return nil
		}
		if time.Now().After(deadline) {
			return fmt.Errorf(
				"procid: process %d still matches token after fatal signal and %s re-check",
				h.pid,
				fallbackSignalConfirmTimeout,
			)
		}
		time.Sleep(10 * time.Millisecond)
	}
}

func isFatalSignal(sig syscall.Signal) bool {
	return sig == syscall.SIGKILL || sig == syscall.SIGTERM
}

func processStartTime(pid int) (string, error) {
	data, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
	if err != nil {
		return "", &processStatReadError{pid: pid, err: err}
	}

View on GitHub (pinned to 71377f2769)

Solutions

1. Increase fallbackSignalConfirmTimeout if your workload legitimately tears down slowly
2. Check the target process state in /proc/<pid>/stat for 'D' (uninterruptible) state and resolve the underlying I/O block (e.g. unstick NFS/FUSE mount)
3. Verify the process is not in a frozen cgroup (check cgroup.freeze / freezer controller) and thaw it before signaling
4. Re-capture the token and retry the confirm loop; if the PID was reused, the token re-check will fail cleanly and the caller can treat the old process as gone

Example fix

// before
if err := procid.KillConfirmed(pid, token); err != nil {
	return fmt.Errorf("worker did not die: %w", err)
}
// after
if err := procid.KillConfirmed(pid, token); err != nil {
	var perr *procid.StillAliveError
	if errors.As(err, &perr) {
		// inspect /proc/<pid>/stat state; thaw cgroup or escalate
	}
	return fmt.Errorf("worker did not die: %w", err)
}

// before issuing the fatal signal, confirm the process is killable
func killable(pid int) bool {
	b, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
	if err != nil {
		return false
	}
	i := strings.LastIndexByte(string(b), ')')
	if i < 0 || len(b) <= i+2 {
		return false
	}
	state := rune(b[i+2])
	return state != 'Z' && state != 'X' && state != 'x' // 'D' state will likely time out
}

err := procid.KillConfirmed(pid, tok)
if errors.Is(err, procid.ErrStillAlive) { // or match on message/timeout
	// one bounded retry after checking process state, then escalate
	time.Sleep(500 * time.Millisecond)
	err = procid.KillConfirmed(pid, tok)
}
if err != nil {
	log.Warnf("process %d not confirmed dead: %v", pid, err)
}

When it happens

Trigger: Calling a procid kill/confirm path (confirmFatalSignal) when the process is stuck in uninterruptible sleep (D state), ignores or blocks the fatal signal (SIGKILL should not, but pre-confirmation re-checks may race), the PID was reused by a different process whose token no longer matches yet still resolves as matching due to stale /proc data, or the 10ms-poll loop exhausts the deadline before the kernel finishes reaping the process.

Common situations: Killing a wedged worker stuck in unkillable I/O (NFS, FUSE, blocked driver); signaling a process in a cgroup frozen state; heavily loaded machines where process teardown is slow; containers with frozen cgroups (SIGKILL deferred); PID namespaces where the reaper is slow.

Related errors

• procid: malformed proc stat: missing comm terminator
• procid: malformed proc stat: missing starttime
• procid: process %d still matches token after fatal signal an
• procid: unsupported signal %v
• procid: pidfd signal %d: %w

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/68fbe16e4ecc052b. Report an issue: GitHub.