What causes "timeout clearing spawn marker %s" in gastownhall/beads?

Wait and retry once other proxy processes have exited (check with ps/lsof which process holds the lock)

timeout clearing spawn marker %s

Error message

timeout clearing spawn marker %s

What it means

This error indicates the spawn-marker clearing loop kept hitting a locked lockfile past its deadline, so clearing the marker timed out. The library throws it to prevent waiting forever on a spawn marker that some other process holds locked, typically when a prior proxy process is still shutting down or is stuck.

Source

Thrown at internal/storage/dbproxy/proxy/endpoint.go:1081

		}
		if *marker != own {
			return nil
		}

		lock, err := util.TryLock(filepath.Join(rootDir, LockFileName))
		if err == nil {
			current, readErr := readSpawnMarker(rootDir)
			if readErr == nil && current != nil && *current == own {
				readErr = clearSpawnMarkerAfterLock(rootDir)
			}
			lock.Unlock()
			return readErr
		}
		if !lockfile.IsLocked(err) {
			return err
		}
		if time.Now().After(deadline) {
			return fmt.Errorf("timeout clearing spawn marker %s", filepath.Join(rootDir, spawnMarkerFileName))
		}
		time.Sleep(shutdownConfirmPoll)
	}
}

View on GitHub (pinned to 71377f2769)

Solutions

Wait and retry once other proxy processes have exited (check with ps/lsof which process holds the lock)
Identify and kill the stale holder: use lsof on the marker file to find the locking PID and terminate it if it is defunct
Remove the marker file manually when no live proxy owns it, then re-run the command
Increase the allowed deadline if shutdown legitimately takes long on this machine

Defensive patterns

Strategy: retry

Validate before calling

func markerLockFree(rootDir string) bool {
    f, err := os.OpenFile(filepath.Join(rootDir, "spawn.marker"), os.O_RDWR, 0o644)
    if err != nil {
        return os.IsNotExist(err)
    }
    defer f.Close()
    return syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB) == nil
}

Try / catch

err := clearSpawnMarkerAfterLock(rootDir)
if err != nil && strings.Contains(err.Error(), "timeout clearing spawn marker") {
    // find and wait for the lock holder, then retry once
    time.Sleep(2 * time.Second)
    err = clearSpawnMarkerAfterLock(rootDir)
}

Prevention

Ensure proxy processes handle SIGTERM and release locks promptly
Avoid running many concurrent bd stop/verify operations on one workspace
Use lsof to check lock holders before force-clearing
Increase shutdown deadline on slow machines

When it happens

Trigger: Calling clearSpawnMarkerAfterLock (or the polling path at endpoint.go:1081) while another process holds an flock/lockfile on the spawn marker, and the lock is not released before the deadline computed from shutdownConfirmDeadline.

Common situations: A proxy process hung mid-shutdown holding the marker lock, a killed process whose lock lingers momentarily, or heavy contention when many bd invocations race to stop/verify proxies in the same workspace.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of gastownhall/beads@71377f2769 (2026-08-30). Data as JSON: /api/errors/337dc66f9a74c7c9. Report an issue: GitHub.