What causes "reconcile stale %s: %w" in multica-ai/multica?

Identify the named entry under hermes-home and check ownership/permissions: fix with chown/chmod -R on the overlay dir.

reconcile stale %s: %w

Error message

reconcile stale %s: %w

What it means

reconcileMirroredEntries tried to os.RemoveAll an overlay entry that is neither overlay-owned nor currently mirrored from the shared home, and the removal failed. This is the pruning half of mirroring: a shared entry deleted between runs (or a Windows copy-fallback file) must not linger, and a failed prune aborts the overlay rather than shipping stale state.

Source

Thrown at server/internal/daemon/execenv/hermes_home.go:562

// reconcileMirroredEntries removes overlay entries that are neither overlay-owned
// nor currently mirrored from the shared home, so a shared entry deleted between
// runs (or a Windows copy-fallback left behind) doesn't survive as stale state.
func reconcileMirroredEntries(hermesHome string, mirrored map[string]struct{}) error {
	entries, err := os.ReadDir(hermesHome)
	if err != nil {
		return fmt.Errorf("read overlay home: %w", err)
	}
	for _, entry := range entries {
		name := entry.Name()
		if isHermesOverlayOwnedEntry(name) {
			continue
		}
		if _, keep := mirrored[name]; keep {
			continue
		}
		if err := os.RemoveAll(filepath.Join(hermesHome, name)); err != nil {
			return fmt.Errorf("reconcile stale %s: %w", name, err)
		}
	}
	return nil
}

// prepareHermesTaskLocalState migrates an overlay built by an older daemon away
// from the shared Hermes SQLite session store. Without the marker, state.db and
// its sidecars may be symlinks or independently copied files; neither is safe to
// reuse as task-local state. Remove only those entries inside the generated
// overlay, then record the migration atomically. Hermes lazily creates a fresh
// database, and later prepares preserve it because the marker is present.
func prepareHermesTaskLocalState(hermesHome string) error {
	marker := filepath.Join(hermesHome, hermesTaskLocalStateMarker)
	if fi, err := os.Lstat(marker); err == nil {
		if !fi.Mode().IsRegular() {
			return fmt.Errorf("state marker is not a regular file: %s", marker)
		}
		return nil

View on GitHub (pinned to 2c0912b6ec)

Solutions

Identify the named entry under hermes-home and check ownership/permissions: fix with chown/chmod -R on the overlay dir.
On Windows, stop processes holding handles to the entry (or reboot) and retry the task.
If ownership is unsalvageable, wipe the entire overlay dir — it is fully regenerable from the shared home.
Run the daemon under a consistent UID so overlay entries are always deletable by later runs.

Defensive patterns

Strategy: try-catch

Try / catch

if err := reconcileMirroredEntries(hermesHome, mirrored); err != nil {
	var pe *os.PathError
	if errors.As(err, &pe) && errors.Is(pe.Err, syscall.EACCES) {
		// foreign-owned or read-only stale entry: reset the whole regenerable overlay
		return os.RemoveAll(hermesHome) // caller rebuilds on next prepare
	}
	return err
}

Prevention

Run all tasks for an env under the same UID.
On Windows, make sure previous task processes exit before overlay reuse.
Do not chmod overlay contents read-only; the daemon must be able to prune them.

When it happens

Trigger: The stale entry is read-only or owned by another user; on Windows the file is open/locked (SQLite or indexer holding a handle); the dir contains a file the daemon UID cannot unlink (sticky bit, immutable flag).

Common situations: Overlay reused across runs with changed UIDs; a previous task still holds an open handle to a pruned file on Windows; a leftover root-owned file from a privileged debugging session.

Related errors

AI-assisted analysis of multica-ai/multica@2c0912b6ec (2026-08-15). Data as JSON: /api/errors/1788429752cb1d44. Report an issue: GitHub.