{"record":{"id":"0681fb513be7ec34","repo":"multica-ai/multica","slug":"clear-empty-hermes-store-s-before-publishing-w","errorCode":null,"errorMessage":"clear empty hermes store %s before publishing: %w","messagePattern":"clear empty hermes store (.+?) before publishing: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"server/internal/daemon/execenv/hermes_memory.go","lineNumber":299,"sourceCode":"// failure. The caller deletes the source directory whenever this returns\n// (false, nil), so a permission error, a read-only filesystem or a Windows\n// sharing violation must fail closed instead of passing for \"someone else\n// published\".\n//\n// published answers \"did a competitor already publish real state here?\" and is\n// the caller's to define, because the two stores disagree about what an\n// occupied directory looks like: any entry at all means memory, while a session\n// store can hold a zero-length database or orphan journal sidecars that carry no\n// transcript. Sharing one definition let a session migration read the store as\n// empty, copy into staging, then read it as occupied at publish time and report\n// a lost race — after which the caller deleted a source database that had never\n// been carried over.\nfunc promoteHermesStoreStaging(staging, storeDir string, published func(string) bool) (bool, error) {\n\tif err := os.Remove(storeDir); err != nil && !os.IsNotExist(err) {\n\t\tif published(storeDir) {\n\t\t\treturn false, nil // another task published first\n\t\t}\n\t\treturn false, fmt.Errorf(\"clear empty hermes store %s before publishing: %w\", storeDir, err)\n\t}\n\tif err := os.Rename(staging, storeDir); err != nil {\n\t\tif published(storeDir) {\n\t\t\treturn false, nil // lost a narrow race between the remove and the rename\n\t\t}\n\t\treturn false, fmt.Errorf(\"publish hermes store %s: %w\", storeDir, err)\n\t}\n\treturn true, nil\n}\n\n// hermesStorePopulated reports whether storeDir is *confirmed* to hold\n// entries. Anything it cannot confirm — an unreadable directory, a path that is\n// not a directory — is false, so a caller asking \"did another task publish?\"\n// treats uncertainty as no.\nfunc hermesStorePopulated(storeDir string) bool {\n\tentries, err := os.ReadDir(storeDir)\n\treturn err == nil && len(entries) > 0\n}","sourceCodeStart":281,"sourceCodeEnd":317,"githubUrl":"https://github.com/multica-ai/multica/blob/2c0912b6ec764b373d44eeea1e80f0d9f11ab417/server/internal/daemon/execenv/hermes_memory.go#L281-L317","documentation":"Returned by promoteHermesStoreStaging when os.Remove of the (expected-empty) store dir fails with a non-NotExist error AND hermesStorePopulated cannot confirm another task already published. The empty-dir-only remove is the guard that makes publishing safe: only after it succeeds can staging be renamed into place, and a remove failure that is not explained by a concurrent winner is fatal.","triggerScenarios":"os.Remove(storeDir) fails because the dir is non-empty and unreadable (so the populated-check cannot confirm entries), the dir is busy (EBUSY, Windows handle open in it), or permissions deny removal — and no concurrent winner explains it.","commonSituations":"Two concurrent tasks for one agent racing the migration (usually resolved as a lost race, not an error); a daemon restart leaving an open handle inside the store on Windows; a store dir whose mode bits block ReadDir so 'populated' cannot be confirmed.","solutions":["Stop concurrent tasks/daemons for the agent and retry — the lost-race path already returns cleanly, so an error means a genuine remove failure.","Fix permissions on the store dir so both the remove and the populated-check can run (0700, daemon-owned).","On Windows, ensure no process (antivirus, another daemon) holds a handle inside the store during upgrade."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-publish check used by the guard itself:\nfunc hermesStorePopulated(dir string) bool {\n    ents, err := os.ReadDir(dir)\n    return err == nil && len(ents) > 0 // unreadable == NOT confirmed, fails closed\n}","typeGuard":"func isLostRace(err error) bool {\n    // promoteHermesStoreStaging already swallows true lost races;\n    // reaching the caller means the remove failed for a real reason.\n    return false\n}","tryCatchPattern":"promoted, err := promoteHermesStoreStaging(staging, store, hermesStorePopulated)\nif err != nil && strings.Contains(err.Error(), \"clear empty hermes store\") {\n    // stop sibling tasks, then a single retry is safe: staging still exists\n    promoted, err = promoteHermesStoreStaging(staging, store, hermesStorePopulated)\n}","preventionTips":["Serialize concurrent first-mounts per agent (one task warms the store, others reuse it)","Ensure store dirs stay readable (0700 daemon-owned) so the populated-check works","On Windows, close/stop anything holding handles under the store before upgrades"],"tags":["filesystem","hermes","migration","race-condition","windows"],"backgroundTag":null,"analyzedSha":"2c0912b6ec764b373d44eeea1e80f0d9f11ab417","analyzedAt":"2026-08-15T13:25:18.241Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}