{"record":{"id":"22ed532d2e6cff0c","repo":"gastownhall/beads","slug":"re-check-proxy-stop-epoch-before-publish-w","errorCode":null,"errorMessage":"re-check proxy stop epoch before publish: %w","messagePattern":"re-check proxy stop epoch before publish: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/dbproxy/proxy/server.go","lineNumber":311,"sourceCode":"\tidentReply.UpstreamID = upstreamID\n\tidentReply.PID = os.Getpid()\n\tidentReply.Birth = string(birth)\n\tidentReply.ControlPort = control.Port()\n\tidentMu.Unlock()\n\n\t// Last fence before publishing: the spawn marker was cleared when this\n\t// process took proxy.lock, so a `bd dolt stop` that began during a slow\n\t// backend start has no record of this attempt. It did advance the stop\n\t// epoch first, so re-check it here and abort instead of publishing a\n\t// running proxy after that stop returned. The startup epoch watcher is\n\t// stopped (synchronously) first: past this fence a stop finds the\n\t// published proxy.pid and stops the proxy through it, so a watcher\n\t// cancellation must not race the publish.\n\tstopEpochWatch()\n\tif changed, err := stopEpochChanged(p.rootDir, p.stopEpoch); err != nil {\n\t\tp.stats.IncBackendStop()\n\t\t_ = stopBackendBounded(p.server)\n\t\treturn fmt.Errorf(\"re-check proxy stop epoch before publish: %w\", err)\n\t} else if changed {\n\t\treturn abortInterruptedStart()\n\t}\n\n\tif err := pidfile.Write(p.rootDir, PIDFileName, pidfile.PidFile{\n\t\tPid:         os.Getpid(),\n\t\tPort:        dataPort.Port,\n\t\tUpstreamID:  upstreamID,\n\t\tSchema:      pidfile.SchemaV2,\n\t\tKind:        pidfile.KindProxy,\n\t\tBirth:       string(birth),\n\t\tRootID:      rootID,\n\t\tControlPort: control.Port(),\n\t}); err != nil {\n\t\tp.stats.IncBackendStop()\n\t\t_ = stopBackendBounded(p.server)\n\t\treturn fmt.Errorf(\"write pid file: %w\", err)\n\t}","sourceCodeStart":293,"sourceCodeEnd":329,"githubUrl":"https://github.com/gastownhall/beads/blob/71377f276968b452ee607177637970a4ff888584/internal/storage/dbproxy/proxy/server.go#L293-L329","documentation":"Wraps an error from the final stopEpochChanged re-check performed just before publishing proxy.pid. To avoid publishing a pidfile for a proxy that a concurrent stop is already tearing down, the epoch is re-read after cancelling the epoch watcher; if that read itself fails (not merely 'changed'), startup aborts with this error and the backend is stopped.","triggerScenarios":"The stopEpochChanged(p.rootDir, p.stopEpoch) call after stopEpochWatch() returns an error - usually an I/O failure reading the stop-epoch state file under rootDir.","commonSituations":"rootDir became unwritable/unreadable (volume detached, permissions flipped); disk I/O errors on the host; concurrent deletion of the proxy state directory by an external cleaner.","solutions":["Check disk health and filesystem errors in system logs (dmesg/journalctl)","Verify rootDir and the epoch state file are readable by the proxy process","Ensure nothing external (cleaner job, another tool) is deleting files in rootDir while the proxy starts","Retry startup once the storage issue is resolved"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify rootDir state files are readable before start\nf, err := os.Open(filepath.Join(rootDir, \"stop-epoch\"))\nif err != nil {\n    return fmt.Errorf(\"cannot read stop-epoch state: %w\", err)\n}\nf.Close()","typeGuard":null,"tryCatchPattern":"if err := p.ListenAndServe(ctx); err != nil {\n    if strings.Contains(err.Error(), \"re-check proxy stop epoch\") {\n        log.Fatalf(\"storage I/O failure during startup: %v\", err)\n    }\n    log.Fatal(err)\n}","preventionTips":["Monitor disk health and filesystem error counters","Prevent external cleanup jobs from touching rootDir while a proxy starts","Run the proxy under the storage owner account with stable permissions"],"tags":["filesystem","io","epoch","shutdown-race"],"backgroundTag":"epoch-state-read-failed","analyzedSha":"71377f276968b452ee607177637970a4ff888584","analyzedAt":"2026-08-30T18:55:39.744Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}