{"record":{"id":"fe07f9f7879b779c","repo":"abhigyanpatwari/GitNexus","slug":"ladybugdb-checkpoint-sidecar-is-missing-for-dbpa-fe07f9","errorCode":null,"errorMessage":"LadybugDB checkpoint sidecar is missing for ${dbPath}. Rebuild the index with `gitnexus analyze --force <repo-path> --index-only` and restart `gitnexus serve`.\n  Original error: ${msg.slice(0, 200)}\n  The workspace appears to be read-only — mount it read-write to perform shadow replay recovery, or re-run `gitnexus analyze` on a writable filesystem to rebuild the index.","messagePattern":"LadybugDB checkpoint sidecar is missing for (.+?)\\. Rebuild the index with `gitnexus analyze --force <repo-path> --index-only` and restart `gitnexus serve`\\.\n  Original error: (.+?)\n  The workspace appears to be read-only — mount it read-write to perform shadow replay recovery, or re-run `gitnexus analyze` on a writable filesystem to rebuild the index\\.","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"gitnexus/src/core/lbug/lbug-adapter.ts","lineNumber":637,"sourceCode":"      await closeLbugConnection(handle);\n      return await reopenReadOnlyAfterMissingShadow(dbPath, err);\n    }\n    if (!isReadOnlyShadowReplayError(err)) {\n      await closeLbugConnection(handle);\n      throw err;\n    }\n    shadowReplayErr = err;\n  }\n\n  await closeLbugConnection(handle);\n\n  let writable: LbugConnectionHandle;\n  try {\n    writable = await openLbugConnection(lbug, dbPath);\n  } catch (openErr) {\n    const code = extractErrnoCode(openErr);\n    if (code === 'EROFS' || code === 'EACCES' || code === 'EPERM') {\n      throw new Error(\n        shadowSidecarRecoveryMessage(dbPath, shadowReplayErr) +\n          '\\n  The workspace appears to be read-only — mount it read-write to perform shadow replay recovery,' +\n          ' or re-run `gitnexus analyze` on a writable filesystem to rebuild the index.',\n      );\n    }\n    throw openErr;\n  }\n  let missingShadowError: unknown;\n  try {\n    await queryAndDrain(writable.conn, READ_ONLY_SHADOW_REPLAY_PROBE);\n  } catch (err) {\n    if (isMissingShadowSidecarError(err)) {\n      missingShadowError = err;\n    } else {\n      throw err;\n    }\n  } finally {\n    await closeLbugConnection(writable);","sourceCodeStart":619,"sourceCodeEnd":655,"githubUrl":"https://github.com/abhigyanpatwari/GitNexus/blob/aac7515d2a8c50a1f8f923c6fb77218b333560d6/gitnexus/src/core/lbug/lbug-adapter.ts#L619-L655","documentation":"lbug-adapter.ts line 637 (ensureReadOnlyConnectionUsable): after a read-path shadow-replay failure, the adapter tries to open a writable connection to perform shadow replay recovery; if that open fails with EROFS, EACCES, or EPERM, the standard sidecar-recovery message gains a suffix explaining the workspace is read-only. The operator must either mount the storage read-write so recovery can run, or rebuild the index on a writable filesystem.","triggerScenarios":"The index lives on a read-only volume (ro Docker mount, mounted CI artifact, write-protected share) AND the checkpoint sidecar is missing so recovery needs a writable open; permission bits deny the process write access to the storage dir.","commonSituations":"Serving prebuilt indexes from read-only container layers; shared RO network mounts in orgs that distribute indexes; running serve under a user without write rights on the storage directory.","solutions":["Mount the storage directory read-write (or fix permissions) and re-run so shadow replay recovery can complete","Alternatively, on a writable machine run `gitnexus analyze --force <repo-path> --index-only` and redistribute the rebuilt index to the read-only location","Design CI to build indexes on writable storage first, then mount them read-only for serving — a read-only index with a healthy sidecar never needs this recovery"],"exampleFix":"# before\n$ docker run -v indexes:/indexes:ro ... gitnexus serve\n# after: give the recovery path write access, or rebuild elsewhere\n$ gitnexus analyze --force /repo --index-only   # on writable storage\n$ docker run -v indexes:/indexes:ro ... gitnexus serve","handlingStrategy":"validation","validationCode":"import fs from 'node:fs';\n\n// Before serving from shared/read-only storage, verify the index is writable-recoverable\ntry {\n  fs.accessSync(path.dirname(dbPath), fs.constants.W_OK);\n} catch {\n  console.warn('Storage is read-only: a missing-sidecar index here cannot self-recover. Serve only verified indexes, or mount read-write.');\n}","typeGuard":null,"tryCatchPattern":"try {\n  await openReadOnly(dbPath);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('workspace appears to be read-only')) {\n    // remount storage read-write and retry once, or rebuild the index on writable storage and redistribute\n  }\n  throw err;\n}","preventionTips":["Build indexes on writable storage, verify serve starts, and only then mount them read-only for serving","In containers, mount the storage volume rw for any role that may trigger recovery (serve, analyze)","Ensure the serving process's user owns or can write the storage directory"],"tags":["wal","ladybugdb","read-only-filesystem","permissions","recovery","containers"],"backgroundTag":"read-only-filesystem","analyzedSha":"aac7515d2a8c50a1f8f923c6fb77218b333560d6","analyzedAt":"2026-08-20T23:29:22.980Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}