{"record":{"id":"2526d3d49c86e717","repo":"clockworklabs/SpacetimeDB","slug":"missing-object","errorCode":null,"errorMessage":"missing object {}","messagePattern":"missing object (.+?)","errorType":"exception","errorClass":"SnapshotError::ReadObject","httpStatus":null,"severity":"error","filePath":"crates/snapshot/src/remote.rs","lineNumber":565,"sourceCode":"\n            self.stats.skipped_object();\n            return Ok(None);\n        }\n\n        if self.try_hardlink(hash).await? {\n            if self.dry_run {\n                return Ok(Some(path));\n            }\n\n            self.stats.hardlinked_object();\n            return Ok(None);\n        }\n\n        if self.dry_run {\n            return Err(SnapshotError::ReadObject {\n                ty,\n                source_repo: self.object_repo.root().to_owned(),\n                cause: io::Error::new(io::ErrorKind::NotFound, format!(\"missing object {}\", path.display())),\n            });\n        }\n\n        Ok(Some(path))\n    }\n\n    async fn try_hardlink(&self, hash: blake3::Hash) -> Result<bool> {\n        let Some(parent) = self.parent_repo.as_ref() else {\n            return Ok(false);\n        };\n\n        let object_repo = Arc::clone(&self.object_repo);\n        let parent_repo = Arc::clone(parent);\n        if !self.dry_run {\n            spawn_blocking(move || object_repo.try_hardlink_from(&parent_repo, hash.as_bytes()))\n                .await\n                .unwrap()\n                .map_err(Into::into)","sourceCodeStart":547,"sourceCodeEnd":583,"githubUrl":"https://github.com/clockworklabs/SpacetimeDB/blob/6dee26c6efc2856793e12b148a59742964f5d783/crates/snapshot/src/remote.rs#L547-L583","documentation":"During snapshot pull/copy (crates/snapshot/src/remote.rs), each object is located in the source object repo before being hardlinked or copied. In dry_run mode the code still validates presence: if the expected object file is missing from the object repo it returns SnapshotError::ReadObject with an inner io::Error kind NotFound, message 'missing object {path}'. Outside dry run, the path is returned and the missing file surfaces as a regular read error later.","triggerScenarios":"Running a dry-run snapshot pull against an object repo that is incomplete — objects pruned by GC, a partial/interrupted prior clone, or a manually truncated objects directory; hardlink source and object repo disagreeing after external tampering; moving/copying an objects directory without all files.","commonSituations":"Pipelines that dry-run a snapshot verification before the real copy, revealing prior corruption; GC racing with snapshot creation; backups restored partially; concurrent spacetimedb instances sharing an object store incorrectly.","solutions":["Run the non-dry-run clone/pull of the snapshot from a known-good source so missing objects are fetched rather than validated as absent","Inspect the object repo for truncation/pruning (compare expected object count/hash listing with on-disk files) and restore missing objects from a backup","Ensure GC/retention cannot delete objects still referenced by snapshots being pulled (check retention ordering)","If the parent-repo hardlink path is involved, verify the parent repo actually contains the object"],"exampleFix":"// before\nlet mut pull = SnapshotPull { dry_run: true, .. };\npull.run().await?; // ReadObject { cause: NotFound, \"missing object objects/ab/cdef...\" }\n\n// after\n// fetch/repair the object repo first, then verify\nspacetime snapshot clone --source <healthy-host> <snapshot-id> --force\nlet mut pull = SnapshotPull { dry_run: true, .. };\npull.run().await?; // passes once objects present","handlingStrategy":"validation","validationCode":"#[cfg(unix)]\nfn object_present(root: &Path, rel: &Path) -> bool { root.join(rel).is_file() }\n// In dry-run flows, check each object before running the pull:\nfor rel in expected_object_paths { if !object_present(&obj_root, &rel) { /* repair first */ } }","typeGuard":"fn is_missing_object_err(e: &SnapshotError) -> bool {\n    matches!(e, SnapshotError::ReadObject { cause, .. }\n        if cause.kind() == std::io::ErrorKind::NotFound)\n}","tryCatchPattern":"match pull.run().await {\n    Err(e) if is_missing_object_err(&e) => {\n        // repair: clone from a healthy source, then re-run\n        spacetime_clone_from_healthy_source(snapshot_id).await?;\n        pull.run().await?\n    }\n    r => r?,\n}","preventionTips":["Run real (non-dry-run) clones from known-good sources before dry-run verification","Ensure GC cannot prune objects still referenced by snapshots being pulled","Back up the objects directory; verify counts/hashes after restore","Do not hand-edit or partially copy objects directories between hosts"],"tags":["snapshot","storage-integrity","rust","dry-run","spacetimedb"],"backgroundTag":"missing-snapshot-object","analyzedSha":"6dee26c6efc2856793e12b148a59742964f5d783","analyzedAt":"2026-08-20T06:08:37.179Z","contentChangedAt":"2026-08-20T06:08:37.179Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}