clockworklabs/SpacetimeDB · error · SnapshotError::ReadObject

missing object

Error message

missing object {}

What it means

During snapshot pull/copy (crates/snapshot/src/remote.rs), each object is located in the source object repo before being hardlinked or copied. In dry_run mode the code still validates presence: if the expected object file is missing from the object repo it returns SnapshotError::ReadObject with an inner io::Error kind NotFound, message 'missing object {path}'. Outside dry run, the path is returned and the missing file surfaces as a regular read error later.

Solutions

1. Run the non-dry-run clone/pull of the snapshot from a known-good source so missing objects are fetched rather than validated as absent
2. Inspect the object repo for truncation/pruning (compare expected object count/hash listing with on-disk files) and restore missing objects from a backup
3. Ensure GC/retention cannot delete objects still referenced by snapshots being pulled (check retention ordering)
4. If the parent-repo hardlink path is involved, verify the parent repo actually contains the object

Example fix

// before
let mut pull = SnapshotPull { dry_run: true, .. };
pull.run().await?; // ReadObject { cause: NotFound, "missing object objects/ab/cdef..." }

// after
// fetch/repair the object repo first, then verify
spacetime snapshot clone --source <healthy-host> <snapshot-id> --force
let mut pull = SnapshotPull { dry_run: true, .. };
pull.run().await?; // passes once objects present

#[cfg(unix)]
fn object_present(root: &Path, rel: &Path) -> bool { root.join(rel).is_file() }
// In dry-run flows, check each object before running the pull:
for rel in expected_object_paths { if !object_present(&obj_root, &rel) { /* repair first */ } }

fn is_missing_object_err(e: &SnapshotError) -> bool {
    matches!(e, SnapshotError::ReadObject { cause, .. }
        if cause.kind() == std::io::ErrorKind::NotFound)
}

match pull.run().await {
    Err(e) if is_missing_object_err(&e) => {
        // repair: clone from a healthy source, then re-run
        spacetime_clone_from_healthy_source(snapshot_id).await?;
        pull.run().await?
    }
    r => r?,
}

When it happens

Trigger: Running a dry-run snapshot pull against an object repo that is incomplete — objects pruned by GC, a partial/interrupted prior clone, or a manually truncated objects directory; hardlink source and object repo disagreeing after external tampering; moving/copying an objects directory without all files.

Common situations: Pipelines that dry-run a snapshot verification before the real copy, revealing prior corruption; GC racing with snapshot creation; backups restored partially; concurrent spacetimedb instances sharing an object store incorrectly.

Related errors

• failed to fsync directory
• failed to fsync file
• failed to open directory
• failed to open file for fsync
• AddColumns: table ` ` not found in new module def

AI-assisted analysis of clockworklabs/SpacetimeDB@6dee26c6ef (2026-08-20). Data as JSON: /api/errors/2526d3d49c86e717. Report an issue: GitHub.

Appendix: source

Thrown at crates/snapshot/src/remote.rs:565

            self.stats.skipped_object();
            return Ok(None);
        }

        if self.try_hardlink(hash).await? {
            if self.dry_run {
                return Ok(Some(path));
            }

            self.stats.hardlinked_object();
            return Ok(None);
        }

        if self.dry_run {
            return Err(SnapshotError::ReadObject {
                ty,
                source_repo: self.object_repo.root().to_owned(),
                cause: io::Error::new(io::ErrorKind::NotFound, format!("missing object {}", path.display())),
            });
        }

        Ok(Some(path))
    }

    async fn try_hardlink(&self, hash: blake3::Hash) -> Result<bool> {
        let Some(parent) = self.parent_repo.as_ref() else {
            return Ok(false);
        };

        let object_repo = Arc::clone(&self.object_repo);
        let parent_repo = Arc::clone(parent);
        if !self.dry_run {
            spawn_blocking(move || object_repo.try_hardlink_from(&parent_repo, hash.as_bytes()))
                .await
                .unwrap()
                .map_err(Into::into)

View on GitHub (pinned to 6dee26c6ef)