clockworklabs/SpacetimeDB · error · io::Error

repo {}: {}: {}

Error message

repo {}: {}: {}

What it means

Raised inside create_segment when acquiring the advisory lock file (<segment>.lock) fails. The message is 'repo <dir>: <lock error>: <cause>' and the ErrorKind is taken from the underlying cause, so the actionable detail is in the inner text. Lock acquisition happens whenever the log initializes or rolls to a new segment.

Source

Thrown at crates/commitlog/src/repo/fs.rs:243

            Ok(len)
        }
    }
}

impl Repo for Fs {
    type SegmentWriter = File;
    type SegmentReader = ReadOnlySegment;

    fn create_segment(&self, offset: u64, header: segment::Header) -> io::Result<Self::SegmentWriter> {
        let path = self.segment_path(offset);

        // We need to check if the segment already exists,
        // so use file locking to prevent a TOCTOU race.
        // Using `flock` means we don't need to worry about stale lockfiles.
        let lock_path = path.0.with_extension("lock");
        let _lock = scopeguard::guard(
            lockfile::advisory::LockedFile::lock(&lock_path)
                .map_err(|e| io::Error::new(e.source.kind(), format!("repo {}: {}: {}", self, e, e.source)))?,
            |lockfile| {
                if let Err(e) = lockfile.release(true) {
                    // It's ok if removing the file fails, but print a warning
                    // anyways.
                    warn!("repo {}: failed to remove {}: {}", self, lock_path.display(), e);
                }
            },
        );

        // Check whether the segment already exists.
        // Overwrite it if its length is zero.
        match fs::metadata(&path) {
            Ok(stat) => {
                if stat.len() > 0 {
                    return Err(io::Error::new(
                        io::ErrorKind::AlreadyExists,
                        format!("repo {}: segment {} already exists and is non-empty", self, offset),
                    ));

View on GitHub (pinned to 524b4487d9)

Solutions

1. Read the inner cause in the message: check write permissions on the directory, free disk space, and the process fd limit
2. Ensure exactly one writer process per log directory
3. Make the directory writable, then retry the commit/open that triggered segment creation
4. If another process holds the lock, stop that process; flock locks die with their holder, so a stuck lock usually means a live holder - do not delete lock files

fn dir_ready_for_log(dir: &std::path::Path) -> io::Result<()> {
    let meta = std::fs::metadata(dir)?;
    if meta.is_dir() && !meta.permissions().readonly() {
        Ok(())
    } else {
        Err(io::Error::new(io::ErrorKind::PermissionDenied, "log dir missing or read-only"))
    }
}

fn is_segment_lock_error(e: &io::Error) -> bool {
    e.to_string().contains(".lock")
}

// transient lock contention or fd pressure: back off and retry
let mut backoff = std::time::Duration::from_millis(50);
loop {
    match attempt {
        Ok(v) => break v,
        Err(e) if is_segment_lock_error(&e) && backoff < std::time::Duration::from_secs(5) => {
            std::thread::sleep(backoff);
            backoff *= 2;
        }
        Err(e) => return Err(e),
    }
}

When it happens

Trigger: The log directory is not writable (PermissionDenied while creating <segment>.lock); the lock file is held by another live process; the filesystem is full or the process exhausted its file-descriptor limit while creating the lock file.

Common situations: Two instances pointed at the same directory; containers/read-only mounts with dropped write permission; ENOSPC or EMFILE under heavy load right when a segment rolls.

Related errors

• repo {}: error getting file metadata for segment {}: {}
• error reading commit header: {e}
• failed to read {} bytes of commit payload: {}
• failed to read checksum: {e}
• InvalidInput

AI-assisted analysis of clockworklabs/SpacetimeDB@524b4487d9 (2026-08-16). Data as JSON: /api/errors/b7e976d44e754429. Report an issue: GitHub.