weaviate/weaviate · error

snapshot shard %v: %w

Error message

snapshot shard %v: %w

What it means

This error wraps any failure from shard.CreateBackupSnapshot during a hard-link based shard backup. CreateBackupSnapshot creates a consistent snapshot of the shard's files (via checkpoint/descriptor files) and hard-links them into the staging directory. Weaviate wraps the underlying cause with the shard name so operators can identify which shard's snapshot failed mid-backup.

Source

Thrown at adapters/repos/db/backup.go:420

		// holds the LazyLoadShard mutex.
		releaseBlock()
	}

	// Acquire preventShutdown before releasing shardCreateLocks: UnloadLocalShard
	// holds only shardCreateLocks (not backupLock), so without the refcount it could
	// call Shard.Shutdown between our release and CreateBackupSnapshot.
	release, err := shard.preventShutdown()
	if err != nil {
		return nil, fmt.Errorf("prevent shutdown of shard %v: %w", name, err)
	}
	releaseShard = release

	i.shardCreateLocks.Unlock(name)
	shardCreateLocksHeld = false

	files, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)
	if err != nil {
		return nil, fmt.Errorf("snapshot shard %v: %w", name, err)
	}

	if err := sd.FillFileInfo(files, shardBaseDescr, stagingRoot); err != nil {
		return nil, fmt.Errorf("gather shard %v file info: %w", name, err)
	}

	return &sd, nil
}

// backupInactiveShardWithHardlinks backs up an inactive (unloaded) shard by reading
// its files from disk and hardlinking them into the staging directory.
func (i *Index) backupInactiveShardWithHardlinks(name string, sd *backup.ShardDescriptor, shardBaseDescr []backup.ShardAndID, stagingRoot string) error {
	shardDir := shardPath(i.path(), name)
	if _, err := os.Stat(shardDir); err != nil {
		if os.IsNotExist(err) {
			// FROZEN/OFFLOADED — no local data. Status is preserved in the
			// sharding state; omit from desc.Shards.
			return errShardNoLocalData

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check the wrapped inner error for the real cause (disk space, permissions, shutdown race) and fix that first
2. Retry the backup once the shard is active and no shutdown/drop is in flight
3. Verify the staging directory path is writable and has sufficient space
4. Check shard status (should be READY) before starting the backup

Example fix

// before
files, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)
if err != nil {
    return nil, fmt.Errorf("snapshot shard %v: %w", name, err)
}
// after — pre-check shard state and ctx before snapshotting
if err := ctx.Err(); err != nil {
    return nil, fmt.Errorf("snapshot shard %v cancelled: %w", name, err)
}
files, err := shard.CreateBackupSnapshot(ctx, &sd, stagingRoot)
if err != nil {
    return nil, fmt.Errorf("snapshot shard %v: %w", name, err)
}

// Go: before requesting the backup, verify shard readiness and staging writability
if err := ctx.Err(); err != nil { return err }
if s, _ := os.Stat(stagingRoot); s == nil || !s.IsDir() {
    return fmt.Errorf("staging root %s missing", stagingRoot)
}
test, err := os.CreateTemp(stagingRoot, ".probe*")
if err != nil { return err }
os.Remove(test.Name()); test.Close()

// errors.As to unwrap the snapshot failure
var pathErr *os.PathError
if errors.As(err, &pathErr) {
    // filesystem-level problem at pathErr.Path
}

err := client.Backup(ctx, ...)
if err != nil {
    if strings.Contains(err.Error(), "snapshot shard") {
        // check wrapped cause, verify disk/permissions, retry backup
        var perr *fs.PathError
        if errors.As(err, &perr) { log.Printf("fs failure on %s: %v", perr.Path, perr.Err) }
    }
    return err
}

When it happens

Trigger: A shard backup with hardlinks is in progress (backupShardWithHardlinks) and CreateBackupSnapshot fails — e.g. the shard's underlying store cannot freeze/flush its memtables, the descriptor cannot be written, or an I/O error occurs while linking files into stagingRoot. Commonly triggered when the shard is being shut down concurrently (the repo has a test named TestBackupShardWithHardlinks_PreventShutdownErrorReleasesLocks) or the staging directory is not writable.

Common situations: Backups racing with shard drops or server shutdown; disk-full or permission problems on the backup staging path; corrupted LSM state that prevents a clean snapshot; backup endpoints called while a shard is being loaded/unloaded.

Related errors

• stat shard dir: %w
• list inactive shard %s files: %w
• list backup files: %w
• create file list for shard %q: %w
• file %q not found at %q or %q: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/3b2e8634b8c5a42e. Report an issue: GitHub.