weaviate/weaviate · error

create backup staging dir: %w

Error message

create backup staging dir: %w

What it means

During backup class-description with hardlinks (Index.descriptorWithHardlinks), Weaviate creates the per-class staging directory under RootPath (backupStagingDir) where shard files will be hardlinked. This error wraps the os.MkdirAll failure, meaning the staging directory could not be created on the backup filesystem. The deferred cleanup then removes any partial staging dir and releases the backup lock, so the backup for the class fails before any shard work starts.

Source

Thrown at adapters/repos/db/backup.go:287

	i.logger.WithField("hardlinks_supported", useHardlinks).Info("backup: probed filesystem hardlink support")

	if useHardlinks {
		return i.descriptorWithHardlinks(ctx, backupID, desc, classBaseDescrs)
	}
	// NO-HARDLINK-BACKUP: only reachable on filesystems without hardlink support.
	// Removed in v1.40; bugs here are not fixed.
	return i.descriptorWithoutHardlinks(ctx, backupID, desc, classBaseDescrs)
}

// descriptorWithHardlinks creates hard-linked snapshots per shard, allowing compaction
// to resume immediately after the snapshot is taken (~2-5s pause per shard).
//
// It iterates the sharding state (single source of truth) to discover all local shards,
// then uses the shardMap to determine the backup method per shard under backupLock.Lock.
func (i *Index) descriptorWithHardlinks(ctx context.Context, backupID string, desc *backup.ClassDescriptor, classBaseDescrs []*backup.ClassDescriptor) (err error) {
	stagingRoot := backupStagingDir(i.Config.RootPath, backupID, i.Config.ClassName)
	if err := os.MkdirAll(stagingRoot, 0o755); err != nil {
		return fmt.Errorf("create backup staging dir: %w", err)
	}

	defer func() {
		if err != nil {
			os.RemoveAll(stagingRoot)
			enterrors.GoWrapper(func() { i.ReleaseBackup(ctx, backupID) }, i.logger)
		}
	}()

	desc.StagingDir = stagingRoot

	shardNames, stateBytes, err := i.readSchema()
	if err != nil {
		return fmt.Errorf("list local shards: %w", err)
	}

	eg, ctx := enterrors.NewErrorGroupWithContextWrapper(i.logger, ctx)
	eg.SetLimit(_NUMCPU)

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check disk space and mount status of the node's RootPath volume (df -h, mount flags); free space or remount read-write.
2. Verify permissions on the backup staging parent directory so the weaviate process user can create directories (chown/chmod the path).
3. Remove any leftover file (not directory) at the staging path from a previous failed backup with the same backupID.
4. Confirm the backupID and RootPath produce a valid path (no illegal characters or excessive length) and retry the backup with a new backupID.
5. If on NFS, ensure the mount is healthy and supports directory creation, or point backups to local disk.

Example fix

// before: retrying blindly after "create backup staging dir: permission denied"
client.Backup().Create(ctx, backend, backupID, class).
// after: preflight the target dir from outside the library
if err := os.MkdirAll(backupRoot, 0o755); err != nil {
    return fmt.Errorf("backup root not writable: %w", err)
}
client.Backup().Create(ctx, backend, backupID, class)

// Preflight: ensure backup root exists and is writable before starting a backup
if err := os.MkdirAll(backupRoot, 0o755); err != nil {
    return fmt.Errorf("backup root not writable: %w", err)
}
if fi, err := os.Stat(backupRoot); err != nil || !fi.IsDir() {
    return fmt.Errorf("backup root is not a directory")
}

// Client-side pattern (REST)
resp, err := client.Backup().Create(ctx, backend, backupID, class).Do(ctx)
if err != nil {
    if strings.Contains(err.Error(), "create backup staging dir") {
        // inspect RootPath volume: permissions, disk full, read-only mount
    }
    return err
}

When it happens

Trigger: Starting a backup (v1 REST backup endpoint, create-backup) on a node whose backup root path is not writable or does not exist: disk full, read-only filesystem, wrong permissions on RootPath, path too long, or an existing file (not directory) at the staging path.

Common situations: Kubernetes/container deployments where RootPath is on a read-only or removed volume; NFS/network mounts that dropped or deny writes; permission changes after running the process as a different user; a leftover non-directory file from a previous failed backup at the same backupID path.

Related errors

• open file %q: %w
• open backups path: %w
• get object path %s
• make dir %s
• stat shard dir: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/670ab2d667567810. Report an issue: GitHub.