{"record":{"id":"670ab2d667567810","repo":"weaviate/weaviate","slug":"create-backup-staging-dir-w","errorCode":null,"errorMessage":"create backup staging dir: %w","messagePattern":"create backup staging dir: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/backup.go","lineNumber":287,"sourceCode":"\ti.logger.WithField(\"hardlinks_supported\", useHardlinks).Info(\"backup: probed filesystem hardlink support\")\n\n\tif useHardlinks {\n\t\treturn i.descriptorWithHardlinks(ctx, backupID, desc, classBaseDescrs)\n\t}\n\t// NO-HARDLINK-BACKUP: only reachable on filesystems without hardlink support.\n\t// Removed in v1.40; bugs here are not fixed.\n\treturn i.descriptorWithoutHardlinks(ctx, backupID, desc, classBaseDescrs)\n}\n\n// descriptorWithHardlinks creates hard-linked snapshots per shard, allowing compaction\n// to resume immediately after the snapshot is taken (~2-5s pause per shard).\n//\n// It iterates the sharding state (single source of truth) to discover all local shards,\n// then uses the shardMap to determine the backup method per shard under backupLock.Lock.\nfunc (i *Index) descriptorWithHardlinks(ctx context.Context, backupID string, desc *backup.ClassDescriptor, classBaseDescrs []*backup.ClassDescriptor) (err error) {\n\tstagingRoot := backupStagingDir(i.Config.RootPath, backupID, i.Config.ClassName)\n\tif err := os.MkdirAll(stagingRoot, 0o755); err != nil {\n\t\treturn fmt.Errorf(\"create backup staging dir: %w\", err)\n\t}\n\n\tdefer func() {\n\t\tif err != nil {\n\t\t\tos.RemoveAll(stagingRoot)\n\t\t\tenterrors.GoWrapper(func() { i.ReleaseBackup(ctx, backupID) }, i.logger)\n\t\t}\n\t}()\n\n\tdesc.StagingDir = stagingRoot\n\n\tshardNames, stateBytes, err := i.readSchema()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"list local shards: %w\", err)\n\t}\n\n\teg, ctx := enterrors.NewErrorGroupWithContextWrapper(i.logger, ctx)\n\teg.SetLimit(_NUMCPU)","sourceCodeStart":269,"sourceCodeEnd":305,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/backup.go#L269-L305","documentation":"During backup class-description with hardlinks (Index.descriptorWithHardlinks), Weaviate creates the per-class staging directory under RootPath (backupStagingDir) where shard files will be hardlinked. This error wraps the os.MkdirAll failure, meaning the staging directory could not be created on the backup filesystem. The deferred cleanup then removes any partial staging dir and releases the backup lock, so the backup for the class fails before any shard work starts.","triggerScenarios":"Starting a backup (v1 REST backup endpoint, create-backup) on a node whose backup root path is not writable or does not exist: disk full, read-only filesystem, wrong permissions on RootPath, path too long, or an existing file (not directory) at the staging path.","commonSituations":"Kubernetes/container deployments where RootPath is on a read-only or removed volume; NFS/network mounts that dropped or deny writes; permission changes after running the process as a different user; a leftover non-directory file from a previous failed backup at the same backupID path.","solutions":["Check disk space and mount status of the node's RootPath volume (df -h, mount flags); free space or remount read-write.","Verify permissions on the backup staging parent directory so the weaviate process user can create directories (chown/chmod the path).","Remove any leftover file (not directory) at the staging path from a previous failed backup with the same backupID.","Confirm the backupID and RootPath produce a valid path (no illegal characters or excessive length) and retry the backup with a new backupID.","If on NFS, ensure the mount is healthy and supports directory creation, or point backups to local disk."],"exampleFix":"// before: retrying blindly after \"create backup staging dir: permission denied\"\nclient.Backup().Create(ctx, backend, backupID, class).\n// after: preflight the target dir from outside the library\nif err := os.MkdirAll(backupRoot, 0o755); err != nil {\n    return fmt.Errorf(\"backup root not writable: %w\", err)\n}\nclient.Backup().Create(ctx, backend, backupID, class)","handlingStrategy":"validation","validationCode":"// Preflight: ensure backup root exists and is writable before starting a backup\nif err := os.MkdirAll(backupRoot, 0o755); err != nil {\n    return fmt.Errorf(\"backup root not writable: %w\", err)\n}\nif fi, err := os.Stat(backupRoot); err != nil || !fi.IsDir() {\n    return fmt.Errorf(\"backup root is not a directory\")\n}","typeGuard":null,"tryCatchPattern":"// Client-side pattern (REST)\nresp, err := client.Backup().Create(ctx, backend, backupID, class).Do(ctx)\nif err != nil {\n    if strings.Contains(err.Error(), \"create backup staging dir\") {\n        // inspect RootPath volume: permissions, disk full, read-only mount\n    }\n    return err\n}","preventionTips":["Provision the backup volume read-write with correct ownership for the weaviate user.","Alert on disk usage of the backup/RootPath volume before thresholds are hit.","Use unique backupIDs and clean up leftover staging dirs from failed runs.","Avoid RootPath on flaky NFS; prefer local SSD or a reliable network filesystem."],"tags":["backup","filesystem","permissions","io"],"backgroundTag":"directory-creation-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}