weaviate/weaviate · error

hardlink inactive shard %s files to staging: %w

Error message

hardlink inactive shard %s files to staging: %w

What it means

Wraps a failure from file.HardlinkFiles when hardlinking shard files into the backup staging directory for an inactive shard. Hardlinks are used to snapshot files cheaply without copying; this fails on filesystems without hardlink support, cross-device targets, permission problems, or if a source file vanished.

Source

Thrown at adapters/repos/db/backup.go:471

	for _, relPath := range files {
		src := filepath.Join(i.Config.RootPath, relPath)
		dst := filepath.Join(stagingRoot, relPath)
		if backup.IsImmutableFile(relPath) {
			hardlinks = append(hardlinks, file.HardlinkPair{Src: src, Dst: dst})
			continue
		}
		// Mutable files are copied, not hard-linked — a shared inode would let
		// post-snapshot writes corrupt the staged copy. CopyFile, unlike
		// HardlinkFiles, doesn't create the destination dir.
		if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
			return fmt.Errorf("create staging subdir for inactive shard %s file %s: %w", name, relPath, err)
		}
		if err := file.CopyFile(src, dst); err != nil {
			return fmt.Errorf("copy inactive shard %s file %s to staging: %w", name, relPath, err)
		}
	}
	if err := file.HardlinkFiles(hardlinks); err != nil {
		return fmt.Errorf("hardlink inactive shard %s files to staging: %w", name, err)
	}

	if err := sd.FillFileInfo(files, shardBaseDescr, i.Config.RootPath); err != nil {
		return fmt.Errorf("gather inactive shard %s file info: %w", name, err)
	}

	return nil
}

// descriptorWithoutHardlinks is the fallback path for filesystems that don't support
// hardlinks. Compaction remains paused for the entire backup upload duration.
//
// Deprecated: NO-HARDLINK-BACKUP. Removed in v1.40; bugs here are not fixed.
func (i *Index) descriptorWithoutHardlinks(ctx context.Context, backupID string, desc *backup.ClassDescriptor, classBaseDescrs []*backup.ClassDescriptor) (err error) {
	defer func() {
		if err != nil {
			// closelock is hold by the caller
			enterrors.GoWrapper(func() { i.ReleaseBackup(ctx, backupID) }, i.logger)

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Ensure the backup staging directory is on the same filesystem as the shard data root
2. If the filesystem does not support hardlinks, use the no-hardlinks fallback path (descriptorWithoutHardlinks)
3. Check mount options and permissions on the staging directory
4. Read the wrapped OS error (EXDEV/EPERM) to confirm the cause

Example fix

// before: hardlinks attempted regardless of FS support
if err := file.HardlinkFiles(hardlinks); err != nil {
	return fmt.Errorf("hardlink inactive shard %s files to staging: %w", name, err)
}
// after: detect cross-device/link failure and fall back to copying
if err := file.HardlinkFiles(hardlinks); err != nil {
	if errors.Is(err, syscall.EXDEV) || errors.Is(err, syscall.EPERM) {
		return i.backupInactiveShardWithoutHardlinks(ctx, name) // copy-based fallback
	}
	return fmt.Errorf("hardlink inactive shard %s files to staging: %w", name, err)
}

// detect hardlink support on the target filesystem
if err := os.Link(probeSrc, probeDst); err != nil {
	// use the no-hardlinks backup path
}

if err := backup(); err != nil {
	if strings.Contains(err.Error(), "hardlink inactive shard") {
		// retry with the no-hardlinks fallback path
	}
	return err
}

When it happens

Trigger: Backup descriptor creation for an inactive shard when HardlinkFiles fails: staging dir on a different filesystem (EXDEV), overlay/NFS/FAT filesystem without hardlink support, or EACCES/EPERM creating links.

Common situations: Docker overlayfs or network mounts (NFS/SMB) where hardlinks are disallowed or cross-device; staging path outside the data root on another mount.

Related errors

• hardlink backup files to staging: %w
• create backup staging dir: %w
• list local shards: %w
• snapshot shard %v: %w
• gather shard %v file info: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/02b94d068a0e41e2. Report an issue: GitHub.