weaviate/weaviate · error

hardlink snapshot: %w

Error message

hardlink snapshot: %w

What it means

The final step of CreateSnapshot hard-links every bucket segment file from the bucket directory into the snapshot directory via snapshotBucketFiles. On any failure (missing file, cross-device link, permission error) the partially created snapshot directory is removed with os.RemoveAll and this wrapped error is returned; compaction/flush resumption already happened via defers, and the hard-linked model stays valid because compaction only creates new files.

Source

Thrown at adapters/repos/db/lsmkv/bucket_snapshot.go:165

		return "", fmt.Errorf("pause compaction: %w", err)
	}
	defer b.resumeCompaction(ctx)

	// Pause the flush cycle so no concurrent FlushAndSwitch can produce new
	// segment files while we enumerate and hard-link. Deactivate waits for
	// any in-progress flush to complete before returning.
	if err := b.flushCallbackCtrl.Deactivate(ctx); err != nil {
		return "", fmt.Errorf("pause flush cycle: %w", err)
	}
	defer b.flushCallbackCtrl.Activate()

	if err := b.FlushMemtable(); err != nil {
		return "", fmt.Errorf("flush memtable: %w", err)
	}

	if err := snapshotBucketFiles(b.disk.dir, snapshotDir, false); err != nil {
		os.RemoveAll(snapshotDir)
		return "", fmt.Errorf("hardlink snapshot: %w", err)
	}

	return snapshotDir, nil
}

// snapshotBucketFiles copies bucket files from srcDir into dstDir.
// Immutable files (segments, bloom filters, count-net-additions) are
// hard-linked for efficiency. WAL files are copied because they are
// mutable — if the shard loads after the snapshot is taken, the original
// WAL could be modified, which would corrupt the hard-linked snapshot.
//
// The caller must ensure no concurrent compaction or flush is modifying
// srcDir — for loaded buckets this means pausing both cycles first; for
// unloaded buckets this is inherently safe because no cycles are running.
//
// When includeWAL is false, .wal files are skipped. This is appropriate when
// snapshotting a loaded bucket that has just been flushed (the WAL belongs to
// the new empty memtable). When includeWAL is true, .wal files are copied.

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Place snapshotsRoot on the same filesystem as the bucket directory — hard links cannot cross devices.
2. Check the wrapped error for EXDEV and move the snapshot root if so.
3. Verify read/write permissions on both bucket dir and snapshots root.
4. Ensure the bucket isn't deleted/compacted away mid-snapshot; retry after fixing storage conditions.

Example fix

// before
snapshotDir := "/mnt/backup-volume/snapshots" // different FS than data dir
bucket.CreateSnapshot(ctx, snapshotDir, name) // hardlink snapshot: EXDEV
// after
snapshotDir := filepath.Join(dataRootParentSameFS, "snapshots")
bucket.CreateSnapshot(ctx, snapshotDir, name)

func sameFilesystem(a, b string) bool {
  sa, ea := syscall.Stat(a, &syscall.Stat_t{})
  sb, eb := syscall.Stat(b, &syscall.Stat_t{})
  return ea == nil && eb == nil && sa.Dev == sb.Dev
}
if !sameFilesystem(bucketDir, snapshotsRoot) {
  return errors.New("snapshots root must be on the same filesystem as data (hard links)")
}

func sameFilesystem(a, b string) bool {
  var sa, sb syscall.Stat_t
  if syscall.Stat(a, &sa) != nil || syscall.Stat(b, &sb) != nil { return false }
  return sa.Dev == sb.Dev
}

if _, err := bucket.CreateSnapshot(ctx, root, name); err != nil {
  if strings.Contains(err.Error(), "hardlink snapshot") {
    if errors.Is(err, syscall.EXDEV) || strings.Contains(err.Error(), "invalid argument") {
      moveSnapshotRootToSameFS()
    }
  }
}

When it happens

Trigger: snapshotBucketFiles fails because a source file disappeared mid-enumeration, the snapshots root is on a different filesystem (hard-link returns EXDEV), or permission/IO errors creating links; snapshotsRoot misconfigured.

Common situations: Backup target volume mounted on a different device than the data directory (EXDEV is the classic case); racing bucket deletion; permissions mismatch between data dir and snapshot root.

Related errors

• create snapshot for shard %s: %w
• hardlink unloaded shard %s: %w
• %q:%w
• flush memtable: %w
• open file: %w

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/2d0baffc49f752bc. Report an issue: GitHub.