weaviate/weaviate · error

failed to copy from old compressed vector bucket to new buck

Error message

failed to copy from old compressed vector bucket to new bucket for target vector: %s: %w

What it means

The compressed-vectors migrator found a legacy vectors_compressed bucket and multiple vectors (2+), so it copies the old bucket to each named target-vector bucket via migrate(..., renameBucket=false). This error wraps any failure inside that copy path: removing a stale *_tmp directory, walking/copying files (copyBucketContents/copyFile), or renaming the tmp directory into place. The shard open is aborted to avoid a half-copied compressed index serving queries.

Source

Thrown at adapters/repos/db/shard_compressed_vectors_migrator.go:76

		case 0:
			// do nothing
		case 1:
			if len(targets) > 0 {
				for targetVector, vectorIndexConfig := range targets {
					// rename old bucket to new target vector bucket
					if err := m.migrate(targetVector, vectorIndexConfig, lsmDir, true); err != nil {
						return fmt.Errorf("failed to rename old compressed vector bucket for target vector %s: %w", targetVector, err)
					}
				}
			}
			if err := m.markMigrationDone(lsmDir); err != nil {
				return fmt.Errorf("failed to mark migration as done: %w", err)
			}
		default:
			// copy old buckets to new target vector buckets
			for targetVector, vectorIndexConfig := range targets {
				if err := m.migrate(targetVector, vectorIndexConfig, lsmDir, false); err != nil {
					return fmt.Errorf("failed to copy from old compressed vector bucket to new bucket for target vector: %s: %w", targetVector, err)
				}
			}
			if legacy == nil {
				// remove the old bucket directory after all copies are complete, only if was not defined for legacy vector
				if err := os.RemoveAll(vectorsCompressedPath); err != nil {
					return fmt.Errorf("failed to remove old bucket directory after copying all target vectors: %w", err)
				}
				m.logger.Info("removed old vectors compressed bucket")
			} else {
				// legacy vector defined together with named vectors, we need to mark that the migration was performed
				if err := m.markMigrationDone(lsmDir); err != nil {
					return fmt.Errorf("failed to mark migration as done: %w", err)
				}
			}
		}
	} else {
		if legacy != nil && m.isQuantizationEnabled(legacy) {
			// a new legacy vector config was created, quantization is enabled but we didn't create the vectors_compressed

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Free disk space: the copy duplicates the whole vectors_compressed bucket, so you need roughly that much free space on the data volume.
2. Fix permissions on the shard directory so the weaviate process can create/remove <targetBucket>_tmp and read the old bucket.
3. If a partial <targetBucket>_tmp directory remains from a crashed migration, delete it (it is safe: it is a temp copy) and restart — migrator also removes it, but only if it can.
4. Check logs for the wrapped inner error to distinguish ENOSPC vs EACCES vs rename failure and act accordingly; then restart to re-run the idempotent migration.

Example fix

// before (log line)
// failed to copy from old compressed vector bucket to new bucket for target vector: first_vec: 
//   open /var/lib/weaviate/.../vectors_compressed/lsm/segments/...: permission denied

// after
// chown -R 1000:1000 /var/lib/weaviate
// rm -rf /var/lib/weaviate/.../vectors_compressed_first_vec_tmp   # stale temp from prior crash
// restart weaviate

# before upgrading a multi-named-vector collection with quantization
DATA_DIR=${PERSISTENCE_DATA_PATH:-/var/lib/weaviate}
# the copy duplicates the whole compressed bucket — require free space >= bucket size
BUCKET=$(find "$DATA_DIR" -type d -name vectors_compressed | head -1)
NEEDED=$(du -sk "$BUCKET" 2>/dev/null | cut -f1)
FREE=$(df --output=avail -k "$DATA_DIR" | tail -1)
[ "$FREE" -ge "${NEEDED:-0}" ] || { echo "insufficient space for migration copy"; exit 1; }

// Go: unwrap to find the concrete OS failure before retrying
if err := shard.Open(); err != nil {
    if strings.Contains(err.Error(), "failed to copy from old compressed vector bucket") {
        if errors.Is(err, syscall.ENOSPC) || strings.Contains(err.Error(), "no space left") {
            log.Fatalf("free disk space on %s and restart: %v", dataPath, err)
        }
    }
    return err
}

When it happens

Trigger: Opening a shard whose collection has 2+ vectors (e.g. legacy + named, or multiple named vectors), the vectors_compressed directory exists, and migrate() fails: ENOSPC during the copy (full duplication of bucket files), EACCES reading old bucket or writing the tmp bucket, stale <bucket>_tmp dir that cannot be removed, or os.Rename(tmp, final) failing (e.g. target on another filesystem after an overlay mount).

Common situations: Upgrading a multi-named-vector collection with quantization from the pre-named-vectors layout; disk full because the copy duplicates the entire compressed bucket; backups restored with wrong ownership; target bucket name colliding with a leftover directory from an earlier interrupted migration.

Related errors

• failed to rename old compressed vector bucket for target vec
• failed to mark migration as done: %w
• failed to remove old bucket directory after copying all targ
• failed saving migration state
• migrate snapshot directory

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/4c9c570760ca4eac. Report an issue: GitHub.