{"record":{"id":"4c9c570760ca4eac","repo":"weaviate/weaviate","slug":"failed-to-copy-from-old-compressed-vector-bucket-t","errorCode":null,"errorMessage":"failed to copy from old compressed vector bucket to new bucket for target vector: %s: %w","messagePattern":"failed to copy from old compressed vector bucket to new bucket for target vector: (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"adapters/repos/db/shard_compressed_vectors_migrator.go","lineNumber":76,"sourceCode":"\t\tcase 0:\n\t\t\t// do nothing\n\t\tcase 1:\n\t\t\tif len(targets) > 0 {\n\t\t\t\tfor targetVector, vectorIndexConfig := range targets {\n\t\t\t\t\t// rename old bucket to new target vector bucket\n\t\t\t\t\tif err := m.migrate(targetVector, vectorIndexConfig, lsmDir, true); err != nil {\n\t\t\t\t\t\treturn fmt.Errorf(\"failed to rename old compressed vector bucket for target vector %s: %w\", targetVector, err)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tif err := m.markMigrationDone(lsmDir); err != nil {\n\t\t\t\treturn fmt.Errorf(\"failed to mark migration as done: %w\", err)\n\t\t\t}\n\t\tdefault:\n\t\t\t// copy old buckets to new target vector buckets\n\t\t\tfor targetVector, vectorIndexConfig := range targets {\n\t\t\t\tif err := m.migrate(targetVector, vectorIndexConfig, lsmDir, false); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to copy from old compressed vector bucket to new bucket for target vector: %s: %w\", targetVector, err)\n\t\t\t\t}\n\t\t\t}\n\t\t\tif legacy == nil {\n\t\t\t\t// remove the old bucket directory after all copies are complete, only if was not defined for legacy vector\n\t\t\t\tif err := os.RemoveAll(vectorsCompressedPath); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to remove old bucket directory after copying all target vectors: %w\", err)\n\t\t\t\t}\n\t\t\t\tm.logger.Info(\"removed old vectors compressed bucket\")\n\t\t\t} else {\n\t\t\t\t// legacy vector defined together with named vectors, we need to mark that the migration was performed\n\t\t\t\tif err := m.markMigrationDone(lsmDir); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to mark migration as done: %w\", err)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t} else {\n\t\tif legacy != nil && m.isQuantizationEnabled(legacy) {\n\t\t\t// a new legacy vector config was created, quantization is enabled but we didn't create the vectors_compressed","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/shard_compressed_vectors_migrator.go#L58-L94","documentation":"The compressed-vectors migrator found a legacy vectors_compressed bucket and multiple vectors (2+), so it copies the old bucket to each named target-vector bucket via migrate(..., renameBucket=false). This error wraps any failure inside that copy path: removing a stale *_tmp directory, walking/copying files (copyBucketContents/copyFile), or renaming the tmp directory into place. The shard open is aborted to avoid a half-copied compressed index serving queries.","triggerScenarios":"Opening a shard whose collection has 2+ vectors (e.g. legacy + named, or multiple named vectors), the vectors_compressed directory exists, and migrate() fails: ENOSPC during the copy (full duplication of bucket files), EACCES reading old bucket or writing the tmp bucket, stale <bucket>_tmp dir that cannot be removed, or os.Rename(tmp, final) failing (e.g. target on another filesystem after an overlay mount).","commonSituations":"Upgrading a multi-named-vector collection with quantization from the pre-named-vectors layout; disk full because the copy duplicates the entire compressed bucket; backups restored with wrong ownership; target bucket name colliding with a leftover directory from an earlier interrupted migration.","solutions":["Free disk space: the copy duplicates the whole vectors_compressed bucket, so you need roughly that much free space on the data volume.","Fix permissions on the shard directory so the weaviate process can create/remove <targetBucket>_tmp and read the old bucket.","If a partial <targetBucket>_tmp directory remains from a crashed migration, delete it (it is safe: it is a temp copy) and restart — migrator also removes it, but only if it can.","Check logs for the wrapped inner error to distinguish ENOSPC vs EACCES vs rename failure and act accordingly; then restart to re-run the idempotent migration."],"exampleFix":"// before (log line)\n// failed to copy from old compressed vector bucket to new bucket for target vector: first_vec: \n//   open /var/lib/weaviate/.../vectors_compressed/lsm/segments/...: permission denied\n\n// after\n// chown -R 1000:1000 /var/lib/weaviate\n// rm -rf /var/lib/weaviate/.../vectors_compressed_first_vec_tmp   # stale temp from prior crash\n// restart weaviate","handlingStrategy":"validation","validationCode":"# before upgrading a multi-named-vector collection with quantization\nDATA_DIR=${PERSISTENCE_DATA_PATH:-/var/lib/weaviate}\n# the copy duplicates the whole compressed bucket — require free space >= bucket size\nBUCKET=$(find \"$DATA_DIR\" -type d -name vectors_compressed | head -1)\nNEEDED=$(du -sk \"$BUCKET\" 2>/dev/null | cut -f1)\nFREE=$(df --output=avail -k \"$DATA_DIR\" | tail -1)\n[ \"$FREE\" -ge \"${NEEDED:-0}\" ] || { echo \"insufficient space for migration copy\"; exit 1; }","typeGuard":null,"tryCatchPattern":"// Go: unwrap to find the concrete OS failure before retrying\nif err := shard.Open(); err != nil {\n    if strings.Contains(err.Error(), \"failed to copy from old compressed vector bucket\") {\n        if errors.Is(err, syscall.ENOSPC) || strings.Contains(err.Error(), \"no space left\") {\n            log.Fatalf(\"free disk space on %s and restart: %v\", dataPath, err)\n        }\n    }\n    return err\n}","preventionTips":["Keep free disk space at least the size of your largest vectors_compressed bucket before upgrading.","Delete stale <bucket>_tmp directories left by previously interrupted migrations.","Monitor shard-open logs during version upgrades with quantization enabled.","Test the upgrade on a staging copy of the data directory first."],"tags":["filesystem","migration","copy","disk-space","startup"],"backgroundTag":"disk-full-io-error","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}