{"record":{"id":"92ee95571fcb03b4","repo":"weaviate/weaviate","slug":"restore-hnsw-index-q","errorCode":null,"errorMessage":"restore hnsw index %q","messagePattern":"restore hnsw index %q","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"adapters/repos/db/vector/hnsw/startup.go","lineNumber":39,"sourceCode":"\t\"github.com/sirupsen/logrus\"\n\n\t\"github.com/weaviate/weaviate/adapters/repos/db/helpers\"\n\t\"github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers\"\n\t\"github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/compact\"\n\t\"github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/visited\"\n\t\"github.com/weaviate/weaviate/entities/cyclemanager\"\n\tenterrors \"github.com/weaviate/weaviate/entities/errors\"\n\t\"github.com/weaviate/weaviate/entities/storobj\"\n\tent \"github.com/weaviate/weaviate/entities/vectorindex/hnsw\"\n)\n\nfunc (h *hnsw) init(cfg Config) error {\n\th.pools = newPools(h.maximumConnectionsLayerZero, h.visitedListPoolMaxSize)\n\n\t// Restore from disk. This loads existing snapshot/sorted/raw files into\n\t// in-memory state.\n\tif err := h.restoreFromDisk(); err != nil {\n\t\treturn errors.Wrapf(err, \"restore hnsw index %q\", cfg.ID)\n\t}\n\n\t// Create commit logger for future writes. The logger unconditionally\n\t// creates a new raw file and never appends to an existing one — see\n\t// createNewCommitFile's comment for why.\n\tcl, err := cfg.MakeCommitLoggerThunk()\n\tif err != nil {\n\t\treturn errors.Wrap(err, \"create commit logger\")\n\t}\n\th.commitLog = cl\n\n\t// report the vector_index_size at server startup.\n\t// otherwise on server restart, prometheus reports\n\t// a vector_index_size of 0 until more vectors are\n\t// added.\n\th.metrics.SetSize(len(h.nodes))\n\n\treturn nil","sourceCodeStart":21,"sourceCodeEnd":57,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/vector/hnsw/startup.go#L21-L57","documentation":"init() wraps restoreFromDisk failures with the index ID. restoreFromDisk loads HNSW state from commit-log files (snapshot/sorted/raw) via the compact loader. A truncated/corrupt WAL is logged but tolerated; hard failures (unreadable files, bad permissions, incompatible formats) abort index startup.","triggerScenarios":"Node restart with unreadable commit-log directory: wrong file permissions, missing directory creation failure, corrupt/truncated commit log beyond the tolerated prefix, or disk-level read errors.","commonSituations":"Disk-full during a previous flush leaving truncated WAL; copying shard files with wrong ownership; restoring a backup of only some commit-log files; version downgrade incompatible with newer commit-log format.","solutions":["Check file permissions and disk health for the shard's commit-log directory","Restore the shard from a full backup if commit-log files are missing or incompatible","Delete the commit-log directory only if the index will be fully rebuilt (data loss for graph structure; vectors must be re-imported)","Confirm no version mismatch between the weaviate binary that wrote the logs and the one reading them","Free disk space and restart so flushes can complete cleanly"],"exampleFix":"// before: partially copied shard directory\nls /var/lib/weaviate/<shard>/... # missing hnsw commit logs\n// after: restore full shard from backup, then restart\nweaviate-backup restore --backup-id my-backup\ncd /var/lib/weaviate && chown -R weaviate:weaviate .","handlingStrategy":"validation","validationCode":"// pre-start check (operator script)\ndir=/var/lib/weaviate/<shard-dir>\ntest -r \"$dir\" && test -w \"$dir\" || { echo \"fix permissions on $dir\"; exit 1; }\ndf -h \"$(dirname \"$dir\")\" | awk 'NR==2 && $5+0 > 90 {print \"disk nearly full\"; exit 1}'","typeGuard":null,"tryCatchPattern":"if err := h.restoreFromDisk(); err != nil {\n  h.logger.WithField(\"index\", cfg.ID).Errorf(\"restore failed: %v\", err)\n  return errors.Wrapf(err, \"restore hnsw index %q\", cfg.ID)\n}","preventionTips":["Verify full shard backups (commit logs + segments) before upgrade","Keep ≥20% disk headroom on the data volume","Restore shard files with correct ownership (not root)","Never downgrade across commit-log format versions without export/import","Use graceful shutdown so WAL truncation stays within the tolerated prefix"],"tags":["go","hnsw","startup","commit-log","restore"],"backgroundTag":"commit-log-corrupt","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}