{"record":{"id":"28c20e62bfd46b78","repo":"weaviate/weaviate","slug":"flush-vector-index-commitlog-of-vector-q-w","errorCode":null,"errorMessage":"flush vector index commitlog of vector %q: %w","messagePattern":"flush vector index commitlog of vector %q: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"adapters/repos/db/shard_shutdown.go","lineNumber":362,"sourceCode":"\t\t\tec.Add(fmt.Errorf(\"shut down geo index queue of prop %q: %w\", propName, err))\n\t\t}\n\n\t\treturn nil\n\t})\n\n\ts.propertyIndicesLock.RLock()\n\terr = s.propertyIndices.ShutdownGeoIndices(ctx)\n\ts.propertyIndicesLock.RUnlock()\n\tec.AddWrapf(err, \"shutdown geo property indices\")\n\n\t_ = s.ForEachVectorIndex(func(targetVector string, index VectorIndex) error {\n\t\t// to ensure that all commitlog entries are written to disk.\n\t\t// otherwise in some cases the tombstone cleanup process'\n\t\t// 'RemoveTombstone' entry is not picked up on restarts\n\t\t// resulting in perpetually attempting to remove a tombstone\n\t\t// which doesn't actually exist anymore\n\t\tif err = index.Flush(); err != nil {\n\t\t\tec.Add(fmt.Errorf(\"flush vector index commitlog of vector %q: %w\", targetVector, err))\n\t\t}\n\n\t\tif err = index.Shutdown(ctx); err != nil {\n\t\t\tec.Add(fmt.Errorf(\"shut down vector index of vector %q: %w\", targetVector, err))\n\t\t}\n\n\t\treturn nil\n\t})\n\n\tstoreDurable := false\n\tif s.store != nil {\n\t\ts.UpdateStatus(storagestate.StatusShutdown.String(), statusReasonShutdown)\n\n\t\t// store would be nil if loading the objects bucket failed, as we would\n\t\t// only return the store on success from s.initLSMStore()\n\t\terr = s.store.Shutdown(ctx)\n\t\tec.AddWrapf(err, \"stop lsmkv store\")\n\t\tstoreDurable = err == nil","sourceCodeStart":344,"sourceCodeEnd":380,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/shard_shutdown.go#L344-L380","documentation":"During teardown, each vector index's Flush() (persisting the HNSW commitlog itself, distinct from the queue flush) can fail and is recorded as \"flush vector index commitlog of vector %q: %w\". This flush is required so tombstone cleanup RemoveTombstone entries survive restarts; failure can cause perpetual attempts to remove already-gone tombstones after recovery.","triggerScenarios":"index.Flush() errors while iterating the shard's vector indexes during performShutdown for the given targetVector — commitlog write/fsync failure, disk full, or corrupted commitlog segment.","commonSituations":"Disk-full at shutdown; failing disk or network storage; unclean prior crash corrupting the HNSW commitlog; process killed mid-flush.","solutions":["Fix disk space/I-O issues and restart; the node replays the commitlog on startup.","Watch for repeated tombstone-cleanup warnings after restart — a sign this flush was skipped; trigger re-cleanup or re-index.","Restore the affected collection from backup if the commitlog is corrupted.","Check for preceding queue flush failures (2683) pointing at the same vector."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := shard.Shutdown(ctx); err != nil {\n    if strings.Contains(err.Error(), \"flush vector index commitlog\") {\n        log.Printf(\"HNSW commitlog flush failed: %v — fix disk and restart; watch tombstone cleanup after\", err)\n    }\n}","preventionTips":["Ensure free disk space and healthy I/O before shutdown.","Never SIGKILL the process during shutdown; let flush complete.","After restart, watch for perpetual tombstone-removal warnings indicating a skipped flush.","Maintain backups so corrupted commitlogs can be restored and re-indexed."],"tags":["shutdown","hnsw","commitlog","tombstone","go"],"backgroundTag":"commitlog-flush-failed","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}