weaviate/weaviate · critical

flush vector index commitlog of vector %q: %w

Error message

flush vector index commitlog of vector %q: %w

What it means

During teardown, each vector index's Flush() (persisting the HNSW commitlog itself, distinct from the queue flush) can fail and is recorded as "flush vector index commitlog of vector %q: %w". This flush is required so tombstone cleanup RemoveTombstone entries survive restarts; failure can cause perpetual attempts to remove already-gone tombstones after recovery.

Source

Thrown at adapters/repos/db/shard_shutdown.go:362

			ec.Add(fmt.Errorf("shut down geo index queue of prop %q: %w", propName, err))
		}

		return nil
	})

	s.propertyIndicesLock.RLock()
	err = s.propertyIndices.ShutdownGeoIndices(ctx)
	s.propertyIndicesLock.RUnlock()
	ec.AddWrapf(err, "shutdown geo property indices")

	_ = s.ForEachVectorIndex(func(targetVector string, index VectorIndex) error {
		// to ensure that all commitlog entries are written to disk.
		// otherwise in some cases the tombstone cleanup process'
		// 'RemoveTombstone' entry is not picked up on restarts
		// resulting in perpetually attempting to remove a tombstone
		// which doesn't actually exist anymore
		if err = index.Flush(); err != nil {
			ec.Add(fmt.Errorf("flush vector index commitlog of vector %q: %w", targetVector, err))
		}

		if err = index.Shutdown(ctx); err != nil {
			ec.Add(fmt.Errorf("shut down vector index of vector %q: %w", targetVector, err))
		}

		return nil
	})

	storeDurable := false
	if s.store != nil {
		s.UpdateStatus(storagestate.StatusShutdown.String(), statusReasonShutdown)

		// store would be nil if loading the objects bucket failed, as we would
		// only return the store on success from s.initLSMStore()
		err = s.store.Shutdown(ctx)
		ec.AddWrapf(err, "stop lsmkv store")
		storeDurable = err == nil

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Fix disk space/I-O issues and restart; the node replays the commitlog on startup.
2. Watch for repeated tombstone-cleanup warnings after restart — a sign this flush was skipped; trigger re-cleanup or re-index.
3. Restore the affected collection from backup if the commitlog is corrupted.
4. Check for preceding queue flush failures (2683) pointing at the same vector.

if err := shard.Shutdown(ctx); err != nil {
    if strings.Contains(err.Error(), "flush vector index commitlog") {
        log.Printf("HNSW commitlog flush failed: %v — fix disk and restart; watch tombstone cleanup after", err)
    }
}

When it happens

Trigger: index.Flush() errors while iterating the shard's vector indexes during performShutdown for the given targetVector — commitlog write/fsync failure, disk full, or corrupted commitlog segment.

Common situations: Disk-full at shutdown; failing disk or network storage; unclean prior crash corrupting the HNSW commitlog; process killed mid-flush.

Related errors

• flush vector index queue commitlog of vector %q: %w
• flush geo index queue commitlog of prop %q: %w
• shut down vector index of vector %q: %w
• index is not hnsw
• tombstone cleanup already running

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/28c20e62bfd46b78. Report an issue: GitHub.