weaviate/weaviate · critical

restore hnsw index %q

Error message

restore hnsw index %q

What it means

init() wraps restoreFromDisk failures with the index ID. restoreFromDisk loads HNSW state from commit-log files (snapshot/sorted/raw) via the compact loader. A truncated/corrupt WAL is logged but tolerated; hard failures (unreadable files, bad permissions, incompatible formats) abort index startup.

Source

Thrown at adapters/repos/db/vector/hnsw/startup.go:39

	"github.com/sirupsen/logrus"

	"github.com/weaviate/weaviate/adapters/repos/db/helpers"
	"github.com/weaviate/weaviate/adapters/repos/db/vector/compressionhelpers"
	"github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/compact"
	"github.com/weaviate/weaviate/adapters/repos/db/vector/hnsw/visited"
	"github.com/weaviate/weaviate/entities/cyclemanager"
	enterrors "github.com/weaviate/weaviate/entities/errors"
	"github.com/weaviate/weaviate/entities/storobj"
	ent "github.com/weaviate/weaviate/entities/vectorindex/hnsw"
)

func (h *hnsw) init(cfg Config) error {
	h.pools = newPools(h.maximumConnectionsLayerZero, h.visitedListPoolMaxSize)

	// Restore from disk. This loads existing snapshot/sorted/raw files into
	// in-memory state.
	if err := h.restoreFromDisk(); err != nil {
		return errors.Wrapf(err, "restore hnsw index %q", cfg.ID)
	}

	// Create commit logger for future writes. The logger unconditionally
	// creates a new raw file and never appends to an existing one — see
	// createNewCommitFile's comment for why.
	cl, err := cfg.MakeCommitLoggerThunk()
	if err != nil {
		return errors.Wrap(err, "create commit logger")
	}
	h.commitLog = cl

	// report the vector_index_size at server startup.
	// otherwise on server restart, prometheus reports
	// a vector_index_size of 0 until more vectors are
	// added.
	h.metrics.SetSize(len(h.nodes))

	return nil

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check file permissions and disk health for the shard's commit-log directory
2. Restore the shard from a full backup if commit-log files are missing or incompatible
3. Delete the commit-log directory only if the index will be fully rebuilt (data loss for graph structure; vectors must be re-imported)
4. Confirm no version mismatch between the weaviate binary that wrote the logs and the one reading them
5. Free disk space and restart so flushes can complete cleanly

Example fix

// before: partially copied shard directory
ls /var/lib/weaviate/<shard>/... # missing hnsw commit logs
// after: restore full shard from backup, then restart
weaviate-backup restore --backup-id my-backup
cd /var/lib/weaviate && chown -R weaviate:weaviate .

// pre-start check (operator script)
dir=/var/lib/weaviate/<shard-dir>
test -r "$dir" && test -w "$dir" || { echo "fix permissions on $dir"; exit 1; }
df -h "$(dirname "$dir")" | awk 'NR==2 && $5+0 > 90 {print "disk nearly full"; exit 1}'

if err := h.restoreFromDisk(); err != nil {
  h.logger.WithField("index", cfg.ID).Errorf("restore failed: %v", err)
  return errors.Wrapf(err, "restore hnsw index %q", cfg.ID)
}

When it happens

Trigger: Node restart with unreadable commit-log directory: wrong file permissions, missing directory creation failure, corrupt/truncated commit log beyond the tolerated prefix, or disk-level read errors.

Common situations: Disk-full during a previous flush leaving truncated WAL; copying shard files with wrong ownership; restoring a backup of only some commit-log files; version downgrade incompatible with newer commit-log format.

Related errors

• create commit logger directory
• create commit logger
• persist entrypoint: %w
• create commit log file
• browse commit logger directory

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/92ee95571fcb03b4. Report an issue: GitHub.