weaviate/weaviate · error

failed to create hfresh merge queue

Error message

failed to create hfresh merge queue

What it means

NewTaskQueue creates the disk-backed merge queue under merge.queue.d for hfresh posting-merge operations. This wrapper fires when queue.NewDiskQueue fails (directory/I/O error), preventing merges from being scheduled and aborting index initialization.

Source

Thrown at adapters/repos/db/vector/hfresh/task_queue.go:150

		return nil, errors.Wrap(err, "failed to initialize hfresh reassign queue")
	}

	tq.analyzeQueue = &analyzeQueue{DiskQueue: aq, reassignQueue: tq.reassignQueue}

	// create queue for merge operations
	tq.mergeQueue, err = queue.NewDiskQueue(
		queue.DiskQueueOptions{
			ID:               fmt.Sprintf("hfresh_merge_queue_%s_%s", index.config.ShardName, index.config.ID),
			Logger:           index.logger,
			Scheduler:        index.scheduler,
			Dir:              filepath.Join(index.config.RootPath, "merge.queue.d"),
			TaskDecoder:      &tq,
			OnBatchProcessed: tq.OnBatchProcessed,
			ChunkSize:        mergeTaskQueueChunkSize,
		},
	)
	if err != nil {
		return nil, errors.Wrap(err, "failed to create hfresh merge queue")
	}
	err = tq.mergeQueue.Init()
	if err != nil {
		return nil, errors.Wrap(err, "failed to initialize hfresh merge queue")
	}

	return &tq, nil
}

func (tq *TaskQueue) Register() {
	tq.scheduler.RegisterQueue(tq.analyzeQueue)
	tq.scheduler.RegisterQueue(tq.splitQueue)
	tq.scheduler.RegisterQueue(tq.reassignQueue)
	tq.scheduler.RegisterQueue(tq.mergeQueue)
}

func (tq *TaskQueue) Close(ctx context.Context) error {
	var errs []error

View on GitHub (pinned to 75aa4b6d11)

Solutions

1. Check permissions/disk space for the shard's merge.queue.d directory
2. Restart once storage is healthy

Example fix

// before
DiskQueueOptions{ID: "", ChunkSize: 0} // invalid in custom harness
// after
DiskQueueOptions{ID: "hfresh_merge_queue_<shard>_<id>", ChunkSize: mergeTaskQueueChunkSize, Logger: logger, Dir: filepath.Join(root, "merge.queue.d")}

mergeDir := filepath.Join(rootPath, "merge.queue.d")
if err := os.MkdirAll(mergeDir, 0o755); err != nil {
    return fmt.Errorf("cannot prepare merge queue dir %s: %v", mergeDir, err)
}
// validate options if constructing manually
if opts.ID == "" || opts.ChunkSize <= 0 || opts.TaskDecoder == nil { return errors.New("invalid DiskQueueOptions") }

if err != nil && strings.Contains(err.Error(), "failed to create hfresh merge queue") {
    return fmt.Errorf("merge queue creation failed at %s: %w", filepath.Join(rootPath, "merge.queue.d"), err)
}

When it happens

Trigger: hfresh.New / NewTaskQueue where NewDiskQueue for the merge queue errors — unusable directory path, invalid DiskQueueOptions, or filesystem failure.

Common situations: Unwritable or missing RootPath, read-only filesystem, disk-full conditions, or custom/test code passing invalid DiskQueueOptions (empty ID, zero chunk size, nil decoder).

Related errors

• failed to create hfresh analyze queue
• failed to create hfresh split queue
• failed to create hfresh reassign queue
• failed to initialize hfresh analyze queue
• failed to initialize hfresh split queue

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/1e1bf78808731d0d. Report an issue: GitHub.