What causes "long-running memtable flush in progress" in weaviate/weaviate?

Retry with a longer context timeout — the doc comment says failing and retrying later is preferable to blocking indefinitely

weaviate/weaviate · warning

long-running memtable flush in progress

Error message

long-running memtable flush in progress

What it means

Store.FlushMemtables deactivates the flush callback controller, flushes every bucket's active memtable to disk, then reactivates flushing. If the initial Deactivate fails (ctx deadline/cancellation because a flush was still running), the error is wrapped with this message. It signals the flush-pause timed out, not that data was lost.

Source

Thrown at adapters/repos/db/lsmkv/store_backup.go:76

	for _, b := range s.bucketsByName {
		b.doStopPauseTimer()
	}

	return nil
}

// FlushMemtable flushes any active memtable and returns only once the memtable
// has been fully flushed and a stable state on disk has been reached.
//
// This is a preparatory stage for creating backups.
//
// A timeout should be specified for the input context as some
// flushes are long-running, in which case it may be better
// to fail the backup attempt and retry later, than to block
// indefinitely.
func (s *Store) FlushMemtables(ctx context.Context) error {
	if err := s.cycleCallbacks.flushCallbacksCtrl.Deactivate(ctx); err != nil {
		return errors.Wrap(err, "long-running memtable flush in progress")
	}
	defer s.cycleCallbacks.flushCallbacksCtrl.Activate()

	flushMemtable := func(ctx context.Context, b *Bucket) (interface{}, error) {
		return nil, b.FlushMemtable()
	}
	_, err := s.runJobOnBuckets(ctx, flushMemtable, nil)
	return err
}

View on GitHub (pinned to 75aa4b6d11)

Solutions

Retry with a longer context timeout — the doc comment says failing and retrying later is preferable to blocking indefinitely
Wait for the in-flight flush to complete (check flush metrics/logs) before retrying
Reduce memtable size or flush interval pressure before maintenance windows
If flushes repeatedly exceed timeouts, investigate disk throughput (IOPS/latency) on the data volume

Example fix

// before
ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
if err := store.FlushMemtables(ctx); err != nil { return err }
// after: allow long flushes to finish
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
if err := store.FlushMemtables(ctx); err != nil {
	logger.Warnf("flush pause failed, will retry: %v", err)
	time.Sleep(time.Minute)
	return store.FlushMemtables(ctx)
}

Defensive patterns

Strategy: retry

Validate before calling

// Go: avoid pausing while a flush is obviously pending (queue depth/size checks)
// large pending memtable => wait before calling FlushMemtables

Try / catch

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()
if err := store.FlushMemtables(ctx); err != nil {
	time.Sleep(30 * time.Second)
	return store.FlushMemtables(ctx) // retry after in-flight flush completes
}

Prevention

Pass explicit long timeouts; the doc comment mandates a timeout on ctx
Don't call during write bursts; drain traffic first
Monitor memtable size/flush duration on the target disks
Use HaltForTransfer-style flows with retry rather than one-shot calls

When it happens

Trigger: FlushMemtables (via HaltForTransfer or tests) called while a memtable flush is in progress and the context deadline expires before flushCallbacksCtrl.Deactivate returns.

Common situations: Backup before node shutdown during write-heavy load with a large memtable mid-flush; shard transfer with short timeout; slow disks making flushes exceed the caller's timeout.

Related errors

AI-assisted analysis of weaviate/weaviate@75aa4b6d11 (2026-09-04). Data as JSON: /api/errors/48f20f4064e3a6a5. Report an issue: GitHub.