{"record":{"id":"48f20f4064e3a6a5","repo":"weaviate/weaviate","slug":"long-running-memtable-flush-in-progress","errorCode":null,"errorMessage":"long-running memtable flush in progress","messagePattern":"long-running memtable flush in progress","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"adapters/repos/db/lsmkv/store_backup.go","lineNumber":76,"sourceCode":"\tfor _, b := range s.bucketsByName {\n\t\tb.doStopPauseTimer()\n\t}\n\n\treturn nil\n}\n\n// FlushMemtable flushes any active memtable and returns only once the memtable\n// has been fully flushed and a stable state on disk has been reached.\n//\n// This is a preparatory stage for creating backups.\n//\n// A timeout should be specified for the input context as some\n// flushes are long-running, in which case it may be better\n// to fail the backup attempt and retry later, than to block\n// indefinitely.\nfunc (s *Store) FlushMemtables(ctx context.Context) error {\n\tif err := s.cycleCallbacks.flushCallbacksCtrl.Deactivate(ctx); err != nil {\n\t\treturn errors.Wrap(err, \"long-running memtable flush in progress\")\n\t}\n\tdefer s.cycleCallbacks.flushCallbacksCtrl.Activate()\n\n\tflushMemtable := func(ctx context.Context, b *Bucket) (interface{}, error) {\n\t\treturn nil, b.FlushMemtable()\n\t}\n\t_, err := s.runJobOnBuckets(ctx, flushMemtable, nil)\n\treturn err\n}\n","sourceCodeStart":58,"sourceCodeEnd":86,"githubUrl":"https://github.com/weaviate/weaviate/blob/75aa4b6d11f8818305aafd4440b4e32794f7ca04/adapters/repos/db/lsmkv/store_backup.go#L58-L86","documentation":"Store.FlushMemtables deactivates the flush callback controller, flushes every bucket's active memtable to disk, then reactivates flushing. If the initial Deactivate fails (ctx deadline/cancellation because a flush was still running), the error is wrapped with this message. It signals the flush-pause timed out, not that data was lost.","triggerScenarios":"FlushMemtables (via HaltForTransfer or tests) called while a memtable flush is in progress and the context deadline expires before flushCallbacksCtrl.Deactivate returns.","commonSituations":"Backup before node shutdown during write-heavy load with a large memtable mid-flush; shard transfer with short timeout; slow disks making flushes exceed the caller's timeout.","solutions":["Retry with a longer context timeout — the doc comment says failing and retrying later is preferable to blocking indefinitely","Wait for the in-flight flush to complete (check flush metrics/logs) before retrying","Reduce memtable size or flush interval pressure before maintenance windows","If flushes repeatedly exceed timeouts, investigate disk throughput (IOPS/latency) on the data volume"],"exampleFix":"// before\nctx, cancel := context.WithTimeout(ctx, 5*time.Second)\nif err := store.FlushMemtables(ctx); err != nil { return err }\n// after: allow long flushes to finish\nctx, cancel := context.WithTimeout(ctx, 5*time.Minute)\nif err := store.FlushMemtables(ctx); err != nil {\n\tlogger.Warnf(\"flush pause failed, will retry: %v\", err)\n\ttime.Sleep(time.Minute)\n\treturn store.FlushMemtables(ctx)\n}","handlingStrategy":"retry","validationCode":"// Go: avoid pausing while a flush is obviously pending (queue depth/size checks)\n// large pending memtable => wait before calling FlushMemtables","typeGuard":null,"tryCatchPattern":"ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)\ndefer cancel()\nif err := store.FlushMemtables(ctx); err != nil {\n\ttime.Sleep(30 * time.Second)\n\treturn store.FlushMemtables(ctx) // retry after in-flight flush completes\n}","preventionTips":["Pass explicit long timeouts; the doc comment mandates a timeout on ctx","Don't call during write bursts; drain traffic first","Monitor memtable size/flush duration on the target disks","Use HaltForTransfer-style flows with retry rather than one-shot calls"],"tags":["lsmkv","flush","memtable","timeout"],"backgroundTag":"compaction-pause-timeout","analyzedSha":"75aa4b6d11f8818305aafd4440b4e32794f7ca04","analyzedAt":"2026-09-04T14:58:20.392Z","contentChangedAt":"2026-09-04T14:58:20.392Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}