{"record":{"id":"5d7df8b8aceffc54","repo":"wavetermdev/waveterm","slug":"too-many-flush-errors-clearing-entry-w","errorCode":null,"errorMessage":"too many flush errors (clearing entry): %w","messagePattern":"too many flush errors \\(clearing entry\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/filestore/blockstore_cache.go","lineNumber":343,"sourceCode":"\t\tFlushErrors: 0,\n\t}\n}\n\nfunc (entry *CacheEntry) flushToDB(ctx context.Context, replace bool) error {\n\tif entry.File == nil {\n\t\treturn nil\n\t}\n\terr := dbWriteCacheEntry(ctx, entry.File, entry.DataEntries, replace)\n\tif ctx.Err() != nil {\n\t\t// transient error\n\t\treturn ctx.Err()\n\t}\n\tif err != nil {\n\t\tflushErrorCount.Add(1)\n\t\tentry.FlushErrors++\n\t\tif entry.FlushErrors > 3 {\n\t\t\tentry.clear()\n\t\t\treturn fmt.Errorf(\"too many flush errors (clearing entry): %w\", err)\n\t\t}\n\t\treturn err\n\t}\n\t// clear cache entry (data is now in db)\n\tentry.clear()\n\treturn nil\n}\n","sourceCodeStart":325,"sourceCodeEnd":351,"githubUrl":"https://github.com/wavetermdev/waveterm/blob/a4447c1563b2df285ab89e76c82f91e1a1a49c1e/pkg/filestore/blockstore_cache.go#L325-L351","documentation":"flushToDB writes dirty cached data parts back to the SQLite filestore db via dbWriteCacheEntry. If the write fails, the error counter is incremented; once an entry has accumulated more than 3 flush errors the library gives up, clears (discards) the entire in-memory cache entry, and returns this wrapped error. The wrap means unflushed cached data was lost for that file — the library deliberately drops the entry rather than retrying indefinitely.","triggerScenarios":"dbWriteCacheEntry fails repeatedly (4+ times) for the same CacheEntry — persistent SQLite write failure such as disk full, db locked, I/O error, or corrupted db — while flushing dirty data parts. Note: if ctx itself is cancelled, the raw ctx.Err() is returned instead (treated as transient).","commonSituations":"Disk full on the machine hosting WAVE_DATA_DIR; SQLite WAL growth under sustained lock contention; a corrupted filestore db; long-running offline connection cache filling up while the db is unavailable.","solutions":["Check the wrapped cause: if 'database is locked' or 'disk I/O error', free disk space / resolve db contention before more writes.","Assume the affected cached file data was discarded — re-read or re-write the file to repopulate cache and db.","Check flushErrorCount metric and logs to identify whether flush failures are clustered on one zone/file or global.","Run PRAGMA integrity_check on the filestore sqlite db; restore from backup or delete to recreate if corrupted.","If writes are consistently timing out, increase _busy_timeout in MakeDB (blockstore_dbsetup.go:67) or reduce concurrent flush pressure."],"exampleFix":"// before\nif err := fs.FlushCacheEntry(ctx, zoneId, name); err != nil {\n    return err\n}\n// after\nif err := fs.FlushCacheEntry(ctx, zoneId, name); err != nil {\n    if strings.Contains(err.Error(), \"too many flush errors\") {\n        log.Printf(\"cache entry for %s/%s discarded after repeated flush failures: %v\", zoneId, name, err)\n        // data was dropped from cache; rewrite to persist\n        return fs.WriteFile(ctx, zoneId, name, data)\n    }\n    return err\n}","handlingStrategy":"try-catch","validationCode":"var free uint64\nif st, err := syscall.Statfs(wavebase.GetWaveDataDir(), &st); ... // check disk space before heavy writes\nif st.Bavail*uint64(st.Bsize) < 100*1024*1024 {\n    return fmt.Errorf(\"low disk space in data dir\")\n}","typeGuard":null,"tryCatchPattern":"err := entry.flushToDB(ctx, false)\nif err != nil && strings.Contains(err.Error(), \"too many flush errors\") {\n    log.Printf(\"cached data for %s/%s was DISCARDED: %v\", entry.ZoneId, entry.Name, errors.Unwrap(err))\n    // treat the file as dirty: re-read/rewrite to restore\n} else if err != nil {\n    // transient: entry still cached, safe to retry later\n}","preventionTips":["Monitor disk space in WAVE_DATA_DIR","Watch flushErrorCount and alert before the 4-error threshold drops data","Avoid external processes locking the sqlite db","Back up the filestore db periodically","Fix underlying db errors (lock/IO) quickly — each failed flush brings the entry closer to being cleared"],"tags":["sqlite","flush","data-loss","cache","disk"],"backgroundTag":"sqlite-disk-io-error","analyzedSha":"a4447c1563b2df285ab89e76c82f91e1a1a49c1e","analyzedAt":"2026-09-01T15:26:23.972Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}