{"record":{"id":"8539f30ca76b8179","repo":"canopy-network/canopy","slug":"flush-before-checkpoint-w","errorCode":null,"errorMessage":"flush before checkpoint: %w","messagePattern":"flush before checkpoint: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"store/store.go","lineNumber":778,"sourceCode":"\t\t\t\trestoreErr := os.Rename(prevBackupDir, backupDir)\n\t\t\t\tif restoreErr != nil && !os.IsNotExist(restoreErr) {\n\t\t\t\t\ts.log.Errorf(\"failed to restore previous backup at height [%d]: %v\", version, restoreErr)\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\t// otherwise, remove dangling backup, continue with current working backup\n\t\t\t\t_ = os.RemoveAll(prevBackupDir)\n\t\t\t}\n\t\t\ts.backup.Store(false)\n\t\t\ts.log.Errorf(\"backup failed at height [%d]: %v\", version, err)\n\t\t}()\n\t\t// flush the memtable to SST before checkpointing so the backup does not\n\t\t// depend on WAL replay for recovery (commits use NoSync so WAL records\n\t\t// may not be durable on disk at checkpoint time)\n\t\ts.mu.Lock()\n\t\tversion = s.Version()\n\t\tif err = s.db.Flush(); err != nil {\n\t\t\ts.mu.Unlock()\n\t\t\terr = fmt.Errorf(\"flush before checkpoint: %w\", err)\n\t\t\treturn\n\t\t}\n\t\ts.mu.Unlock()\n\t\t// perform the backup using pebble's checkpointing mechanism which creates a\n\t\t// consistent snapshot of the database at the specified directory\n\t\tif err = s.db.Checkpoint(tempBackupDir); err != nil {\n\t\t\terr = fmt.Errorf(\"checkpoint creation: %w\", err)\n\t\t\treturn\n\t\t}\n\t\t// write the current height to a separate file\n\t\theightFile := filepath.Join(tempBackupDir, \"height.txt\")\n\t\tif err = os.WriteFile(heightFile, fmt.Appendf(nil, \"%d\", version), 0644); err != nil {\n\t\t\terr = fmt.Errorf(\"write height file: %w\", err)\n\t\t\treturn\n\t\t}\n\t\tif err = os.Rename(backupDir, prevBackupDir); err != nil && !os.IsNotExist(err) {\n\t\t\terr = fmt.Errorf(\"rotate backup: %w\", err)\n\t\t\treturn","sourceCodeStart":760,"sourceCodeEnd":796,"githubUrl":"https://github.com/canopy-network/canopy/blob/ee8197d91dd410f6592cb650a94c925ee6dc8bad/store/store.go#L760-L796","documentation":"Before creating a pebble checkpoint (backup), the store flushes the memtable to disk so the checkpoint does not depend on WAL replay (commits use NoSync). If that pre-checkpoint Flush fails, the backup is aborted with this wrapped error.","triggerScenarios":"Triggering the backup/checkpoint path (CreateBackup-style API around store.go:778) while db.Flush() fails — disk full, read-only filesystem, or I/O error at backup time.","commonSituations":"Scheduled backups running when the volume is full; backups on ephemeral/read-only container storage; concurrent heavy writes plus insufficient disk headroom during checkpointing.","solutions":["Inspect the wrapped pebble error and free disk space / fix filesystem permissions","Re-run the backup after resolving the disk condition","Monitor disk usage ahead of scheduled checkpoints to prevent recurrence","Verify WAL/memtable state and DB health after a failed flush before retrying"],"exampleFix":"// before\nerr := backup(dbDir) // fails silently on full disk\n// after\nif freeDisk(dbDir) < requiredHeadroom {\n    return fmt.Errorf(\"insufficient disk space for checkpoint backup\")\n}\nif err := backup(dbDir); err != nil {\n    return fmt.Errorf(\"backup failed: %w\", err)\n}","handlingStrategy":"try-catch","validationCode":"if freeDisk(dataDir) < requiredCheckpointSize { return errors.New(\"not enough disk space for checkpoint backup\") }\nif !writable(backupParentDir) { return errors.New(\"backup destination not writable\") }","typeGuard":null,"tryCatchPattern":"if err := st.CreateBackup(dest); err != nil {\n    if strings.Contains(err.Error(), \"flush before checkpoint\") {\n        log.Errorf(\"pre-checkpoint flush failed: %v — resolve disk/I/O then retry backup\", err)\n        os.RemoveAll(tempBackupDir)\n        return err\n    }\n    return err\n}","preventionTips":["Schedule backups only after disk-space checks with sufficient headroom for a full checkpoint","Clean up tempBackupDir on failure to avoid stale partial state","Keep WAL/memtable pressure low around scheduled checkpoints","Alert on any flush errors — they precede checkpoint and close failures"],"tags":["store","backup","checkpoint","flush","disk"],"backgroundTag":"database-write-failed","analyzedSha":"ee8197d91dd410f6592cb650a94c925ee6dc8bad","analyzedAt":"2026-09-06T09:30:15.973Z","contentChangedAt":"2026-09-06T09:30:15.973Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}