What causes "flush before checkpoint: %w" in canopy-network/canopy?

Inspect the wrapped pebble error and free disk space / fix filesystem permissions

flush before checkpoint: %w

Error message

flush before checkpoint: %w

What it means

Before creating a pebble checkpoint (backup), the store flushes the memtable to disk so the checkpoint does not depend on WAL replay (commits use NoSync). If that pre-checkpoint Flush fails, the backup is aborted with this wrapped error.

Source

Thrown at store/store.go:778

				restoreErr := os.Rename(prevBackupDir, backupDir)
				if restoreErr != nil && !os.IsNotExist(restoreErr) {
					s.log.Errorf("failed to restore previous backup at height [%d]: %v", version, restoreErr)
				}
			} else {
				// otherwise, remove dangling backup, continue with current working backup
				_ = os.RemoveAll(prevBackupDir)
			}
			s.backup.Store(false)
			s.log.Errorf("backup failed at height [%d]: %v", version, err)
		}()
		// flush the memtable to SST before checkpointing so the backup does not
		// depend on WAL replay for recovery (commits use NoSync so WAL records
		// may not be durable on disk at checkpoint time)
		s.mu.Lock()
		version = s.Version()
		if err = s.db.Flush(); err != nil {
			s.mu.Unlock()
			err = fmt.Errorf("flush before checkpoint: %w", err)
			return
		}
		s.mu.Unlock()
		// perform the backup using pebble's checkpointing mechanism which creates a
		// consistent snapshot of the database at the specified directory
		if err = s.db.Checkpoint(tempBackupDir); err != nil {
			err = fmt.Errorf("checkpoint creation: %w", err)
			return
		}
		// write the current height to a separate file
		heightFile := filepath.Join(tempBackupDir, "height.txt")
		if err = os.WriteFile(heightFile, fmt.Appendf(nil, "%d", version), 0644); err != nil {
			err = fmt.Errorf("write height file: %w", err)
			return
		}
		if err = os.Rename(backupDir, prevBackupDir); err != nil && !os.IsNotExist(err) {
			err = fmt.Errorf("rotate backup: %w", err)
			return

View on GitHub (pinned to ee8197d91d)

Solutions

Inspect the wrapped pebble error and free disk space / fix filesystem permissions
Re-run the backup after resolving the disk condition
Monitor disk usage ahead of scheduled checkpoints to prevent recurrence
Verify WAL/memtable state and DB health after a failed flush before retrying

Example fix

// before
err := backup(dbDir) // fails silently on full disk
// after
if freeDisk(dbDir) < requiredHeadroom {
    return fmt.Errorf("insufficient disk space for checkpoint backup")
}
if err := backup(dbDir); err != nil {
    return fmt.Errorf("backup failed: %w", err)
}

Defensive patterns

Strategy: try-catch

Validate before calling

if freeDisk(dataDir) < requiredCheckpointSize { return errors.New("not enough disk space for checkpoint backup") }
if !writable(backupParentDir) { return errors.New("backup destination not writable") }

Try / catch

if err := st.CreateBackup(dest); err != nil {
    if strings.Contains(err.Error(), "flush before checkpoint") {
        log.Errorf("pre-checkpoint flush failed: %v — resolve disk/I/O then retry backup", err)
        os.RemoveAll(tempBackupDir)
        return err
    }
    return err
}

Prevention

Schedule backups only after disk-space checks with sufficient headroom for a full checkpoint
Clean up tempBackupDir on failure to avoid stale partial state
Keep WAL/memtable pressure low around scheduled checkpoints
Alert on any flush errors — they precede checkpoint and close failures

When it happens

Trigger: Triggering the backup/checkpoint path (CreateBackup-style API around store.go:778) while db.Flush() fails — disk full, read-only filesystem, or I/O error at backup time.

Common situations: Scheduled backups running when the volume is full; backups on ephemeral/read-only container storage; concurrent heavy writes plus insufficient disk headroom during checkpointing.

Related errors

AI-assisted analysis of canopy-network/canopy@ee8197d91d (2026-09-06). Data as JSON: /api/errors/8539f30ca76b8179. Report an issue: GitHub.