benbjohnson/litestream · error

read dirty page %d from buffer: %w

Error message

read dirty page %d from buffer: %w

What it means

applySyncedPagesToHydratedFile copies dirty pages from the local buffer file into the hydrated database file. Reading a dirty page at its recorded buffer offset failed with a wrapped "read dirty page %d from buffer" error. This indicates the local buffer file is shorter than expected or its offsets are inconsistent with the dirty map.

Source

Thrown at vfs.go:1439

			return
		}
	}

	f.hydrator.SetComplete()

	// Clear cache since we'll now read from hydration file
	f.cache.Purge()

	f.logger.Debug("hydration complete", "path", f.hydrationPath, "txid", f.hydrator.TXID().String())
}

// applySyncedPagesToHydratedFile writes synced dirty pages to the hydrated file.
// Must be called with f.mu held.
func (f *VFSFile) applySyncedPagesToHydratedFile() error {
	for pgno, bufferOff := range f.dirty {
		data := make([]byte, f.pageSize)
		if _, err := f.bufferFile.ReadAt(data, bufferOff); err != nil {
			return fmt.Errorf("read dirty page %d from buffer: %w", pgno, err)
		}

		if err := f.hydrator.WritePage(pgno, data); err != nil {
			return err
		}
	}

	f.hydrator.SetTXID(f.expectedTXID)
	return nil
}

func (f *VFSFile) Close() error {
	f.logger.Debug("closing file")

	// Stop sync loop and ticker if running (need mutex for syncStop)
	f.mu.Lock()
	if f.syncStop != nil {
		close(f.syncStop)

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Check the wrapped error: io.EOF/short read means stale offsets — clear f.dirty and re-hydrate.
2. Verify the local buffer/scratch directory has free space and is writable.
3. Recreate the VFSFile (re-hydrate from replica) to rebuild buffer and dirty map consistently.
4. Report/persist buffer file and dirty-map lifecycle if offsets can survive buffer rebuilds.

Example fix

// before: buffer rebuilt but dirty map not cleared
f.bufferFile = newBuffer()
// stale offsets in f.dirty now point past EOF

// after: invalidate dirty entries when the buffer is recreated
f.bufferFile = newBuffer()
f.dirty = make(map[uint32]int64)

stat, err := f.bufferFile.Stat()
if err != nil { return err }
for pgno, off := range f.dirty {
    if off < 0 || off+int64(f.pageSize) > stat.Size() {
        delete(f.dirty, pgno) // stale offset; re-read from remote instead
    }
}

if err := f.applySyncedPagesToHydratedFile(); err != nil {
    if errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF) {
        f.dirty = make(map[uint32]int64) // rebuild buffer/dirty state
        err = f.hydrate(ctx)
    }
}

When it happens

Trigger: After sync, iterating f.dirty and calling bufferFile.ReadAt when the buffer file was truncated/recreated, the offset is stale (buffer rebuilt without clearing f.dirty), or an OS-level read error occurs (disk full, I/O error).

Common situations: Crash or restart leaving f.dirty entries pointing past the end of a recreated buffer file; disk-full conditions on the local scratch volume; tmpfs cleared under the process; bugs in buffer compaction not invalidating dirty offsets.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• read dirty page from buffer: %w
• failed to scan directory %s: %w
• copy L0 file: %w
• close L0 file: %w
• write hydration meta: %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/0d330cc51ff3f44d. Report an issue: GitHub.