benbjohnson/litestream · error

encode header: %w

Error message

encode header: %w

What it means

While building an LTX file during sync, the encoder's EncodeHeader call failed and the error is wrapped as "encode header". The LTX header (page size, commit, min/max TXID, timestamp) is the first record in the immutable LTX file; if it cannot be encoded/written the transaction file is invalid and sync for this transaction fails.

Source

Thrown at vfs.go:2143

		}()

		enc, encErr := ltx.NewEncoder(pw)
		if encErr != nil {
			err = encErr
			return
		}

		// Encode header
		if err = enc.EncodeHeader(ltx.Header{
			Version:   ltx.Version,
			Flags:     ltx.HeaderFlagNoChecksum,
			PageSize:  pageSize,
			Commit:    commit,
			MinTXID:   pendingTXID,
			MaxTXID:   pendingTXID,
			Timestamp: time.Now().UnixMilli(),
		}); err != nil {
			err = fmt.Errorf("encode header: %w", err)
			return
		}

		// Encode each dirty page
		lockPgno := ltx.LockPgno(pageSize)
		for _, pgno := range pgnos {
			if pgno == lockPgno {
				continue // Skip lock page
			}

			// Read page data from buffer file
			bufferOff := dirtyOffsets[pgno]
			data := make([]byte, pageSize)
			if _, err = bufferFile.ReadAt(data, bufferOff); err != nil {
				err = fmt.Errorf("read page %d from buffer: %w", pgno, err)
				return
			}

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Check the wrapped cause: a broken-pipe error usually means the replica upload failed — fix the storage/network issue and retry the sync.
2. Verify disk space on the volume staging the LTX file.
3. Avoid cancelling the sync context mid-upload (shutdown timeouts too short); increase graceful-shutdown timeout.
4. If reproducible with no I/O cause, capture the exact wrapped error and file an upstream issue; LTX files are immutable so a partially written file must be discarded.

Example fix

// before: svc shuts down with 1s timeout, killing uploads mid-header
// after: allow in-flight syncs to finish
srv.Shutdown(ctx) // ensure ctx has >= 30s grace so LTX encode/upload completes

// ensure staging volume has headroom before large syncs
if st, err := os.Statvfs(stagingDir); err == nil && st.Bavail*uint64(st.Bsize) < minFreeBytes { return errors.New("staging disk low") }

if err := db.Sync(ctx); err != nil {
    if strings.Contains(err.Error(), "encode header") && errors.Is(err, os.ErrClosed) || strings.Contains(err.Error(), "pipe") {
        // upload side failed; check replica connectivity and retry
    }
    return err
}

When it happens

Trigger: enc.EncodeHeader(ltx.Header{...}) returns an error while streaming dirty pages into a new LTX file — typically the underlying writer (pipe or file) is broken, closed early, or hit a disk error mid-write.

Common situations: Reader side of the streaming pipe exited early (e.g. the replica client Upload failed or was cancelled), breaking the pipe with ErrClosedPipe; disk full on the staging volume; context cancellation during a slow upload.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• encode page %d: %w
• close encoder: %w
• encode ltx snapshot header: %w
• write snapshot ltx: %w
• close ltx snapshot encoder: %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/c4bff3e38d534f58. Report an issue: GitHub.