What causes "open transaction at end of WAL file" in rqlite/rqlite?

Discard or regenerate the incomplete WAL — SQLite will ignore non-commit frames on recovery

ErrOpenTransaction

ErrOpenTransaction

Error message

open transaction at end of WAL file

What it means

ErrOpenTransaction is returned by the CompactingFrameScanner when the final frame in a WAL file is not a commit frame, i.e. the WAL ends mid-transaction. SQLite treats trailing non-committed frames as incomplete, so the scanner refuses to process them rather than silently dropping or compacting a partial transaction.

Source

Thrown at db/wal/compacting_section_scanner.go:17

package wal

import (
	"encoding/binary"
	"errors"
	"expvar"
	"fmt"
	"io"
	"maps"
	"math"
	"sort"
	"time"
)

var (
	// ErrOpenTransaction is returned when the final frame in the WAL file is not a committing frame.
	ErrOpenTransaction = errors.New("open transaction at end of WAL file")
)

// CompactingFrameScanner implements WALIterator to iterate over compacted WAL
// frames starting from a given frame index. It scans all valid frames from the
// start position to the end of the WAL (or the first invalid frame), keeps only
// the latest version of each page (respecting transaction boundaries), and
// returns them in file offset order. If fullScan is false, frame checksums are
// not verified since the WAL file is trusted.
type CompactingFrameScanner struct {
	readSeeker io.ReadSeeker
	walReader  *Reader
	header     *WALHeader
	fullScan   bool
	start      int64

	// pageBuf is a scratch buffer reused across Next() calls to avoid a
	// page-sized allocation per frame. The Frame returned by Next aliases
	// this buffer in its Data field; see Next's doc comment.

View on GitHub (pinned to 7586a4d1bd)

Solutions

Discard or regenerate the incomplete WAL — SQLite will ignore non-commit frames on recovery
Re-copy the database and WAL with the SQLite backup API or the checkpoint API instead of raw file copy
Ensure the WAL file transfer completes atomically before scanning

Example fix

// before
scanner, err := NewCompactingFrameScanner(bytes.NewReader(truncated), 0, false)
// after
scanner, err := NewCompactingFrameScanner(bytes.NewReader(fullWAL), 0, false)
if errors.Is(err, dbwal.ErrOpenTransaction) {
    log.Printf("WAL ends mid-transaction; ignoring trailing frames")
}

Defensive patterns

Strategy: try-catch

Validate before calling

// inspect final frame commit flag before scanning
// a frame whose header lacks the commit bit means the WAL ends mid-transaction

Type guard

func walEndsInCommit(wal []byte) bool {
    if len(wal) < 32+24 { return false }
    off := len(wal) - 24
    return binary.BigEndian.Uint32(wal[off+4:off+8]) != 0 // commit field != 0
}

Try / catch

iter, err := NewCompactingFrameScanner(r, start, check)
if errors.Is(err, ErrOpenTransaction) {
    // treat trailing frames as incomplete; fall back to full WAL or regenerate
    return handleIncompleteWAL(err)
}

Prevention

Copy WAL files atomically (SQLite backup API or checkpoint first)
Never truncate a WAL at an arbitrary frame boundary
Validate the last frame is a commit frame before compacting
Treat ErrOpenTransaction as 'incomplete tail', not fatal data loss

When it happens

Trigger: Passing a WAL file whose last frame lacks the commit flag to NewCompactingFrameScanner / scan — e.g. a WAL truncated at db/wal/compacting_section_scanner.go:250 because the writer crashed or the file was copied mid-write.

Common situations: Copying or snapshotting a live SQLite database without proper locking; process crash mid-transaction; truncated WAL transfer between nodes; test fixtures that cut the WAL at a frame boundary.

Related errors

AI-assisted analysis of rqlite/rqlite@7586a4d1bd (2026-09-03). Data as JSON: /api/errors/778b6a2dea43e37d. Report an issue: GitHub.