rqlite/rqlite · error

queue wait timeout

Error message

queue wait timeout

What it means

rqlite returns HTTP 408 'queue wait timeout' when a write submitted to the queued-write pipeline (?queue) waited longer than the client's configured timeout for its statements to be flushed to the Raft log. The server accepted the request into the queue, but the flusher did not pick it up before qp.Timeout(defaultTimeout) elapsed, so the wait on the flush channel aborted and the request is failed without any guarantee the write was applied.

Source

Thrown at http/service.go:1325

		stats.Add(numQueuedExecutionsWait, 1)
		fc = make(queue.FlushChannel)
	}

	seqNum, err := s.stmtQueue.Write(stmts, fc)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	resp.SequenceNum = seqNum

	if qp.Wait() {
		// Wait for the flush channel to close, or timeout.
		select {
		case <-fc:
			break
		case <-time.NewTimer(qp.Timeout(defaultTimeout)).C:
			stats.Add(numQueuedExecutionsWaitTimeout, 1)
			http.Error(w, "queue wait timeout", http.StatusRequestTimeout)
			return
		}
	}

	resp.end = time.Now()
	s.writeResponse(w, qp, resp)
}

// execute handles queries that modify the database.
func (s *Service) execute(w http.ResponseWriter, r *http.Request, qp QueryParams) {
	resp := NewResponse()
	resp.Results.AssociativeJSON = qp.Associative()
	resp.Results.BlobsAsArrays = qp.BlobArray()

	var stmts []*proto.Statement
	var err error
	if strings.HasPrefix(r.Header.Get("Content-Type"), "text/plain") {
		sql, err := io.ReadAll(r.Body)

View on GitHub (pinned to 7586a4d1bd)

Solutions

1. Increase the 'timeout' query parameter on the queued request so it exceeds the queue flush interval and expected flush latency
2. Reduce write pressure or batch size, or add replicas/read-only nodes and shard writes so the queued writer keeps up
3. Check leader health and disk I/O (Raft fsync is the bottleneck); move rqlited to faster storage
4. If immediate durability feedback is needed, drop ?queue and use standard (consensus-waiting) writes

Example fix

// before
curl -XPOST 'localhost:4001/db/execute?queue&timeout=1' -d '["INSERT INTO t VALUES(1)"]'
// after
curl -XPOST 'localhost:4001/db/execute?queue&timeout=10' -d '["INSERT INTO t VALUES(1)"]'

const timeoutMs = 10000; // ensure timeout param > queue flush interval
const url = `http://localhost:4001/db/execute?queue&timeout=${timeoutMs/1000}`;
if (timeoutMs <= 1000) throw new Error('queued-write timeout too small');

function isQueueWaitTimeout(resp) {
  return resp.status === 408 && resp.headers.get('Content-Type')?.includes('text/plain') === false;
}

const res = await fetch(url, {method:'POST', body});
if (res.status === 408) {
  // write MAY still have been applied — treat as indeterminate, check before retrying
  const check = await fetch('http://localhost:4001/db/query', {method:'POST', body: verifySql});
  if (!check.ok) await retryWithBackoff(() => postWrite(url, body));
}

When it happens

Trigger: POST /db/execute?queue=now (or via CLI queued writes) while the queued-writer's batch flush interval is large, the queue is backed up, or the supplied 'timeout' query parameter is shorter than the queue flush latency; the select in execute() at http/service.go:1325 times out waiting on the flush channel.

Common situations: High write throughput overwhelming the queue batcher; setting timeout=1 on queued writes while the default flush interval is longer; leader under heavy Raft fsync latency on slow disks; large batches from bulk writes delaying the flusher.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• unauthorized
• error sending HTTP request: %w
• remote Execute not authorized
• text || ("HTTP " + resp.status)
• ErrInvalidVersion

AI-assisted analysis of rqlite/rqlite@7586a4d1bd (2026-09-03). Data as JSON: /api/errors/28e474b5a2384a57. Report an issue: GitHub.