{"record":{"id":"947f8cd256f008e7","repo":"benbjohnson/litestream","slug":"max-retries-exceeded-reading-ltx-file-level-d-m","errorCode":null,"errorMessage":"max retries exceeded reading ltx file (level=%d, min=%s, max=%s, offset=%d): %w","messagePattern":"max retries exceeded reading ltx file \\(level=(.+?), min=(.+?), max=(.+?), offset=(.+?)\\): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/resumable_reader.go","lineNumber":165,"sourceCode":"\t\treturn r.rc.Close()\n\t}\n\treturn nil\n}\n\nfunc (r *ResumableReader) close() {\n\t// The stream is already being discarded after a read failure, so a close\n\t// error should not stop recovery. Log it only to aid debugging.\n\tif err := r.rc.Close(); err != nil {\n\t\tr.logger.Debug(\"close ltx file\",\n\t\t\t\"level\", r.level, \"min\", r.minTXID, \"max\", r.maxTXID,\n\t\t\t\"offset\", r.offset, \"error\", err)\n\t}\n}\n\nfunc (r *ResumableReader) retry(err error) error {\n\tr.retryN++\n\tif r.retryN > resumableReaderMaxRetries {\n\t\tr.err = fmt.Errorf(\"max retries exceeded reading ltx file (level=%d, min=%s, max=%s, offset=%d): %w\",\n\t\t\tr.level, r.minTXID, r.maxTXID, r.offset, err)\n\t\treturn r.err\n\t}\n\n\t// Wait before the caller reopens. Retrying with no delay lands every\n\t// attempt inside the same provider throttle window, so the attempts are\n\t// spent without the provider ever getting a chance to recover.\n\tselect {\n\tcase <-r.ctx.Done():\n\t\tr.err = r.ctx.Err()\n\t\treturn r.err\n\tcase <-time.After(resumableReaderBackoff << (r.retryN - 1)):\n\t}\n\treturn nil\n}\n","sourceCodeStart":147,"sourceCodeEnd":181,"githubUrl":"https://github.com/benbjohnson/litestream/blob/4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3/internal/resumable_reader.go#L147-L181","documentation":"ResumableReader.readAt retries failed reads of an LTX file from replica storage up to resumableReaderMaxRetries times. When every attempt fails (typically due to provider throttling or transient storage errors), it wraps the last error in this message with the level, TXID range, and byte offset that could not be read. It signals that replication/restore reading of this LTX segment must stop because storage is persistently unavailable.","triggerScenarios":"ReplicaClient.Read repeatedly returns errors while reading an LTX object at r.offset for the given level/TXID range — e.g. S3 returns 503 SlowDown on every attempt, the object was deleted mid-read, or the connection keeps dropping — so r.retryN exceeds resumableReaderMaxRetries.","commonSituations":"Cloud storage rate limiting during heavy compaction/restore; network partitions between litestream and the replica bucket; accidentally deleted or lifecycle-expired LTX objects; provider outages; retry backoff window still falling inside the provider throttle window.","solutions":["Check the wrapped inner error (%w) to find the root cause — it is usually a storage-provider error, not an LTX problem","Verify the LTX object still exists in the replica storage for the reported level/TXID range","Back off and retry the operation later if the cause is provider throttling (503/SlowDown)","Check network connectivity / egress limits between the host and the replica provider","If objects are persistently missing or corrupt, run `litestream reset` to clear local LTX state and resync"],"exampleFix":"// caller side: tolerate transient storage errors with outer backoff\n// before\nerr := reader.Read(buf)\n// after\nvar maxRetriesErr *fmt.Errorf\nerr := backoff.Retry(func() error {\n\terr = reader.Read(buf)\n\tif err != nil && strings.Contains(err.Error(), \"max retries exceeded reading ltx file\") {\n\t\treturn backoff.Permanent(err) // storage unavailable; outer alert/recover\n\t}\n\treturn err\n}, backoff.NewExponentialBackOff())","handlingStrategy":"retry","validationCode":"// Pre-check replica reachability before long reads\nif err := db.Sync(ctx); err != nil {\n\tlog.Printf(\"replica unreachable, deferring read: %v\", err)\n}","typeGuard":"// Check availability through the client before bulk reads\nif err := client.Init(ctx); err != nil {\n\treturn fmt.Errorf(\"replica storage unavailable: %w\", err)\n}","tryCatchPattern":"// Treat as terminal for this pass; alert and retry the whole operation later\nif err := reader.Read(buf); err != nil {\n\tif strings.Contains(err.Error(), \"max retries exceeded reading ltx file\") {\n\t\treturn fmt.Errorf(\"replica storage persistently unavailable (check inner cause): %w\", err)\n\t}\n\treturn err\n}","preventionTips":["Monitor and alert on replica storage error rates (throttling/5xx)","Keep sufficient provider request-rate headroom during restore/compaction","Ensure LTX retention policies in the provider don't delete objects mid-read","Run litestream close to the replica region to reduce network failures"],"tags":["storage","retry","replication","throttling"],"backgroundTag":"max-retries-exceeded","analyzedSha":"4ed7a308f6271ebfd2b0a6e4b70b03011a37e4a3","analyzedAt":"2026-09-06T18:29:25.564Z","contentChangedAt":"2026-09-06T18:29:25.564Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}