benbjohnson/litestream · error

gs: cannot delete ltx file %q: %w

Error message

gs: cannot delete ltx file %q: %w

What it means

DeleteLTXFiles() failed to delete a specific LTX object in GCS. Unlike DeleteAll, this loop only tolerates NotExists errors explicitly; any other delete error (permissions, retention lock, transient API failure) aborts the deletion batch with the object key named.

Source

Thrown at gs/replica_client.go:231

	internal.OperationTotalCounterVec.WithLabelValues(ReplicaClientType, "GET").Inc()
	internal.OperationBytesCounterVec.WithLabelValues(ReplicaClientType, "GET").Add(float64(r.Attrs.Size))

	return r, nil
}

// DeleteLTXFiles deletes a set of LTX files.
func (c *ReplicaClient) DeleteLTXFiles(ctx context.Context, a []*ltx.FileInfo) error {
	if err := c.Init(ctx); err != nil {
		return err
	}

	for _, info := range a {
		key := litestream.LTXFilePath(c.Path, info.Level, info.MinTXID, info.MaxTXID)

		c.logger.Debug("deleting ltx file", "level", info.Level, "minTXID", info.MinTXID, "maxTXID", info.MaxTXID, "key", key)

		if err := c.bkt.Object(key).Delete(ctx); err != nil && !isNotExists(err) {
			return fmt.Errorf("gs: cannot delete ltx file %q: %w", key, err)
		}
		internal.OperationTotalCounterVec.WithLabelValues(ReplicaClientType, "DELETE").Inc()
	}

	return nil
}

type ltxFileIterator struct {
	it     *storage.ObjectIterator
	client *ReplicaClient
	level  int
	info   *ltx.FileInfo
	err    error
}

func newLTXFileIterator(it *storage.ObjectIterator, client *ReplicaClient, level int) *ltxFileIterator {
	return &ltxFileIterator{
		it:     it,

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Grant roles/storage.objectAdmin to the litestream service account on the bucket
2. Check the wrapped error's status code; 403 => permissions, 429/5xx => retry
3. Verify no bucket retention policy or hold prevents object deletion
4. Re-run retention — remaining deletions are retried on the next sweep since it aborts at first failure
5. Check connectivity/proxy settings if errors are transport-related

if err := rc.DeleteLTXFiles(ctx, a); err != nil {
    if isRetryable(err) { // 429/5xx
        time.Sleep(backoff)
        return rc.DeleteLTXFiles(ctx, a)
    }
    return err
}

When it happens

Trigger: Calling DeleteLTXFiles() during retention/compaction cleanup when the service account lacks storage.objects.delete, a GCS retention policy holds the object, or a transient API error occurs mid-batch.

Common situations: Read-only credentials used for replication (write-only or viewer role); bucket with object versioning/retention enabled; GCS throttling during large retention sweeps.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• gs: cannot delete object %q: %w
• failed to create GCS client (bucket: %s): %w
• failed to list objects in GCS bucket %s (path: %s): %w
• snapshot retention must be greater than 0
• l0 retention must not be negative

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/d96a5830d18610f3. Report an issue: GitHub.