benbjohnson/litestream · error

gs: cannot delete object %q: %w

Error message

gs: cannot delete object %q: %w

What it means

DeleteAll() successfully listed objects but failed to delete a specific object in the bucket. Only genuine delete errors are wrapped here — Object Not Found (isNotExists) is tolerated and skipped. The wrapped cause carries the GCS API error for the named object.

Source

Thrown at gs/replica_client.go:113

func (c *ReplicaClient) DeleteAll(ctx context.Context) error {
	if err := c.Init(ctx); err != nil {
		return err
	}

	// Iterate over every object and delete it.
	internal.OperationTotalCounterVec.WithLabelValues(ReplicaClientType, "LIST").Inc()
	for it := c.bkt.Objects(ctx, &storage.Query{Prefix: c.Path + "/"}); ; {
		attrs, err := it.Next()
		if errors.Is(err, iterator.Done) {
			break
		} else if err != nil {
			return fmt.Errorf("failed to list objects in GCS bucket %s (path: %s): %w", c.Bucket, c.Path, err)
		}

		if err := c.bkt.Object(attrs.Name).Delete(ctx); isNotExists(err) {
			continue
		} else if err != nil {
			return fmt.Errorf("gs: cannot delete object %q: %w", attrs.Name, err)
		}
		internal.OperationTotalCounterVec.WithLabelValues(ReplicaClientType, "DELETE").Inc()
	}

	// log.Printf("%s(%s): retainer: deleting", r.db.Path(), r.Name())

	return nil
}

// LTXFiles returns an iterator over all available LTX files for a level.
// GCS always uses accurate timestamps from metadata since they're included in LIST operations at zero cost.
// The useMetadata parameter is ignored.
func (c *ReplicaClient) LTXFiles(ctx context.Context, level int, seek ltx.TXID, useMetadata bool) (ltx.FileIterator, error) {
	if err := c.Init(ctx); err != nil {
		return nil, err
	}

	dir := litestream.LTXLevelDir(c.Path, level)

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Grant storage.objects.delete permission (roles/storage.objectAdmin) to the service account
2. Check for bucket retention policies or event-based holds locking the objects
3. Inspect the wrapped error's GCS status code (403 permission vs 412 condition vs 5xx transient)
4. Verify no second litestream instance is replicating to the same bucket/path concurrently
5. Retry transient failures

Example fix

# before: delete denied
gsutil iam ch serviceAccount:sa@project.iam.gserviceaccount.com:roles/storage.objectAdmin gs://my-bucket
# also check retention
gsutil retention ls gs://my-bucket

// preflight delete permission
obj := bkt.Object("__litestream_permcheck__")
if err := obj.Delete(ctx); err != nil && !isNotExists(err) {
    return fmt.Errorf("service account lacks delete permission: %w", err)
}

if err := rc.DeleteAll(ctx); err != nil {
    var ee *googleapi.Error
    if errors.As(err, &ee) && ee.Code == 403 {
        // escalate IAM role and retry
    }
    return err
}

When it happens

Trigger: Calling DeleteAll() when the credentials have list but not delete permission (storage.objects.delete denied), the object was concurrently modified/has a retention policy/held, or a transient GCS error occurs on the delete call.

Common situations: IAM role grants objectViewer only (can list, can't delete); bucket with retention/bucket-lock policies preventing deletion; object recreated concurrently by another litestream instance; storage class/legal-hold restrictions.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• failed to list objects in GCS bucket %s (path: %s): %w
• gs: cannot delete ltx file %q: %w
• abs: cannot delete ltx file %q: %w
• abs: cannot delete blob %q: %w
• failed to create GCS client (bucket: %s): %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/ad7f223b6df5230a. Report an issue: GitHub.