benbjohnson/litestream · error

failed to delete files:

Error message

failed to delete files:

What it means

The S3 replica client builds this error when a multi-object delete returns per-key errors; each failed key and the AWS-reported message are appended under the 'failed to delete files:' header. It is returned by DeleteLTXFiles/DeleteAll paths when the batched delete partially or fully fails.

Source

Thrown at s3/replica_client.go:1859

	var apiErr smithy.APIError
	if errors.As(err, &apiErr) {
		return apiErr.ErrorCode() == "NoSuchKey"
	}
	return false
}

func deleteOutputError(out *s3.DeleteObjectsOutput) error {
	if len(out.Errors) == 0 {
		return nil
	}

	// Build generic error
	var b strings.Builder
	b.WriteString("failed to delete files:")
	for _, err := range out.Errors {
		fmt.Fprintf(&b, "\n%s: %s", aws.ToString(err.Key), aws.ToString(err.Message))
	}
	return errors.New(b.String())
}

// parseS3DebugEnv parses the LITESTREAM_S3_DEBUG environment variable and returns
// the corresponding AWS SDK ClientLogMode. Supports comma-separated values.
func parseS3DebugEnv() aws.ClientLogMode {
	v := os.Getenv("LITESTREAM_S3_DEBUG")
	if v == "" {
		return 0
	}

	var logMode aws.ClientLogMode
	for _, mode := range strings.Split(v, ",") {
		switch strings.ToLower(strings.TrimSpace(mode)) {
		case "signing":
			logMode |= aws.LogSigning
		case "request":
			logMode |= aws.LogRequest
		case "request-with-body":

View on GitHub (pinned to 4ed7a308f6)

Solutions

1. Inspect the key: message pairs listed in the error to see which objects failed and the AWS reason
2. Verify IAM permissions include s3:DeleteObject (and s3:DeleteObjectVersion for versioned buckets) on the litestream prefix
3. Adjust S3 lifecycle rules so they don't race with litestream's retention-based deletion
4. Retry with backoff; SlowDown/throttling errors are transient and often resolve on retry

Example fix

// before
err := db.Sync(ctx)
// after
err := db.Sync(ctx)
if err != nil && strings.HasPrefix(err.Error(), "failed to delete files:") {
    log.Printf("S3 batch delete partially failed (check listed keys/IAM): %v", err)
}

// verify IAM delete permission before bulk deletes
iamSimulate(s3:DeleteObject, bucketArn+"/db/*")

func isS3BatchDeleteErr(err error) bool {
    return strings.HasPrefix(err.Error(), "failed to delete files:")
}

if err := replicaClient.DeleteLTXFiles(ctx, files); err != nil {
    if isS3BatchDeleteErr(err) {
        // inspect per-key messages (NoSuchKey vs AccessDenied vs SlowDown)
        // retry SlowDown with backoff; skip NoSuchKey; fix IAM for AccessDenied
    }
    return err
}

When it happens

Trigger: Deleting LTX files on an S3 replica where the DeleteObjects API response contains entries in out.Errors (e.g. NoSuchKey, AccessDenied, SlowDown) — s3/replica_client.go:1859 formats them into one error.

Common situations: Objects already removed by an S3 lifecycle/expiration rule; IAM policy missing s3:DeleteObject on the bucket/prefix; S3 throttling during large batch deletes; versioned buckets with deny rules on delete markers.

Related errors

• oss: failed to delete files:
• s3: put object %s: %w
• lease not held
• abs: cannot delete ltx file %q: %w
• abs: cannot delete blob %q: %w

AI-assisted analysis of benbjohnson/litestream@4ed7a308f6 (2026-09-06). Data as JSON: /api/errors/36d4703180cf9ed6. Report an issue: GitHub.