kubernetes/kops · error

error removing file/marker versions: %v

Error message

error removing file/marker versions: %v

What it means

DeleteObjects can return HTTP 200 with per-object Errors inside the result. RemoveAllVersions checks deleteResult.Errors and throws "error removing file/marker versions: [...]" listing those per-key failures. This means the request succeeded but some individual version deletions were rejected.

Source

Thrown at util/pkg/vfs/s3fs.go:256

		}

		// DeleteObjects can only process 1000 objects per call
		if len(objects) > 1000 {
			request.Delete.Objects = objects[:1000]
			objects = objects[1000:]
		} else {
			request.Delete.Objects = objects
			objects = nil
		}

		klog.V(8).Infof("removing %d file/marker versions\n", len(request.Delete.Objects))

		deleteResult, err := client.DeleteObjects(ctx, request)
		if err != nil {
			return fmt.Errorf("error removing %d file/marker versions: %v", len(request.Delete.Objects), err)
		}
		if len(deleteResult.Errors) > 0 {
			return fmt.Errorf("error removing file/marker versions: %v", deleteResult.Errors)
		}
	}

	return nil
}

func (p *S3Path) Join(relativePath ...string) Path {
	args := []string{p.key}
	args = append(args, relativePath...)
	joined := path.Join(args...)
	return &S3Path{
		s3Context: p.s3Context,
		bucket:    p.bucket,
		key:       joined,
		scheme:    p.scheme,
		sse:       p.sse,
		optFn:     p.optFn,
	}

View on GitHub (pinned to 4c8573c808)

Solutions

1. Inspect the listed per-object errors (Key, VersionId, Code, Message) in the error text
2. Remove Object Lock retention/legal hold or wait for the retention period to expire if objects are locked
3. Ensure no other process is deleting from the same state store concurrently (avoid parallel kOps deletes)
4. Adjust key-scoped IAM/bucket policy conditions that reject specific prefixes

Example fix

// before
if len(deleteResult.Errors) > 0 { return fmt.Errorf("error removing file/marker versions: %v", deleteResult.Errors) }
// after (caller): surface key + code clearly
var failed []string
for _, e := range result.Errors { failed = append(failed, e.Key+":"+aws.ToString(e.Code)) }
return fmt.Errorf("partial version delete failure: %s", strings.Join(failed, ", "))

// Detect Object Lock before attempting version purge
lock, err := s3Client.GetObjectLockConfiguration(ctx, &s3.GetObjectLockConfigurationInput{Bucket: bucket})
locked := err == nil && lock.ObjectLockConfiguration.ObjectLockEnabled == types.ObjectLockEnabled

func hasDeleteErrors(r *types.DeleteResult) bool { return r != nil && len(r.Errors) > 0 }

err := p.RemoveAllVersions(ctx)
var partials []types.Error
if err != nil && strings.Contains(err.Error(), "Code: AccessDenied") {
	// inspect per-key failures referenced in err, skip locked objects
}

When it happens

Trigger: Calling RemoveAllVersions when specific version deletes fail server-side: permission denied on particular keys (key-level policy conditions), invalid version IDs (e.g. version removed concurrently), or protected objects (Object Lock/legal hold).

Common situations: Object Lock retention/legal hold on cluster state files; concurrent deletions of the same state store by two kOps runs; IAM key-prefix conditions that reject some keys in the batch.

Related errors

• error listing all versions of file %s: %v
• error removing %d file/marker versions: %v
• failed to generate AWS IAM S3 access statements: %v
• cannot parse VFS path %q: %v
• path is not cluster readable: %v

AI-assisted analysis of kubernetes/kops@4c8573c808 (2026-09-05). Data as JSON: /api/errors/75921f049e85677e. Report an issue: GitHub.