juicedata/juicefs · error

found invalid key %s from List, prefix: %s, marker: %s

Error message

found invalid key %s from List, prefix: %s, marker: %s

What it means

During TOS ListAll, each key returned by Volcano Engine TOS must sort after the marker and carry the requested prefix. When the SDK returns a key violating this invariant (key <= marker or not matching prefix), List aborts with this error instead of returning wrong results.

Source

Thrown at pkg/object/tos.go:176

func (t *tosClient) List(ctx context.Context, prefix, start, token, delimiter string, limit int64, followLink bool) ([]Object, bool, string, error) {
	resp, err := t.client.ListObjectsType2(ctx, &tos.ListObjectsType2Input{
		Bucket:            t.bucket,
		Delimiter:         delimiter,
		Prefix:            prefix,
		StartAfter:        start,
		MaxKeys:           int(limit),
		ContinuationToken: token,
	})
	if err != nil {
		return nil, false, "", err
	}
	n := len(resp.Contents)
	objs := make([]Object, n)
	for i := 0; i < n; i++ {
		o := resp.Contents[i]
		if !strings.HasPrefix(o.Key, prefix) || o.Key <= start {
			return nil, false, "", fmt.Errorf("found invalid key %s from List, prefix: %s, marker: %s", o.Key, prefix, start)
		}
		objs[i] = &obj{
			o.Key,
			o.Size,
			o.LastModified,
			strings.HasSuffix(o.Key, "/"),
			string(o.StorageClass),
			"",
		}
	}
	if delimiter != "" {
		for _, p := range resp.CommonPrefixes {
			objs = append(objs, &obj{p.Prefix, 0, time.Unix(0, 0), true, "", ""})
		}
		sort.Slice(objs, func(i, j int) bool { return objs[i].Key() < objs[j].Key() })
	}
	return objs, resp.IsTruncated, resp.NextContinuationToken, nil
}

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Retry the list; if transient (concurrent modification), re-run the iteration
2. Check the marker value passed to List — it must be a real key (or empty) in the bucket
3. Verify the prefix matches actual object naming conventions in the bucket
4. Check TOS SDK/endpoint version for known listing-order bugs

Example fix

// before
store.List(ctx, prefix, "", limit, false) // marker with trailing slash not matching keys
// after
store.List(ctx, prefix, "", limit, false) // use empty marker or exact key returned previously

func validMarker(marker, prefix string) error {
	if marker != "" && !strings.HasPrefix(marker, prefix) {
		return fmt.Errorf("marker %q does not carry prefix %q", marker, prefix)
	}
	return nil
}

_, _, err := store.List(ctx, prefix, marker, limit, false)
if err != nil && strings.Contains(err.Error(), "found invalid key") {
	// restart listing with an empty marker after a backoff
}

When it happens

Trigger: Listing a TOS bucket with a prefix/marker while the bucket contains keys that sort oddly relative to the marker (e.g. marker shorter/different case), or a page boundary crossing where start marker doesn't align with returned keys.

Common situations: Bucket with keys containing unusual characters or case differences; concurrent writers adding/deleting keys around the marker; using a marker not previously returned by a real key; non-prefixed objects present with a prefix filter.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• invalid endpoint: %v, error: %v
• list all blocks: %s
• delimiter list returned nested object %s
• list chunk prefix %s: %s
• ceph: can't put empty file

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/5bad7fcef82f80fa. Report an issue: GitHub.