{"record":{"id":"5bad7fcef82f80fa","repo":"juicedata/juicefs","slug":"found-invalid-key-s-from-list-prefix-s-marker","errorCode":null,"errorMessage":"found invalid key %s from List, prefix: %s, marker: %s","messagePattern":"found invalid key (.+?) from List, prefix: (.+?), marker: (.+?)","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/object/tos.go","lineNumber":176,"sourceCode":"\nfunc (t *tosClient) List(ctx context.Context, prefix, start, token, delimiter string, limit int64, followLink bool) ([]Object, bool, string, error) {\n\tresp, err := t.client.ListObjectsType2(ctx, &tos.ListObjectsType2Input{\n\t\tBucket:            t.bucket,\n\t\tDelimiter:         delimiter,\n\t\tPrefix:            prefix,\n\t\tStartAfter:        start,\n\t\tMaxKeys:           int(limit),\n\t\tContinuationToken: token,\n\t})\n\tif err != nil {\n\t\treturn nil, false, \"\", err\n\t}\n\tn := len(resp.Contents)\n\tobjs := make([]Object, n)\n\tfor i := 0; i < n; i++ {\n\t\to := resp.Contents[i]\n\t\tif !strings.HasPrefix(o.Key, prefix) || o.Key <= start {\n\t\t\treturn nil, false, \"\", fmt.Errorf(\"found invalid key %s from List, prefix: %s, marker: %s\", o.Key, prefix, start)\n\t\t}\n\t\tobjs[i] = &obj{\n\t\t\to.Key,\n\t\t\to.Size,\n\t\t\to.LastModified,\n\t\t\tstrings.HasSuffix(o.Key, \"/\"),\n\t\t\tstring(o.StorageClass),\n\t\t\t\"\",\n\t\t}\n\t}\n\tif delimiter != \"\" {\n\t\tfor _, p := range resp.CommonPrefixes {\n\t\t\tobjs = append(objs, &obj{p.Prefix, 0, time.Unix(0, 0), true, \"\", \"\"})\n\t\t}\n\t\tsort.Slice(objs, func(i, j int) bool { return objs[i].Key() < objs[j].Key() })\n\t}\n\treturn objs, resp.IsTruncated, resp.NextContinuationToken, nil\n}","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/object/tos.go#L158-L194","documentation":"During TOS ListAll, each key returned by Volcano Engine TOS must sort after the marker and carry the requested prefix. When the SDK returns a key violating this invariant (key <= marker or not matching prefix), List aborts with this error instead of returning wrong results.","triggerScenarios":"Listing a TOS bucket with a prefix/marker while the bucket contains keys that sort oddly relative to the marker (e.g. marker shorter/different case), or a page boundary crossing where start marker doesn't align with returned keys.","commonSituations":"Bucket with keys containing unusual characters or case differences; concurrent writers adding/deleting keys around the marker; using a marker not previously returned by a real key; non-prefixed objects present with a prefix filter.","solutions":["Retry the list; if transient (concurrent modification), re-run the iteration","Check the marker value passed to List — it must be a real key (or empty) in the bucket","Verify the prefix matches actual object naming conventions in the bucket","Check TOS SDK/endpoint version for known listing-order bugs"],"exampleFix":"// before\nstore.List(ctx, prefix, \"\", limit, false) // marker with trailing slash not matching keys\n// after\nstore.List(ctx, prefix, \"\", limit, false) // use empty marker or exact key returned previously","handlingStrategy":"retry","validationCode":"func validMarker(marker, prefix string) error {\n\tif marker != \"\" && !strings.HasPrefix(marker, prefix) {\n\t\treturn fmt.Errorf(\"marker %q does not carry prefix %q\", marker, prefix)\n\t}\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"_, _, err := store.List(ctx, prefix, marker, limit, false)\nif err != nil && strings.Contains(err.Error(), \"found invalid key\") {\n\t// restart listing with an empty marker after a backoff\n}","preventionTips":["Only pass markers previously returned by List, or empty","Avoid prefixes that only partially match key naming conventions","Retry listings under concurrent write workloads"],"tags":["tos","listing","object-storage","invariant-violation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}