{"record":{"id":"47e6e7e2c70b4f44","repo":"juicedata/juicefs","slug":"list-chunks-from-s-s","errorCode":null,"errorMessage":"list chunks from %s: %s","messagePattern":"list chunks from (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/gc.go","lineNumber":515,"sourceCode":"\t\t\tselect {\n\t\t\tcase prefixes <- prefix:\n\t\t\t\treturn nil\n\t\t\tcase <-scanCtx.Done():\n\t\t\t\treturn scanCtx.Err()\n\t\t\t}\n\t\t})\n\t})\n\n\tfor range threads {\n\t\teg.Go(func() error {\n\t\t\tfor {\n\t\t\t\tselect {\n\t\t\t\tcase prefix, ok := <-prefixes:\n\t\t\t\t\tif !ok {\n\t\t\t\t\t\treturn nil\n\t\t\t\t\t}\n\t\t\t\t\tif err := scanGcChunkObjectsPrefix(scanCtx, blob, prefix, handle); err != nil {\n\t\t\t\t\t\treturn errors.Errorf(\"list chunks from %s: %s\", blob, err)\n\t\t\t\t\t}\n\t\t\t\t\tprefixSpin.Increment()\n\t\t\t\tcase <-scanCtx.Done():\n\t\t\t\t\treturn scanCtx.Err()\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\t}\n\treturn eg.Wait()\n}\n\nfunc walkGcChunkObjectPrefixes(ctx context.Context, blob object.ObjectStorage, prefix string, depth int, allowFallback bool, fn func(string) error) error {\n\tseenPrefixes := make(map[string]struct{})\n\tvar marker, token string\n\tfirstPage := true\n\tfor {\n\t\tobjs, hasMore, nextToken, err := blob.List(ctx, prefix, marker, token, \"/\", gcObjectListBatch, true)\n\t\tif err != nil {","sourceCodeStart":497,"sourceCodeEnd":533,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/cmd/gc.go#L497-L533","documentation":"Error produced by the prefix-walking worker in scanGcChunkObjects when a single-prefix scan (`scanGcChunkObjectsPrefix`) of a hashed chunk prefix fails. The blob name and underlying error are embedded, identifying which object-storage prefix enumeration broke. It aborts that worker, propagating up as a failed GC object listing.","triggerScenarios":"During `juicefs gc`, listing a specific 2-hex-character prefix under chunks/ fails due to object-store ListObjects errors: throttling (503 SlowDown), expired credentials, transient network errors, or ctx cancellation racing with the list call.","commonSituations":"Very large buckets where each prefix listing is slow and the object store rate-limits parallel prefix listings; temporary S3 unavailability; minio restart mid-GC; context cancelled because another worker failed first.","solutions":["Retry the GC; transient 5xx/throttle errors from the store are the usual cause.","Lower `--threads` to reduce parallel ListObjects pressure on the object store.","Check store-side rate limits (S3 request rate per prefix) and credentials validity.","Inspect the embedded error for 503/timeout and re-run during lower load.","If the volume is huge, use `--ext-sort` mode which handles large listings with external sort."],"exampleFix":"// before\njuicefs gc --threads 256 redis://localhost/1\n// after\njuicefs gc --threads 16 redis://localhost/1","handlingStrategy":"retry","validationCode":"# probe one prefix before GC\naws s3api list-objects-v2 --bucket $BUCKET --prefix chunks/ --max-items 1 > /dev/null","typeGuard":null,"tryCatchPattern":"if strings.Contains(out, \"list chunks from \") {\n    // likely transient store error; exponential backoff and re-run gc\n}","preventionTips":["Size --threads under the store's per-prefix request-rate limits","Retry GC on 5xx/throttle errors","Keep object store endpoints on a stable network path"],"tags":["go","object-storage","s3","gc","prefix-listing"],"backgroundTag":"api-request-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}