{"record":{"id":"ea7d3f70be5d22d6","repo":"juicedata/juicefs","slug":"produce-object-records-s","errorCode":null,"errorMessage":"produce object records: %s","messagePattern":"produce object records: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/gc_external.go","lineNumber":71,"sourceCode":"\tif err != nil {\n\t\treturn err\n\t}\n\n\tleakedObj, waitLeakedObj := startGcObjectDeleters(c, blobWithPrefix, threads, delFlag)\n\n\teg.Go(func() error {\n\t\tdefer stats.slices.Done()\n\t\tif err := scanGcMetaRecords(sortMetaCtx, m, metaSorter.Input(), stats.slices); err != nil {\n\t\t\treturn errors.Errorf(\"produce meta records: %s\", err)\n\t\t}\n\t\tmetaSorter.CloseInput()\n\t\treturn nil\n\t})\n\n\teg.Go(func() error {\n\t\tdefer stats.scanned.Done()\n\t\tif err := scanGcObjectRecords(sortCtx, blobWithPrefix, objSorter.Input(), threads, chunkConf.HashPrefix, maxMtime, stats.prefixes, stats.scanned, stats.skipped); err != nil {\n\t\t\treturn errors.Errorf(\"produce object records: %s\", err)\n\t\t}\n\t\tobjSorter.CloseInput()\n\t\treturn nil\n\t})\n\n\teg.Go(func() error {\n\t\tif err := mergeGcSortedRecords(sortCtx, metaSorter, objSorter, chunkConf.BlockSize, stats, leakedObj); err != nil {\n\t\t\treturn errors.Errorf(\"merge sorted records: %s\", err)\n\t\t}\n\t\treturn nil\n\t})\n\n\tsortErr := eg.Wait()\n\t_ = metaSorter.Done()\n\t_ = objSorter.Done()\n\twaitLeakedObj()\n\tif sortErr != nil {\n\t\treturn sortErr","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/cmd/gc_external.go#L53-L89","documentation":"Wrapped error from the external-sort GC worker that scans all chunk objects in the object store (scanGcObjectRecords) and feeds them into the object sorter. Any listing failure — auth, throttling, timeout, or nested prefix errors — aborts the errgroup and thus the GC run, because missing object records would make live blocks look leaked.","triggerScenarios":"`juicefs gc --ext-sort` while enumerating chunks/ objects: expired credentials, 503 SlowDown from S3, network interruption, or a prefix walker returning an error (including 'list chunks from %s' or delimiter errors).","commonSituations":"Huge buckets (hundreds of millions of blocks) hitting listing rate limits; IAM policy changes removing ListObjects permission; minio/S3 outage during a long GC window.","solutions":["Inspect the embedded cause; fix credentials/permissions first.","Retry GC with lower `--threads` to avoid throttling.","Re-check bucket IAM allows listing; `aws iam simulate-principal-action` or `mc ls` to validate.","Split GC into maintenance windows with stable object-store availability.","Ensure the storage endpoint in the volume config is correct after migrations."],"exampleFix":"// before (throttled)\njuicefs gc --ext-sort /mnt/sort --threads 200 $META\n// after\njuicefs gc --ext-sort /mnt/sort --threads 32 $META","handlingStrategy":"retry","validationCode":"aws s3 ls s3://$BUCKET/chunks/ --max-items 1 >/dev/null 2>&1 || { echo 'object listing precheck failed'; exit 1; }","typeGuard":null,"tryCatchPattern":"if strings.Contains(out, \"produce object records:\") {\n    // object scan failed; fix embedded cause (auth/throttle) and re-run\n}","preventionTips":["Pre-validate list permissions with IAM policy checks","Throttle --threads on large buckets","Schedule GC within stable object-store availability windows"],"tags":["go","object-storage","s3","gc","external-sort"],"backgroundTag":"api-request-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}