What causes "produce object records: %s" in juicedata/juicefs?

Inspect the embedded cause; fix credentials/permissions first.

produce object records: %s

Error message

produce object records: %s

What it means

Wrapped error from the external-sort GC worker that scans all chunk objects in the object store (scanGcObjectRecords) and feeds them into the object sorter. Any listing failure — auth, throttling, timeout, or nested prefix errors — aborts the errgroup and thus the GC run, because missing object records would make live blocks look leaked.

Source

Thrown at cmd/gc_external.go:71

	if err != nil {
		return err
	}

	leakedObj, waitLeakedObj := startGcObjectDeleters(c, blobWithPrefix, threads, delFlag)

	eg.Go(func() error {
		defer stats.slices.Done()
		if err := scanGcMetaRecords(sortMetaCtx, m, metaSorter.Input(), stats.slices); err != nil {
			return errors.Errorf("produce meta records: %s", err)
		}
		metaSorter.CloseInput()
		return nil
	})

	eg.Go(func() error {
		defer stats.scanned.Done()
		if err := scanGcObjectRecords(sortCtx, blobWithPrefix, objSorter.Input(), threads, chunkConf.HashPrefix, maxMtime, stats.prefixes, stats.scanned, stats.skipped); err != nil {
			return errors.Errorf("produce object records: %s", err)
		}
		objSorter.CloseInput()
		return nil
	})

	eg.Go(func() error {
		if err := mergeGcSortedRecords(sortCtx, metaSorter, objSorter, chunkConf.BlockSize, stats, leakedObj); err != nil {
			return errors.Errorf("merge sorted records: %s", err)
		}
		return nil
	})

	sortErr := eg.Wait()
	_ = metaSorter.Done()
	_ = objSorter.Done()
	waitLeakedObj()
	if sortErr != nil {
		return sortErr

View on GitHub (pinned to c9a67b23e8)

Solutions

Inspect the embedded cause; fix credentials/permissions first.
Retry GC with lower `--threads` to avoid throttling.
Re-check bucket IAM allows listing; `aws iam simulate-principal-action` or `mc ls` to validate.
Split GC into maintenance windows with stable object-store availability.
Ensure the storage endpoint in the volume config is correct after migrations.

Example fix

// before (throttled)
juicefs gc --ext-sort /mnt/sort --threads 200 $META
// after
juicefs gc --ext-sort /mnt/sort --threads 32 $META

Defensive patterns

Strategy: retry

Validate before calling

aws s3 ls s3://$BUCKET/chunks/ --max-items 1 >/dev/null 2>&1 || { echo 'object listing precheck failed'; exit 1; }

Try / catch

if strings.Contains(out, "produce object records:") {
    // object scan failed; fix embedded cause (auth/throttle) and re-run
}

Prevention

Pre-validate list permissions with IAM policy checks
Throttle --threads on large buckets
Schedule GC within stable object-store availability windows

When it happens

Trigger: `juicefs gc --ext-sort` while enumerating chunks/ objects: expired credentials, 503 SlowDown from S3, network interruption, or a prefix walker returning an error (including 'list chunks from %s' or delimiter errors).

Common situations: Huge buckets (hundreds of millions of blocks) hitting listing rate limits; IAM policy changes removing ListObjects permission; minio/S3 outage during a long GC window.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/ea7d3f70be5d22d6. Report an issue: GitHub.