What causes "produce meta records: %s" in juicedata/juicefs?

Check the embedded engine error and confirm the metadata service is healthy/reachable.

produce meta records: %s

Error message

produce meta records: %s

What it means

Wrapped error from the external-sort GC path when scanGcMetaRecords — the errgroup worker that streams all slices from the metadata engine into the meta sorter — fails. The metadata scan is the source of truth for which blocks are live, so its failure aborts the whole GC run via errgroup. The engine error code is embedded in the message.

Source

Thrown at cmd/gc_external.go:62

	delFlag bool,
	maxMtime time.Time,
) error {
	logger.Infof("Using external sort mode, dir: %s", extSortDir)

	eg, sortCtx := errgroup.WithContext(c)
	sortMetaCtx := meta.WrapWithoutCancel(sortCtx, c.Pid(), c.Uid(), c.Gids())
	blobWithPrefix := object.WithPrefix(blob, "chunks/")
	metaSorter, objSorter, err := newGcExternalSorters(sortCtx, extSortDir, threads)
	if err != nil {
		return err
	}

	leakedObj, waitLeakedObj := startGcObjectDeleters(c, blobWithPrefix, threads, delFlag)

	eg.Go(func() error {
		defer stats.slices.Done()
		if err := scanGcMetaRecords(sortMetaCtx, m, metaSorter.Input(), stats.slices); err != nil {
			return errors.Errorf("produce meta records: %s", err)
		}
		metaSorter.CloseInput()
		return nil
	})

	eg.Go(func() error {
		defer stats.scanned.Done()
		if err := scanGcObjectRecords(sortCtx, blobWithPrefix, objSorter.Input(), threads, chunkConf.HashPrefix, maxMtime, stats.prefixes, stats.scanned, stats.skipped); err != nil {
			return errors.Errorf("produce object records: %s", err)
		}
		objSorter.CloseInput()
		return nil
	})

	eg.Go(func() error {
		if err := mergeGcSortedRecords(sortCtx, metaSorter, objSorter, chunkConf.BlockSize, stats, leakedObj); err != nil {
			return errors.Errorf("merge sorted records: %s", err)
		}

View on GitHub (pinned to c9a67b23e8)

Solutions

Check the embedded engine error and confirm the metadata service is healthy/reachable.
Re-run GC when the engine is stable; use a read replica if available.
Increase engine-side timeouts (e.g. Redis timeout config) for very large volumes.
Lower `--threads` to reduce engine load during the scan.
Pin client/engine versions to avoid mixed-version scan incompatibility.

Example fix

// before
juicefs gc --ext-sort /tmp/sort mysql://user:pass@host/jfs
// after increasing resilience
juicefs gc --ext-sort /tmp/sort --threads 8 mysql://user:pass@host/jfs

Defensive patterns

Strategy: retry

Validate before calling

juicefs status $META_URL > /dev/null || { echo 'metadata engine unreachable'; exit 1; }

Try / catch

if strings.Contains(out, "produce meta records:") {
    // engine error during slice scan; backoff and retry, check engine logs
}

Prevention

Verify engine stability before starting GC on large volumes
Increase engine timeouts for long scans
Use replicas for read-heavy maintenance work

When it happens

Trigger: `juicefs gc --ext-sort ...` while m.ScanSlices over Redis/SQL/TiKV fails: engine connection dropped mid-scan, engine restarted, scan timeout on very large volumes, or the progress callback returning an error on ctx cancellation.

Common situations: Long GC on a huge volume outlasting a flaky Redis connection; SQL server max_execution_time or connection limit hit; TiKV region errors; another admin restarting the metadata service mid-GC.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/47640ad8a6b092f1. Report an issue: GitHub.