What causes "chunk pipeline exec err: %w" in juicedata/juicefs?

Retry the dump; transient connection drops are the most common cause.

chunk pipeline exec err: %w

Error message

chunk pipeline exec err: %w

What it means

During `juicefs dump` of a Redis-backed metadata engine, dumpChunks batches SCAN-followup reads of chunk records into a Redis pipeline and calls pipe.Exec. This error wraps any transport/protocol failure returned by the pipeline itself (connection dropped, cluster redirection, timeout), aborting the metadata dump.

Source

Thrown at pkg/meta/redis_bak.go:561

	pipe := m.rdb.Pipeline()
	inos := make([]uint64, 0, len(keys))
	idxs := make([]uint32, 0, len(keys))
	for _, key := range keys {
		ps := strings.Split(key, "_")
		if len(ps) != 2 {
			logger.Warnf("invalid chunk key: %s", key)
			continue
		}
		ino, _ := strconv.ParseUint(ps[0][len(m.prefix)+1:], 10, 64)
		idx, _ := strconv.ParseUint(ps[1], 10, 32)
		pipe.LRange(ctx, m.chunkKey(Ino(ino), uint32(idx)), 0, -1)
		inos = append(inos, ino)
		idxs = append(idxs, uint32(idx))
	}

	cmds, err := pipe.Exec(ctx)
	if err != nil {
		return fmt.Errorf("chunk pipeline exec err: %w", err)
	}

	chunks := make([]*pb.Chunk, 0, len(cmds))
	for k, cmd := range cmds {
		vals, err := cmd.(*redis.StringSliceCmd).Result()
		if err != nil {
			return fmt.Errorf("get chunk result err: %w", err)
		}
		if len(vals) == 0 {
			continue
		}

		pc := pools[0].Get().(*pb.Chunk)
		pc.Inode = inos[k]
		pc.Index = idxs[k]

		pc.Slices = pools[1].Get().([]byte)
		if len(pc.Slices) < len(vals)*sliceBytes {

View on GitHub (pinned to c9a67b23e8)

Solutions

Retry the dump; transient connection drops are the most common cause.
Check redis-cli connectivity and server logs for OOM/restart/connection-limit events during the dump window.
If behind a load balancer/proxy, raise its idle timeout or connect directly to the Redis endpoint.
For ClusterClient, verify the cluster is healthy (cluster info) and all nodes reachable.
Upgrade go-redis / JuiceFS if errors are protocol-related with a newer Redis version.

Example fix

// before
if err := dumpChunks(...); err != nil { return err }
// after
var lastErr error
for i := 0; i < 3; i++ {
    if err := dumpChunks(...); err == nil { lastErr = nil; break }
    else { lastErr = err; time.Sleep(time.Second) }
}
return lastErr

Defensive patterns

Strategy: retry

Validate before calling

if err := rdb.Ping(ctx).Err(); err != nil { return fmt.Errorf("redis unreachable before dump: %w", err) }

Try / catch

for attempt := 0; attempt < 3; attempt++ {
    err := dumpMeta(ctx, src, dst)
    if err == nil { break }
    var netErr net.Error
    if errors.As(err, &netErr) || isRedisConnErr(err) { time.Sleep(backoff(attempt)); continue }
    return err
}

Prevention

Ping Redis immediately before starting a dump.
Avoid LB/proxy idle timeouts by keeping dumps short or raising idle timeout.
Monitor Redis memory and connection limits (maxmemory, maxclients) before bulk operations.
Pin a stable Redis endpoint rather than a DNS name behind flaky resolution.

When it happens

Trigger: Calling dump (or dumpChunks) against Redis when the connection fails mid-pipeline: server closed the connection, MOVED/ASK cluster redirection issues, read timeout, or Redis restarted during the dump.

Common situations: Dumping a large volume so the pipeline takes long enough for Redis to close an idle connection or for a proxy/LB (e.g. cloud-managed Redis) to time it out; pointing at a ClusterClient with a misconfigured node; network flakiness between client and Redis.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/b1f6ca42b574a1da. Report an issue: GitHub.