juicedata/juicefs · error

worker config is too large

Error message

worker config is too large

What it means

ReadClusterWorkerConfig caps the stdin payload at maxClusterWorkerConfigSize bytes; it reads one extra byte to detect overflow. If the payload exceeds the cap it throws 'worker config is too large' instead of unmarshaling an oversized/possibly malicious config.

Source

Thrown at pkg/sync/cluster.go:417

	}

	args := []string{host, path}
	args = append(args, workerArgs...)
	args = append(args, "--manager", address)
	if !config.Verbose && !config.Quiet {
		args = append(args, "-q")
	}
	return shellescape.EscapeArgs(args), payload, nil
}

// ReadClusterWorkerConfig reads storage URLs and environment variables from worker stdin.
func ReadClusterWorkerConfig(r io.Reader) (string, string, map[string]string, error) {
	data, err := io.ReadAll(io.LimitReader(r, maxClusterWorkerConfigSize+1))
	if err != nil {
		return "", "", nil, fmt.Errorf("read worker config: %s", err)
	}
	if len(data) > maxClusterWorkerConfigSize {
		return "", "", nil, fmt.Errorf("worker config is too large")
	}
	var config clusterWorkerConfig
	if err := json.Unmarshal(data, &config); err != nil {
		return "", "", nil, fmt.Errorf("unmarshal worker config: %s", err)
	}
	if config.Source == "" || config.Destination == "" {
		return "", "", nil, fmt.Errorf("worker config is missing source or destination")
	}
	return config.Source, config.Destination, config.Env, nil
}

func launchWorker(address string, config *Config, wg *sync.WaitGroup) {
	workers := strings.Split(strings.Join(config.Workers, ","), ",")
	for _, host := range workers {
		wg.Add(1)
		go func(host string) {
			defer wg.Done()
			// copy

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Reduce the environment size passed to the sync (unset unneeded variables before launching, or set --env-filter if available)
2. Shorten source/destination URLs (avoid embedding huge credentials/tokens in the URL)
3. Check maxClusterWorkerConfigSize in pkg/sync/cluster.go and, if legitimately needed, increase it in a local build
4. Verify manager and worker binary versions are compatible

Example fix

// before (huge env inherited)
juicefs sync src dst --cluster w1
// after (trim env)
env -i HOME=$HOME PATH=$PATH juicefs sync src dst --cluster w1

const maxClusterWorkerConfigSize = 1 << 20 // mirror of the code constant
if len(payloadJSON) > maxClusterWorkerConfigSize {
	return fmt.Errorf("worker env/URLs exceed %d bytes; trim environment before launching sync", maxClusterWorkerConfigSize)
}

When it happens

Trigger: The marshaled clusterWorkerConfig sent by the manager over stdin exceeds maxClusterWorkerConfigSize — typically because Env contains a very large number of variables or very long source/destination URLs.

Common situations: Sync launched from an environment with a huge inherited environment (hundreds of KBs of env vars, container with large injected config); extremely long presigned URLs with tokens; older manager version without size trimming paired with a newer worker.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

• read worker config: %s
• unmarshal worker config: %s
• worker config is missing source or destination
• can't locate source or destination in command arguments
• marshal worker config: %s

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/1b9fdf4fd0d50536. Report an issue: GitHub.