juicedata/juicefs · warning

errUnstableCoLimit

errUnstableCoLimit

Error message

exceed concurrency %d limit for unstable disk cache

What it means

JuiceFS marks a disk-cache dir 'unstable' after I/O failures and then throttles access: checkCacheOp returns errUnstableCoLimit when the number of in-flight cache operations has reached maxConcurrencyForUnstable. This is a deliberate back-pressure signal, not corruption — operations against the unstable cache are temporarily limited.

Source

Thrown at pkg/chunk/disk_cache_state.go:45

var (
	numIOErrToUnstable         uint32  = 3                // from normal to unstable
	minIOSuccToNormal          uint32  = 60               // from unstable to normal
	maxIOErrPercentageToNormal float64 = 0                // from unstable to normal
	maxDurToDown                       = 30 * time.Minute // from unstable to down
	maxConcurrencyForUnstable  int64   = 10
	tickDurForNormal                   = 1 * time.Minute
	tickDurForUnstable                 = 1 * time.Minute

	probeDur  = 500 * time.Millisecond
	probeDir  = "probe"
	probeData = []byte{1, 2, 3}
	probeBuff = make([]byte, 3)
)

var (
	errCacheDown       = errors.New("cache down")
	errUnstableCoLimit = fmt.Errorf("exceed concurrency %d limit for unstable disk cache", maxConcurrencyForUnstable)
)

var diskStateNames = map[int]string{
	dcUnknown:   "unknown",
	dcNormal:    "normal",
	dcUnstable:  "unstable",
	dcDown:      "down",
	dcUnchanged: "unchanged",
}

const (
	dcUnknown = iota
	dcNormal
	dcUnstable
	dcDown
	dcUnchanged
)

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Reduce client read/write concurrency or the number of in-flight block operations hitting the cache.
2. Check and fix the cache disk (I/O errors, SMART, mount health) so the dir returns to normal state; restart the client to reset state after the disk recovers.
3. Move the cache to faster/reliable storage, or disable/replace the unstable cache dir.
4. In tests, always pair state.afterCacheOp() with each op as the test does when priming the limit.

Example fix

// before
// hammering cache with unbounded goroutines
for _, b := range blocks { go load(b) }

// after
sem := make(chan struct{}, maxConcurrencyForUnstable-1)
for _, b := range blocks {
    sem <- struct{}{}
    go func(b) { defer func() { <-sem }(); load(b) }(b)
}

// bound your own concurrency below the limit
if inflight >= maxConcurrencyForUnstable { backoff() }

for {
    _, err := cache.Load(key)
    if errors.Is(err, errUnstableCoLimit) {
        time.Sleep(backoff); continue
    }
    break
}

When it happens

Trigger: Loading/storing blocks through a DiskCache whose cache dir is in dcUnstable state while maxConcurrencyForUnstable goroutines already hold concurrent cache ops; seen in load()/store paths and reproduced in disk_cache_state_test.go by exhausting the concurrency counter without calling afterCacheOp.

Common situations: Slow or degraded cache disk (high latency, intermittent errors) marking the dir unstable under heavy read/write load; test code that drives the state machine without decrementing concurrency; very high client concurrency against a single cache dir.

Related errors

• concurrent staging limit reached
• multipart upload error: %s
• data checksum %d != expect %d
• insert/update failed
• write conflict: %s was version %d, now deleted

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/c7cb800f6b9a5cd3. Report an issue: GitHub.