{"record":{"id":"c7cb800f6b9a5cd3","repo":"juicedata/juicefs","slug":"errunstablecolimit","errorCode":"errUnstableCoLimit","errorMessage":"exceed concurrency %d limit for unstable disk cache","messagePattern":"exceed concurrency (.+?) limit for unstable disk cache","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/chunk/disk_cache_state.go","lineNumber":45,"sourceCode":"\nvar (\n\tnumIOErrToUnstable         uint32  = 3                // from normal to unstable\n\tminIOSuccToNormal          uint32  = 60               // from unstable to normal\n\tmaxIOErrPercentageToNormal float64 = 0                // from unstable to normal\n\tmaxDurToDown                       = 30 * time.Minute // from unstable to down\n\tmaxConcurrencyForUnstable  int64   = 10\n\ttickDurForNormal                   = 1 * time.Minute\n\ttickDurForUnstable                 = 1 * time.Minute\n\n\tprobeDur  = 500 * time.Millisecond\n\tprobeDir  = \"probe\"\n\tprobeData = []byte{1, 2, 3}\n\tprobeBuff = make([]byte, 3)\n)\n\nvar (\n\terrCacheDown       = errors.New(\"cache down\")\n\terrUnstableCoLimit = fmt.Errorf(\"exceed concurrency %d limit for unstable disk cache\", maxConcurrencyForUnstable)\n)\n\nvar diskStateNames = map[int]string{\n\tdcUnknown:   \"unknown\",\n\tdcNormal:    \"normal\",\n\tdcUnstable:  \"unstable\",\n\tdcDown:      \"down\",\n\tdcUnchanged: \"unchanged\",\n}\n\nconst (\n\tdcUnknown = iota\n\tdcNormal\n\tdcUnstable\n\tdcDown\n\tdcUnchanged\n)\n","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/chunk/disk_cache_state.go#L27-L63","documentation":"JuiceFS marks a disk-cache dir 'unstable' after I/O failures and then throttles access: checkCacheOp returns errUnstableCoLimit when the number of in-flight cache operations has reached maxConcurrencyForUnstable. This is a deliberate back-pressure signal, not corruption — operations against the unstable cache are temporarily limited.","triggerScenarios":"Loading/storing blocks through a DiskCache whose cache dir is in dcUnstable state while maxConcurrencyForUnstable goroutines already hold concurrent cache ops; seen in load()/store paths and reproduced in disk_cache_state_test.go by exhausting the concurrency counter without calling afterCacheOp.","commonSituations":"Slow or degraded cache disk (high latency, intermittent errors) marking the dir unstable under heavy read/write load; test code that drives the state machine without decrementing concurrency; very high client concurrency against a single cache dir.","solutions":["Reduce client read/write concurrency or the number of in-flight block operations hitting the cache.","Check and fix the cache disk (I/O errors, SMART, mount health) so the dir returns to normal state; restart the client to reset state after the disk recovers.","Move the cache to faster/reliable storage, or disable/replace the unstable cache dir.","In tests, always pair state.afterCacheOp() with each op as the test does when priming the limit."],"exampleFix":"// before\n// hammering cache with unbounded goroutines\nfor _, b := range blocks { go load(b) }\n\n// after\nsem := make(chan struct{}, maxConcurrencyForUnstable-1)\nfor _, b := range blocks {\n    sem <- struct{}{}\n    go func(b) { defer func() { <-sem }(); load(b) }(b)\n}","handlingStrategy":"retry","validationCode":"// bound your own concurrency below the limit\nif inflight >= maxConcurrencyForUnstable { backoff() }","typeGuard":null,"tryCatchPattern":"for {\n    _, err := cache.Load(key)\n    if errors.Is(err, errUnstableCoLimit) {\n        time.Sleep(backoff); continue\n    }\n    break\n}","preventionTips":["Fix underlying cache-disk I/O problems promptly so the dir doesn't go unstable","Limit client concurrency against a single cache dir","In tests, always decrement concurrency (afterCacheOp) after each op","Use fast local storage for cache"],"tags":["disk-cache","concurrency","throttling","unstable-disk"],"backgroundTag":"rate-limit-exceeded","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}