{"record":{"id":"a7d34dac2c52d814","repo":"juicedata/juicefs","slug":"max-tries-upload-block-s-s-after-d-tries","errorCode":null,"errorMessage":"(max tries) upload block %s: %s (after %d tries)","messagePattern":"\\(max tries\\) upload block (.+?): (.+?) \\(after (.+?) tries\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/chunk/cached_store.go","lineNumber":383,"sourceCode":"\tbuf.Data = buf.Data[:n]\n\n\ttry, max := 0, 3\n\tif sync {\n\t\tmax = store.conf.MaxRetries + 1\n\t}\n\tfor ; try < max; try++ {\n\t\ttime.Sleep(time.Second * time.Duration(try*try))\n\t\tif s != nil && s.uploadError != nil {\n\t\t\terr = fmt.Errorf(\"(cancelled) upload block %s: %s (after %d tries)\", key, err, try)\n\t\t\tbreak\n\t\t}\n\t\tif err = store.put(ctx, key, buf); err == nil {\n\t\t\tbreak\n\t\t}\n\t\tlogger.Debugf(\"Upload %s: %s (try %d)\", key, err, try+1)\n\t}\n\tif err != nil && try >= max {\n\t\terr = fmt.Errorf(\"(max tries) upload block %s: %s (after %d tries)\", key, err, try)\n\t}\n\treturn err\n}\n\nfunc (s *wSlice) upload(indx int) {\n\tblen := s.blockSize(indx)\n\tkey := s.key(indx)\n\tpages := s.pages[indx]\n\ts.pages[indx] = nil\n\ts.pendings++\n\n\tgo func() {\n\t\tvar block *Page\n\t\tvar off int\n\t\tif len(pages) == 1 {\n\t\t\tblock = pages[0]\n\t\t\toff = len(block.Data)\n\t\t} else {","sourceCodeStart":365,"sourceCodeEnd":401,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/chunk/cached_store.go#L365-L401","documentation":"cached_store's block uploader retries `store.put` (upload of a chunk block to object storage) up to `max` times; if every attempt fails, it wraps the last underlying error as \"(max tries) upload block <key>: <cause> (after <N> tries)\". It signals that a block could not be persisted to the object store despite internal retries, so the write path (uploadStagingFile -> upload) aborts and the pending slice is left un-uploaded.","triggerScenarios":"Calling the write path (wSlice upload via uploadStagingFile) while every `store.put` attempt to object storage fails: storage endpoint unreachable, expired/invalid credentials, bucket missing, per-request throttling, or a client whose retry budget (max tries) is exhausted within one call.","commonSituations":"Object storage outage or network partition during heavy writes; misconfigured S3 credentials or wrong endpoint in the mount config; rate limiting from the provider under parallel upload load; DNS or proxy problems in containerized/Kubernetes deployments.","solutions":["Check connectivity and credentials to the object storage endpoint (e.g. `juicefs status <meta-url>` or aws cli against the same bucket).","Inspect the wrapped cause after 'upload block <key>:' — it contains the actual storage error (403, timeout, DNS, etc.) and fix that root cause.","Increase upload retry tolerance by lowering upload pressure (reduce `--upload-limit`, `--max-uploads`) or fixing throttling on the provider side.","Verify bucket name/region/endpoint configuration and that the bucket exists and is writable.","If failures persist transiently, remount or restart the client so the pending slice is re-uploaded after the storage recovers."],"exampleFix":"// before: storage endpoint misconfigured\n//   juicefs mount --storage-url error ...\n// after: verify and correct storage config before mounting\n//   juicefs format --storage s3 --bucket https://s3.us-east-1.amazonaws.com/mybucket ...\n//   juicefs mount redis://localhost/1 /mnt/jfs","handlingStrategy":"retry","validationCode":"if err := storeHealthCheck(ctx); err != nil {\n    return fmt.Errorf(\"object storage unreachable, deferring upload: %w\", err)\n}","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"(max tries) upload block\") {\n    // storage persistently unavailable; schedule retry with backoff\n    time.Sleep(backoff)\n    return retryUpload(key, buf)\n}","preventionTips":["Monitor object-storage availability and credentials before/at mount time with `juicefs status`.","Keep local disk staging sized to absorb writes during storage outages.","Alert on objectReqErrors metrics (this path increments them) to catch endpoint problems early.","Avoid aggressive parallel upload settings that trip provider rate limits."],"tags":["object-storage","upload","retry-exhausted","network","s3"],"backgroundTag":"network-request-failed","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}