{"record":{"id":"09a68000395cb2c6","repo":"juicedata/juicefs","slug":"cancelled-upload-block-s-s-after-d-tries","errorCode":null,"errorMessage":"(cancelled) upload block %s: %s (after %d tries)","messagePattern":"\\(cancelled\\) upload block (.+?): (.+?) \\(after (.+?) tries\\)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/chunk/cached_store.go","lineNumber":374,"sourceCode":"\tif sync && (blen < store.conf.BlockSize || store.conf.CacheLargeWrite) {\n\t\t// block will be freed after written into disk\n\t\tstore.bcache.cache(key, block, false, false)\n\t}\n\tn, err := store.compressor.Compress(buf.Data, block.Data)\n\tblock.Release()\n\tif err != nil {\n\t\treturn fmt.Errorf(\"Compress block key %s: %s\", key, err)\n\t}\n\tbuf.Data = buf.Data[:n]\n\n\ttry, max := 0, 3\n\tif sync {\n\t\tmax = store.conf.MaxRetries + 1\n\t}\n\tfor ; try < max; try++ {\n\t\ttime.Sleep(time.Second * time.Duration(try*try))\n\t\tif s != nil && s.uploadError != nil {\n\t\t\terr = fmt.Errorf(\"(cancelled) upload block %s: %s (after %d tries)\", key, err, try)\n\t\t\tbreak\n\t\t}\n\t\tif err = store.put(ctx, key, buf); err == nil {\n\t\t\tbreak\n\t\t}\n\t\tlogger.Debugf(\"Upload %s: %s (try %d)\", key, err, try+1)\n\t}\n\tif err != nil && try >= max {\n\t\terr = fmt.Errorf(\"(max tries) upload block %s: %s (after %d tries)\", key, err, try)\n\t}\n\treturn err\n}\n\nfunc (s *wSlice) upload(indx int) {\n\tblen := s.blockSize(indx)\n\tkey := s.key(indx)\n\tpages := s.pages[indx]\n\ts.pages[indx] = nil","sourceCodeStart":356,"sourceCodeEnd":392,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/pkg/chunk/cached_store.go#L356-L392","documentation":"Upload retries with quadratic backoff (try*try seconds) up to `max` attempts (3, or MaxRetries+1 for sync writes). If the owning slice signals s.uploadError (the upload was cancelled/superseded) during the retry loop, the loop breaks early and reports \"(cancelled) upload block ...\" with the last error and try count. The block will be handled by a subsequent upload instead.","triggerScenarios":"While sleeping/backing off between put() retries, another goroutine sets s.uploadError (slice closed, aborted, or a newer state superseded this upload); also occurs when put() repeatedly fails and the slice is aborted mid-retry — often due to unmount, context cancellation, or network outage lasting past the retry window.","commonSituations":"Unmounting or killing a client while uploads are in flight; object storage outages (S3 timeouts, 5xx) exhausting retries; writeback mode where the process exits before pending uploads complete.","solutions":["Inspect the wrapped inner error (%s) and try count to see why the upload failed or was cancelled.","Check object storage connectivity/credentials (S3 endpoint reachability, keys, clocks) if the cause is repeated put() failure.","Keep the client mounted until uploads drain (avoid unmount/kill during heavy writes); enable --writeback with sufficient retention.","Increase --max-uploads/retry budget for large or slow uploads and ensure network stability; re-mount to let pending blocks re-upload."],"exampleFix":"// before (client killed during writeback, blocks lost from retry window)\nkill <juicefs-pid>\n// after (graceful drain)\numount /mnt/jfs  # lets pending uploads finish/flush before process exit","handlingStrategy":"retry","validationCode":"// verify object storage reachability before heavy writes\nresp, err := http.Head(os.Getenv(\"JFS_MINIO\") + \"/minio/health/ready\")","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"(cancelled) upload block\") {\n    // re-mount or wait; pending blocks are re-uploaded on next pass\n}","preventionTips":["Avoid unmounting/killing clients while uploads are in flight","Set adequate retry/timeout budgets for slow object storage","Alert on object storage 5xx/latency so outages are caught early"],"tags":["upload","retry","cancellation","network"],"backgroundTag":"request-timeout","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}