juicedata/juicefs · warning

(cancelled) upload block %s: %s (after %d tries)

Error message

(cancelled) upload block %s: %s (after %d tries)

What it means

Upload retries with quadratic backoff (try*try seconds) up to `max` attempts (3, or MaxRetries+1 for sync writes). If the owning slice signals s.uploadError (the upload was cancelled/superseded) during the retry loop, the loop breaks early and reports "(cancelled) upload block ..." with the last error and try count. The block will be handled by a subsequent upload instead.

Source

Thrown at pkg/chunk/cached_store.go:374

	if sync && (blen < store.conf.BlockSize || store.conf.CacheLargeWrite) {
		// block will be freed after written into disk
		store.bcache.cache(key, block, false, false)
	}
	n, err := store.compressor.Compress(buf.Data, block.Data)
	block.Release()
	if err != nil {
		return fmt.Errorf("Compress block key %s: %s", key, err)
	}
	buf.Data = buf.Data[:n]

	try, max := 0, 3
	if sync {
		max = store.conf.MaxRetries + 1
	}
	for ; try < max; try++ {
		time.Sleep(time.Second * time.Duration(try*try))
		if s != nil && s.uploadError != nil {
			err = fmt.Errorf("(cancelled) upload block %s: %s (after %d tries)", key, err, try)
			break
		}
		if err = store.put(ctx, key, buf); err == nil {
			break
		}
		logger.Debugf("Upload %s: %s (try %d)", key, err, try+1)
	}
	if err != nil && try >= max {
		err = fmt.Errorf("(max tries) upload block %s: %s (after %d tries)", key, err, try)
	}
	return err
}

func (s *wSlice) upload(indx int) {
	blen := s.blockSize(indx)
	key := s.key(indx)
	pages := s.pages[indx]
	s.pages[indx] = nil

View on GitHub (pinned to c9a67b23e8)

Solutions

  1. Inspect the wrapped inner error (%s) and try count to see why the upload failed or was cancelled.
  2. Check object storage connectivity/credentials (S3 endpoint reachability, keys, clocks) if the cause is repeated put() failure.
  3. Keep the client mounted until uploads drain (avoid unmount/kill during heavy writes); enable --writeback with sufficient retention.
  4. Increase --max-uploads/retry budget for large or slow uploads and ensure network stability; re-mount to let pending blocks re-upload.

Example fix

// before (client killed during writeback, blocks lost from retry window)
kill <juicefs-pid>
// after (graceful drain)
umount /mnt/jfs  # lets pending uploads finish/flush before process exit
Defensive patterns

Strategy: retry

Validate before calling

// verify object storage reachability before heavy writes
resp, err := http.Head(os.Getenv("JFS_MINIO") + "/minio/health/ready")

Try / catch

if err != nil && strings.Contains(err.Error(), "(cancelled) upload block") {
    // re-mount or wait; pending blocks are re-uploaded on next pass
}

Prevention

When it happens

Trigger: While sleeping/backing off between put() retries, another goroutine sets s.uploadError (slice closed, aborted, or a newer state superseded this upload); also occurs when put() repeatedly fails and the slice is aborted mid-retry — often due to unmount, context cancellation, or network outage lasting past the retry window.

Common situations: Unmounting or killing a client while uploads are in flight; object storage outages (S3 timeouts, 5xx) exhausting retries; writeback mode where the process exits before pending uploads complete.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/09a68000395cb2c6. Report an issue: GitHub.