juicedata/juicefs · error

get %s: %s

Error message

get %s: %s

What it means

cachedStore.load fetches a block from object storage via its store client; when that GET fails, the underlying error is wrapped as "get <key>: <cause>" after recording metrics and incrementing objectReqErrors. It is the generic object-storage read failure surface for the chunk layer: any transport, auth, or remote error downloading a block comes back in this form.

Source

Thrown at pkg/chunk/cached_store.go:807

		return getErr
	}, store.conf.GetTimeout)
	if errors.Is(err, context.Canceled) {
		return err
	}
	used := time.Since(start)
	res := getResult{sc: object.DefaultStorageClass}
	if err == nil {
		res = tmp
	}
	logRequest("GET", key, "", res.reqID, err, used)
	if store.downLimit != nil && compressed {
		store.downLimit.Wait(int64(res.n))
	}
	store.objectDataBytes.WithLabelValues("GET", res.sc).Add(float64(res.n))
	store.objectReqsHistogram.WithLabelValues("GET", res.sc).Observe(used.Seconds())
	if err != nil {
		store.objectReqErrors.Add(1)
		return fmt.Errorf("get %s: %s", key, err)
	}
	if compressed {
		res.n, err = store.compressor.Decompress(page.Data, p.Data[:res.n])
	}
	if err != nil || res.n < len(page.Data) {
		return fmt.Errorf("read %s fully: %v (%d < %d) after %s", key, err, res.n, len(page.Data), used)
	}
	if cache {
		store.bcache.cache(key, page, forceCache, !store.conf.OSCache)
	}
	return nil
}

// NewCachedStore create a cached store.
func NewCachedStore(storage object.ObjectStorage, config Config, reg prometheus.Registerer) ChunkStore {
	compressor := compress.NewCompressor(config.Compress)
	if compressor == nil {
		logger.Fatalf("unknown compress algorithm: %s", config.Compress)

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Read the wrapped cause after 'get <key>:' — it carries the specific storage/client error to fix.
2. Verify the object still exists in the bucket (404 ⇒ data was deleted or written by a client with different storage config).
3. Refresh or fix storage credentials (expired STS tokens, rotated keys) and remount.
4. Check network path to the endpoint (DNS, proxy, security groups) and retry once connectivity is restored.
5. If reads are failing under load, enable/rely on the local cache and increase `--max-downloads` headroom or check provider rate limits.

Example fix

// before: expired credentials
//   get 0/0/1_0 myobject: AccessDenied ...
// after: refresh credentials before mount
//   aws s3 cp fails too => update keys, then
//   juicefs mount --storage s3 ... (with valid credentials)

if err := pingObjectStorage(ctx); err != nil {
    return fmt.Errorf("storage endpoint unreachable before read: %w", err)
}

var serr *StorageError
if errors.As(err, &serr) || strings.HasPrefix(err.Error(), "get ") {
    if isRetryable(serr) {
        return withBackoff(func() error { return readBlock(ctx, key, page) }, 3)
    }
    return err // non-retryable (404/403): surface to caller
}

When it happens

Trigger: Any GET issued by load against the object store that returns an error: network timeout, HTTP 403/404/5xx from S3-compatible storage, expired credentials, bucket/object deleted between lookup and fetch, or client-side connection reset.

Common situations: Object removed from the bucket by lifecycle rules or manual deletion while the client still expects it; IAM/STS credentials expiring mid-session; storage endpoint outage; proxy or firewall blocking egress in corporate/Kubernetes environments.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

• Failed to create bucket %s: %s, previous error: %s
• (max tries) upload block %s: %s (after %d tries)
• Failed to put: %s
• Failed to create bucket %s: %s, previous error: %s Please cr
• Failed to get: %s

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/889645e15533e209. Report an issue: GitHub.