juicedata/juicefs · critical

read %s fully: %v (%d < %d) after %s

Error message

read %s fully: %v (%d < %d) after %s

What it means

After downloading a block, load checks that the (optionally decompressed) payload is at least as long as the requested page: if the compressor reports an error or res.n < len(page.Data), it returns "read <key> fully: <err> (<got> < <want>) after <duration>". This indicates the object storage returned truncated or corrupt data — the block could not be read in full, so the caller must not use the page.

Source

Thrown at pkg/chunk/cached_store.go:813

	res := getResult{sc: object.DefaultStorageClass}
	if err == nil {
		res = tmp
	}
	logRequest("GET", key, "", res.reqID, err, used)
	if store.downLimit != nil && compressed {
		store.downLimit.Wait(int64(res.n))
	}
	store.objectDataBytes.WithLabelValues("GET", res.sc).Add(float64(res.n))
	store.objectReqsHistogram.WithLabelValues("GET", res.sc).Observe(used.Seconds())
	if err != nil {
		store.objectReqErrors.Add(1)
		return fmt.Errorf("get %s: %s", key, err)
	}
	if compressed {
		res.n, err = store.compressor.Decompress(page.Data, p.Data[:res.n])
	}
	if err != nil || res.n < len(page.Data) {
		return fmt.Errorf("read %s fully: %v (%d < %d) after %s", key, err, res.n, len(page.Data), used)
	}
	if cache {
		store.bcache.cache(key, page, forceCache, !store.conf.OSCache)
	}
	return nil
}

// NewCachedStore create a cached store.
func NewCachedStore(storage object.ObjectStorage, config Config, reg prometheus.Registerer) ChunkStore {
	compressor := compress.NewCompressor(config.Compress)
	if compressor == nil {
		logger.Fatalf("unknown compress algorithm: %s", config.Compress)
	}
	if config.MaxRetries == 0 {
		config.MaxRetries = 10
	}
	if config.GetTimeout == 0 {
		config.GetTimeout = time.Second * 60

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Delete the offending object (or remove local cache entries) and re-write/re-upload the data — a truncated stored object cannot be repaired in place.
2. Verify the volume's compression setting matches what was used at format time; mismatched compressor causes Decompress errors.
3. Check who else writes to the bucket — conflicting external writers or lifecycle rules corrupting keys must be removed.
4. Re-run the failing upload (e.g. fsck/gc or re-write the file) to repopulate the block from a good copy.
5. If the network is dropping data mid-read, stabilize connectivity or increase retry budget, then retry the read.

Example fix

// before: object truncated by interrupted upload
//   read 0/0/1_0 fully: <nil> (1048576 < 4194304) after 120ms
// after: re-upload the block, then invalidate cache
store.bcache.remove(key)
_ = store.client.Delete(key)
err := slice.upload(...) // rewrites the block from staging

if got, want := len(page.Data), expectedSize; got < want {
    return fmt.Errorf("object %s shorter than expected: %d < %d", key, got, want)
}

if err != nil && strings.HasPrefix(err.Error(), "read ") && strings.Contains(err.Error(), "fully:") {
    // truncated/corrupt object: drop cache entry, delete object, re-read after re-upload
    store.bcache.remove(key)
    return readAfterInvalidation(ctx, key, page)
}

When it happens

Trigger: The stored object is shorter than expected (truncated upload, partial multipart, externally-managed bucket content changed), or Decompress fails on malformed compressed data — surfacing when load reads a page whose backing object yields too few bytes.

Common situations: Buckets written by other tools/humans with colliding keys; a previous upload interrupted leaving a truncated object; bit-rot or wrong compressor setting on the volume (data written with one compression read as another); flaky network causing short reads from an S3-compatible store.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• The entry of the root inode was not found
• ceph: can't put empty file
• GOOGLE_CLOUD_PROJECT environment variable must be set
• object key cannot be empty
• not exists

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/72f423adea3da1c6. Report an issue: GitHub.