dgraph-io/badger · error

Request size offset is bigger than maximum offset

Error message

Request size offset %d is bigger than maximum offset %d

What it means

validateWrites estimates each incoming request's size and rejects any request whose cumulative value-log offset would exceed maxVlogFileSize (the hard maximum vlog file size). Such a request could never fit in any value log file, so badger fails it before writing anything.

Solutions

1. Shrink or chunk the large value so no single entry exceeds maxVlogFileSize, or store the blob externally and keep a reference key in badger.
2. Check opt.ValueLogFileSize: it must be <= maxVlogFileSize (default max ~2GB); set it within the supported range.
3. Split large batches into smaller requests; estimateRequestSize counts the whole request.
4. For genuinely huge values, use badger's Stream framework or an external store (S3/filesystem) with badger holding metadata.

Example fix

// before
opt.ValueLogFileSize = 4 << 30 // exceeds maxVlogFileSize
db.Set(key, hugeValue) // Request size offset X is bigger than maximum offset Y
// after
opt.ValueLogFileSize = 1 << 30 // within max
if len(hugeValue) > 100<<20 {
    ref := storeExternal(hugeValue)
    db.Set(key, ref)
} else {
    db.Set(key, hugeValue)
}

// Go: validate request size before writing
const maxVlogFileSize = 2 << 30 // badger hard cap
func validateEntrySize(k, v []byte) error {
    est := len(k) + len(v) + 64 // key + value + header overhead
    if uint64(est) > uint64(maxVlogFileSize) {
        return fmt.Errorf("entry too large: %d bytes", est)
    }
    return nil
}

func isRequestTooLarge(err error) bool {
    return err != nil && strings.Contains(err.Error(), "bigger than maximum offset")
}

err := db.Set(key, value)
if err != nil && isRequestTooLarge(err) {
    ref, err2 := storeExternal(value)
    if err2 != nil { return err2 }
    return db.Set(key, ref)
}

When it happens

Trigger: db.Set/db.Write/txn.Commit with a single entry or batch whose estimated size exceeds maxVlogFileSize — commonly when opt.ValueLogFileSize was set above the allowed maximum, or a single value is enormous (hundreds of MB+).

Common situations: Users raising ValueLogFileSize beyond the hard cap and pushing one giant blob; storing large files/blobs in badger instead of object storage; building huge write batches that exceed the limit in aggregate.

Related errors

• BlockCacheSize should be set since compression/encryption…
• Cannot have 1 compactor. Need at least 2
• Cannot use badger in Disk-less mode with Dir or ValueDir set
• Duplicate file found. Please delete one.
• Empty value: %+v

AI-assisted analysis of dgraph-io/badger@2a001d466f (2026-09-05). Data as JSON: /api/errors/1b3643a57bf2ab5c. Report an issue: GitHub.

Appendix: source

Thrown at value.go:791

	curlf.lock.RUnlock()
	return err
}

func (vlog *valueLog) woffset() uint32 {
	return vlog.writableLogOffset.Load()
}

// validateWrites will check whether the given requests can fit into 4GB vlog file.
// NOTE: 4GB is the maximum size we can create for vlog because value pointer offset is of type
// uint32. If we create more than 4GB, it will overflow uint32. So, limiting the size to 4GB.
func (vlog *valueLog) validateWrites(reqs []*request) error {
	vlogOffset := uint64(vlog.woffset())
	for _, req := range reqs {
		// calculate size of the request.
		size := estimateRequestSize(req)
		estimatedVlogOffset := vlogOffset + size
		if estimatedVlogOffset > uint64(maxVlogFileSize) {
			return fmt.Errorf("Request size offset %d is bigger than maximum offset %d",
				estimatedVlogOffset, maxVlogFileSize)
		}

		if estimatedVlogOffset >= uint64(vlog.opt.ValueLogFileSize) {
			// We'll create a new vlog file if the estimated offset is greater or equal to
			// max vlog size. So, resetting the vlogOffset.
			vlogOffset = 0
			continue
		}
		// Estimated vlog offset will become current vlog offset if the vlog is not rotated.
		vlogOffset = estimatedVlogOffset
	}
	return nil
}

// estimateRequestSize returns the size that needed to be written for the given request.
func estimateRequestSize(req *request) uint64 {
	size := uint64(0)

View on GitHub (pinned to 2a001d466f)