Entry with size MiBs inserted into the cache. This is…

Error message

Entry with size {} MiBs inserted into the cache. This is larger than the maximum individual entry size of {} MiBs. The cache will under report its memory usage by the difference. This may lead to OutOfMemoryErrors.

What it means

Caches.java defines a weight function for cache entries; when the computed weight exceeds Integer.MAX_VALUE, it logs this warning, returns Integer.MAX_VALUE, and the cache under-reports actual memory usage. This can lead to OutOfMemoryErrors because the size-bounded cache believes it holds less than it does.

Solutions

Split the large value into smaller entries so each stays under the 2 GiB weight cap.
Reduce what is cached (e.g. stream side inputs instead of materializing them in the process-wide cache).
Increase worker memory as mitigation, but prefer shrinking entries since under-reporting persists.
Review custom Weight implementations to ensure they reflect true byte size.

Example fix

// before
cache.put("big", hugeList); // > 2GiB single entry -> warn + OOM risk
// after
for (List<T> chunk : partition(hugeList, maxChunkBytes)) {
  cache.put(nextKey(), chunk); // each entry under the weight cap
}

Defensive patterns

Strategy: validation

Validate before calling

long estimatedBytes = key.getWeight() + value.getWeight() + 128;
if (estimatedBytes > Integer.MAX_VALUE) {
  throw new IllegalArgumentException("Cache entry exceeds 2GiB; split it before caching");
}

Prevention

Never cache multi-GB values as single entries; partition them
Watch worker memory headroom; treat this warning as OOM precursor
Validate custom Weight implementations reflect real byte sizes

When it happens

Trigger: Inserting a very large key/value (e.g. a huge ProcessWideCache entry whose byte-based weights sum to >2^31 / >2 GiB) into a Cache weighed by Caches.newWeightedCache.

Common situations: Caching extremely large byte arrays or entire side-inputs as single entries; jobs processing giant elements with process-wide caching enabled; misestimated Weights for custom objects.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/22eb5a30cbaa118a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/harness/src/main/java/org/apache/beam/fn/harness/Caches.java:213

    // We specifically use Guava cache since it allows for recursive computeIfAbsent calls
    // preventing deadlock from occurring when a loading function mutates the underlying cache
    LongAdder weightInBytes = new LongAdder();
    return new SubCache<>(
        new ShrinkOnEviction(
                CacheBuilder.newBuilder()
                    .maximumWeight(maximumBytes >> WEIGHT_RATIO)
                    .weigher(
                        new Weigher<CompositeKey, WeightedValue<Object>>() {

                          @Override
                          public int weigh(CompositeKey key, WeightedValue<Object> value) {
                            // Since our weights are tracking bytes used, we need to account for the
                            // cache internal bytes.
                            long weight = key.getWeight() + value.getWeight() + REFERENCE_SIZE * 15;
                            // Round up to the next closest multiple of WEIGHT_RATIO
                            weight = ((weight - 1) >> WEIGHT_RATIO) + 1;
                            if (weight > Integer.MAX_VALUE) {
                              LOG.warn(
                                  "Entry with size {} MiBs inserted into the cache. This is larger than the maximum individual entry size of {} MiBs. The cache will under report its memory usage by the difference. This may lead to OutOfMemoryErrors.",
                                  ((weight - 1) >> 20) + 1,
                                  2 << (WEIGHT_RATIO + 10));
                              return Integer.MAX_VALUE;
                            }
                            return (int) weight;
                          }
                        })
                    // The maximum size of an entry in the cache is maxWeight / concurrencyLevel
                    // which is why we set the concurrency level to 1. See
                    // https://github.com/google/guava/issues/3462 for further details.
                    //
                    // The PrecombineGroupingTable showed contention here since it was working in
                    // a tight loop. We were able to resolve the contention by reducing the
                    // frequency of updates. Reconsider this value if we could solve the maximum
                    // entry size issue. Note that using Runtime.getRuntime().availableProcessors()
                    // is subject to docker CPU shares issues
                    // (https://bugs.openjdk.org/browse/JDK-8281181).

View on GitHub (pinned to 12126d8942)