{"record":{"id":"22eb5a30cbaa118a","repo":"apache/beam","slug":"entry-with-size-mibs-inserted-into-the-cache-this-is-larger","errorCode":null,"errorMessage":"Entry with size {} MiBs inserted into the cache. This is larger than the maximum individual entry size of {} MiBs. The cache will under report its memory usage by the difference. This may lead to OutOfMemoryErrors.","messagePattern":"Entry with size (.+?) MiBs inserted into the cache\\. This is larger than the maximum individual entry size of (.+?) MiBs\\. The cache will under report its memory usage by the difference\\. This may lead to OutOfMemoryErrors\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/harness/src/main/java/org/apache/beam/fn/harness/Caches.java","lineNumber":213,"sourceCode":"    // We specifically use Guava cache since it allows for recursive computeIfAbsent calls\n    // preventing deadlock from occurring when a loading function mutates the underlying cache\n    LongAdder weightInBytes = new LongAdder();\n    return new SubCache<>(\n        new ShrinkOnEviction(\n                CacheBuilder.newBuilder()\n                    .maximumWeight(maximumBytes >> WEIGHT_RATIO)\n                    .weigher(\n                        new Weigher<CompositeKey, WeightedValue<Object>>() {\n\n                          @Override\n                          public int weigh(CompositeKey key, WeightedValue<Object> value) {\n                            // Since our weights are tracking bytes used, we need to account for the\n                            // cache internal bytes.\n                            long weight = key.getWeight() + value.getWeight() + REFERENCE_SIZE * 15;\n                            // Round up to the next closest multiple of WEIGHT_RATIO\n                            weight = ((weight - 1) >> WEIGHT_RATIO) + 1;\n                            if (weight > Integer.MAX_VALUE) {\n                              LOG.warn(\n                                  \"Entry with size {} MiBs inserted into the cache. This is larger than the maximum individual entry size of {} MiBs. The cache will under report its memory usage by the difference. This may lead to OutOfMemoryErrors.\",\n                                  ((weight - 1) >> 20) + 1,\n                                  2 << (WEIGHT_RATIO + 10));\n                              return Integer.MAX_VALUE;\n                            }\n                            return (int) weight;\n                          }\n                        })\n                    // The maximum size of an entry in the cache is maxWeight / concurrencyLevel\n                    // which is why we set the concurrency level to 1. See\n                    // https://github.com/google/guava/issues/3462 for further details.\n                    //\n                    // The PrecombineGroupingTable showed contention here since it was working in\n                    // a tight loop. We were able to resolve the contention by reducing the\n                    // frequency of updates. Reconsider this value if we could solve the maximum\n                    // entry size issue. Note that using Runtime.getRuntime().availableProcessors()\n                    // is subject to docker CPU shares issues\n                    // (https://bugs.openjdk.org/browse/JDK-8281181).","sourceCodeStart":195,"sourceCodeEnd":231,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/Caches.java#L195-L231","documentation":"Caches.java defines a weight function for cache entries; when the computed weight exceeds Integer.MAX_VALUE, it logs this warning, returns Integer.MAX_VALUE, and the cache under-reports actual memory usage. This can lead to OutOfMemoryErrors because the size-bounded cache believes it holds less than it does.","triggerScenarios":"Inserting a very large key/value (e.g. a huge ProcessWideCache entry whose byte-based weights sum to >2^31 / >2 GiB) into a Cache weighed by Caches.newWeightedCache.","commonSituations":"Caching extremely large byte arrays or entire side-inputs as single entries; jobs processing giant elements with process-wide caching enabled; misestimated Weights for custom objects.","solutions":["Split the large value into smaller entries so each stays under the 2 GiB weight cap.","Reduce what is cached (e.g. stream side inputs instead of materializing them in the process-wide cache).","Increase worker memory as mitigation, but prefer shrinking entries since under-reporting persists.","Review custom Weight implementations to ensure they reflect true byte size."],"exampleFix":"// before\ncache.put(\"big\", hugeList); // > 2GiB single entry -> warn + OOM risk\n// after\nfor (List<T> chunk : partition(hugeList, maxChunkBytes)) {\n  cache.put(nextKey(), chunk); // each entry under the weight cap\n}","handlingStrategy":"validation","validationCode":"long estimatedBytes = key.getWeight() + value.getWeight() + 128;\nif (estimatedBytes > Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\"Cache entry exceeds 2GiB; split it before caching\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never cache multi-GB values as single entries; partition them","Watch worker memory headroom; treat this warning as OOM precursor","Validate custom Weight implementations reflect real byte sizes"],"tags":["java","cache","memory","out-of-memory"],"backgroundTag":"cache-capacity-exceeded","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}