{"record":{"id":"a6d0a35555caf8d5","repo":"apache/beam","slug":"lazily-observed-byte-size-will-be-under-reported-due-to","errorCode":null,"errorMessage":"Lazily observed byte size will be under reported due to exception","messagePattern":"Lazily observed byte size will be under reported due to exception","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateBackedIterable.java","lineNumber":176,"sourceCode":"      try {\n        boolean cheap = elementCoder.isRegisterByteSizeObserverCheap(value);\n        if (cheap || sampleElement()) {\n          observerProxy.setScalingFactor(\n              cheap ? 1.0 : Math.max(samplingToken, SAMPLING_CUTOFF) / (double) SAMPLING_CUTOFF);\n          elementCoder.registerByteSizeObserver(value, observerProxy);\n          if (observerProxy.getIsLazy()) {\n            // The observer will only be notified of bytes as the result\n            // is used. We defer advancing the observer until hasNext in an\n            // attempt to capture those bytes.\n            observerNeedsAdvance = true;\n          } else {\n            observerNeedsAdvance = false;\n            observerProxy.advance();\n          }\n        }\n      } catch (Exception e) {\n        if (!exceptionLogged) {\n          LOG.warn(\"Lazily observed byte size will be under reported due to exception\", e);\n          exceptionLogged = true;\n        }\n      }\n      return value;\n    }\n\n    @Override\n    public void remove() {\n      super.remove();\n    }\n  }\n\n  @Override\n  protected ElementByteSizeObservableIterator<T> createIterator() {\n    return WrappedObservingIterator.create(\n        PrefetchableIterators.concat(prefix.iterator(), suffix.iterator()), elemCoder);\n  }\n","sourceCodeStart":158,"sourceCodeEnd":194,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/state/StateBackedIterable.java#L158-L194","documentation":"StateBackedIterable.next() advances the underlying state-backed observer to lazily materialize elements and estimate byte size; if advancing throws, the value is still returned but the lazily observed byte size is skipped and this warning is logged once. Data is not lost — only the size accounting is under reported.","triggerScenarios":"The state read call (observer advance) against the runner/state service throws (network error, state not found, cancelled stream) while lazily measuring element byte size during iteration.","commonSituations":"Unstable connection between harness and runner state service; iterables backed by state that expire or become unavailable; high-volume pipelines relying on byte-size accounting for autoscaling metrics.","solutions":["Check the logged cause for state read failures and fix connectivity to the runner state service.","Increase state read timeouts / retry budgets on the gRPC channel.","If metrics accuracy matters, ensure state-backed iterables are fully consumable (don't abort bundles mid-iteration).","Verify the runner didn't garbage-collect or invalidate the state token backing the iterable."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  observerProxy.advance();\n} catch (Exception e) {\n  if (!exceptionLogged) { LOG.warn(\"Lazily observed byte size will be under reported\", e); exceptionLogged = true; }\n}","preventionTips":["Keep the harness-runner state channel healthy (timeouts, retries)","Avoid aborting bundles mid-iteration over state-backed iterables","Confirm state tokens aren't expiring during long iterations","Accept minor size-metric inaccuracy when this warning is rare"],"tags":["java","apache-beam","state","metrics"],"backgroundTag":"state-read-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}