{"record":{"id":"15be3286e8a1a120","repo":"apache/beam","slug":"average-partition-bytes-size-has-not-been-initialized","errorCode":null,"errorMessage":"Average partition bytes size has not been initialized, GetSize will always return 0, which will interfere with autoscaling.","messagePattern":"Average partition bytes size has not been initialized, GetSize will always return 0, which will interfere with autoscaling\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/DetectNewPartitionsDoFn.java","lineNumber":127,"sourceCode":"  }\n\n  /**\n   * Uses an {@link TimestampRange} with a max range. This is because it does not know beforehand\n   * how many partitions it will schedule.\n   *\n   * @return the timestamp range for the component\n   */\n  @GetInitialRestriction\n  public TimestampRange initialRestriction(@Element PartitionMetadata partition) {\n    final com.google.cloud.Timestamp createdAt = partition.getCreatedAt();\n    return TimestampRange.of(\n        TimestampUtils.previous(createdAt), com.google.cloud.Timestamp.MAX_VALUE);\n  }\n\n  @GetSize\n  public double getSize(@Restriction TimestampRange restriction) {\n    if (!averagePartitionBytesSizeSet) {\n      LOG.warn(\n          \"Average partition bytes size has not been initialized, GetSize will always return 0, which will interfere with autoscaling.\");\n    }\n    final com.google.cloud.Timestamp readTimestamp = restriction.getFrom();\n    final PartitionMetadataDao dao = daoFactory.getPartitionMetadataDao();\n    final long partitionsToSchedule = dao.countPartitionsCreatedAfter(readTimestamp);\n    final long sizeEstimate = partitionsToSchedule * averagePartitionBytesSize;\n\n    LOG.debug(\n        \"getSize() = {} ({} partitionsToSchedule * {} averagePartitionBytesSize)\",\n        sizeEstimate,\n        partitionsToSchedule,\n        averagePartitionBytesSize);\n    return sizeEstimate;\n  }\n\n  @NewTracker\n  public DetectNewPartitionsRangeTracker newTracker(@Restriction TimestampRange restriction) {\n    return new DetectNewPartitionsRangeTracker(restriction);","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/DetectNewPartitionsDoFn.java#L109-L145","documentation":"DetectNewPartitionsDoFn.getSize, annotated with @GetSize, reports estimated work based on averagePartitionBytesSize. When the restriction tracker has not yet initialized this value (averagePartitionBytesSizeSet is false), the estimate is always 0, which breaks Beam Runner V2 autoscaling (the runner sees no work to scale on). The DoFn logs a warning each time GetSize is called in this state; the value returned is 0.","triggerScenarios":"Reading a Spanner change stream via SpannerIO.readChangeStream() on a runner/version where the average partition bytes size is never populated in the restriction tracker before GetSize is invoked — e.g. early in partition detection or when the metadata/dao has not supplied statistics yet.","commonSituations":"Running change-stream pipelines on Flink/Dataflow with autoscaling enabled and observing that the pipeline never scales because work is reported as 0; using a Beam version where DetectNewPartitionsDoFn does not initialize the size estimate from the partition metadata.","solutions":["Upgrade Apache Beam to a version that initializes averagePartitionBytesSize in DetectNewPartitionsDoFn's restriction tracker (the warning was addressed in later releases).","Verify the change stream and partition metadata tables are correctly configured so partition statistics are available.","If autoscaling matters, run on a runner/version where this warning does not appear, or manually fix the number of workers.","Check that the change stream name and start time are valid so partitions are detected and statistics populated."],"exampleFix":"// before: stale Beam version with uninitialized size estimate\nSpannerIO.readChangeStream()\n    .withProjectId(projectId)\n    .withInstanceId(instanceId)\n    .withDatabaseId(databaseId)\n    .withChangeStreamName(streamName);\n\n// after: upgrade to a Beam release where DetectNewPartitionsDoFn\n// initializes averagePartitionBytesSize before GetSize is queried","handlingStrategy":"validation","validationCode":"// At pipeline startup, log the Beam version and check for the fix in DetectNewPartitionsDoFn:\n// grep -r \"averagePartitionBytesSizeSet\" sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dofn/DetectNewPartitionsDoFn.java","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin Apache Beam to a release where averagePartitionBytesSize is initialized in the restriction tracker.","Smoke-test change-stream pipelines with autoscaling enabled and watch for this warning in logs.","Confirm change stream name and metadata tables are configured so partition statistics are populated."],"tags":["java","spanner","changestream","autoscaling"],"backgroundTag":"missing-configuration","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}