{"record":{"id":"38de9c4e0c8b812e","repo":"apache/beam","slug":"encountered-exceptions-in-finalize-write-s","errorCode":null,"errorMessage":"Encountered exceptions in finalize_write: %s","messagePattern":"Encountered exceptions in finalize_write: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/filebasedsink.py","lineNumber":431,"sourceCode":"              exceptions.append(exception)\n            else:\n              _LOGGER.debug('Rename successful: %s -> %s', src, dst)\n          return exceptions\n\n      if w is None or isinstance(w, window.GlobalWindow):\n        # bounded input was handled by finalize_write legacy method\n        # the implementation here should be called by finalize_write\n        # Use a thread pool for renaming operations.\n        exception_batches = util.run_using_threadpool(\n            _rename_batch,\n            list(zip(source_file_batch, destination_file_batch)),\n            num_threads)\n\n        all_exceptions = [\n            e for exception_batch in exception_batches for e in exception_batch\n        ]\n        if all_exceptions:\n          raise Exception(\n              'Encountered exceptions in finalize_write: %s' % all_exceptions)\n\n        yield from dst_files\n      else:\n        # unbounded input\n        batch = list([src_files, dst_files])\n        exception_batches = _rename_batch(batch)\n\n        all_exceptions = [\n            e for exception_batch in exception_batches for e in exception_batch\n        ]\n        if all_exceptions:\n          raise Exception(\n              'Encountered exceptions in finalize_write: %s' % all_exceptions)\n\n        yield from dst_files\n\n      _LOGGER.info(","sourceCodeStart":413,"sourceCodeEnd":449,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/filebasedsink.py#L413-L449","documentation":"finalize_windowed_write executes shard renames in a thread pool; all worker exceptions are collected and, if any occurred, re-raised as a single generic Exception listing them. It signals that one or more temp-shard renames failed during finalization of windowed output.","triggerScenarios":"Any _rename_batch failure in the multi-threaded rename path: missing temp files, permission errors, transient GCS 5xx responses, or rename collisions when dst already exists and src differs.","commonSituations":"GCS throttling/outages during finalize; concurrent pipelines renaming the same output; bucket permissions that allow create but not delete of temp files.","solutions":["Inspect the nested all_exceptions list in the message for the root cause of each failed rename","Rerun the pipeline; transient storage errors during rename typically succeed on retry","Verify the writer account has permission to move/delete objects in the output bucket","Ensure no other job is writing to the same file_path_prefix concurrently"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# pre-check rename permissions\nFileSystems.match(['gs://bucket/output/*'])","typeGuard":null,"tryCatchPattern":"try:\n    result = pipeline.run()\n    result.wait_until_finish()\nexcept Exception as e:\n    if 'Encountered exceptions in finalize_write' in str(e):\n        log.error('rename failures: %s', e)  # then retry pipeline","preventionTips":["Ensure the service account can move/delete objects in the output bucket","Avoid concurrent pipelines sharing a file_path_prefix","Set up alerts on storage API error rates during job finalization"],"tags":["python","apache-beam","io","rename","concurrency"],"backgroundTag":"api-error-response","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}