{"record":{"id":"785ef9039edda56f","repo":"apache/beam","slug":"not-all-temporary-files-are-present-for-importing","errorCode":null,"errorMessage":"Not all temporary files are present for importing.","messagePattern":"Not all temporary files are present for importing\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java","lineNumber":1330,"sourceCode":"      }\n\n      @FinishBundle\n      public void importBatch(FinishBundleContext context) throws IOException {\n        // Move files to a temporary subdir (to provide common prefix) to execute import with single\n        // GCS URI and allow for retries.\n        // IGNORE_MISSING_FILES ignores missing source files, we enable this as if this is a retry\n        // files should have already been moved over.\n        FileSystems.rename(\n            ImmutableList.copyOf(files),\n            tempDestinations,\n            StandardMoveOptions.IGNORE_MISSING_FILES);\n        // Even in a retry we need to check that all temporary files are present in the temporary\n        // destination.\n        boolean hasMissingFile =\n            FileSystems.matchResources(tempDestinations).stream()\n                .anyMatch((MatchResult r) -> r.status() != Status.OK);\n        if (hasMissingFile) {\n          throw new IllegalStateException(\"Not all temporary files are present for importing.\");\n        }\n        ResourceId importUri = tempDir.resolve(\"*\", StandardResolveOptions.RESOLVE_FILE);\n\n        try {\n          // Blocking fhirStores.import request.\n          assert contentStructure != null;\n          Operation operation =\n              client.importFhirResource(\n                  fhirStore.get(), importUri.toString(), contentStructure.name());\n          operation = client.pollOperation(operation, 15000L);\n          incrementLroCounters(\n              operation,\n              IMPORT_OPERATION_SUCCESS,\n              IMPORT_OPERATION_ERRORS,\n              RESOURCES_IMPORTED_SUCCESS,\n              RESOURCES_IMPORTED_ERRORS);\n\n          // Clean up temp files on GCS as they were successfully imported to FHIR store and no","sourceCodeStart":1312,"sourceCodeEnd":1348,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java#L1312-L1348","documentation":"Before executing a FHIR store import from GCS, FhirIO verifies that every staged temporary file it expects in the temporary destination actually exists by calling FileSystems.matchResources and checking each MatchResult status. If any match is not Status.OK (missing or unreadable file), an IllegalStateException is thrown to avoid launching an import over an incomplete data set. This also applies on retry paths, since a retried import still needs all files present.","triggerScenarios":"Running FhirIO.Write with a tempDirectory whose staged NDJSON files were deleted or expired (GCS lifecycle rules), or where the Beam worker lacks read permission on the bucket; also on retry when the original staging files were cleaned up between attempts.","commonSituations":"GCS bucket lifecycle policy deleting staged files before the import fires; temp dir shared/overwritten by concurrent pipelines; typo'd or mismatched temp path prefix; service account missing storage.objects.get permission.","solutions":["Verify the temporary directory contents exist (gsutil ls) and re-run the write/export step to re-stage the files","Remove or relax GCS lifecycle/retention rules that delete staged files before the import completes","Grant the pipeline's service account read access to the temp bucket","Use a dedicated, unique tempDirectory per job/run so concurrent runs do not purge each other's files"],"exampleFix":"// before\n.withTempDirectory(\"gs://shared-bucket/tmp/\") // lifecycle: delete after 1 day\n// after\n.withTempDirectory(\"gs://my-bucket/fhir-import/\" + jobId + \"/\") // no lifecycle deletion","handlingStrategy":"validation","validationCode":"boolean allPresent = FileSystems.matchResources(tempDestinations)\n    .stream().allMatch(r -> r.status() == MatchResult.Status.OK);\nif (!allPresent) { /* re-stage files before running import */ }","typeGuard":null,"tryCatchPattern":"try {\n  // run FhirIO import/write\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"temporary files are present\")) {\n    // re-run the staging step and retry the import\n  } else throw e;\n}","preventionTips":["Give each run a unique temp directory to avoid cross-job cleanup","Disable GCS lifecycle rules that delete objects before the import completes","Verify service-account read access on the temp bucket"],"tags":["java","fhir","gcs","import","file-not-found"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}