apache/beam · error · IllegalStateException

Not all temporary files are present for importing.

Error message

Not all temporary files are present for importing.

What it means

Before executing a FHIR store import from GCS, FhirIO verifies that every staged temporary file it expects in the temporary destination actually exists by calling FileSystems.matchResources and checking each MatchResult status. If any match is not Status.OK (missing or unreadable file), an IllegalStateException is thrown to avoid launching an import over an incomplete data set. This also applies on retry paths, since a retried import still needs all files present.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java:1330

      }

      @FinishBundle
      public void importBatch(FinishBundleContext context) throws IOException {
        // Move files to a temporary subdir (to provide common prefix) to execute import with single
        // GCS URI and allow for retries.
        // IGNORE_MISSING_FILES ignores missing source files, we enable this as if this is a retry
        // files should have already been moved over.
        FileSystems.rename(
            ImmutableList.copyOf(files),
            tempDestinations,
            StandardMoveOptions.IGNORE_MISSING_FILES);
        // Even in a retry we need to check that all temporary files are present in the temporary
        // destination.
        boolean hasMissingFile =
            FileSystems.matchResources(tempDestinations).stream()
                .anyMatch((MatchResult r) -> r.status() != Status.OK);
        if (hasMissingFile) {
          throw new IllegalStateException("Not all temporary files are present for importing.");
        }
        ResourceId importUri = tempDir.resolve("*", StandardResolveOptions.RESOLVE_FILE);

        try {
          // Blocking fhirStores.import request.
          assert contentStructure != null;
          Operation operation =
              client.importFhirResource(
                  fhirStore.get(), importUri.toString(), contentStructure.name());
          operation = client.pollOperation(operation, 15000L);
          incrementLroCounters(
              operation,
              IMPORT_OPERATION_SUCCESS,
              IMPORT_OPERATION_ERRORS,
              RESOURCES_IMPORTED_SUCCESS,
              RESOURCES_IMPORTED_ERRORS);

          // Clean up temp files on GCS as they were successfully imported to FHIR store and no

View on GitHub (pinned to 12126d8942)

Solutions

Verify the temporary directory contents exist (gsutil ls) and re-run the write/export step to re-stage the files
Remove or relax GCS lifecycle/retention rules that delete staged files before the import completes
Grant the pipeline's service account read access to the temp bucket
Use a dedicated, unique tempDirectory per job/run so concurrent runs do not purge each other's files

Example fix

// before
.withTempDirectory("gs://shared-bucket/tmp/") // lifecycle: delete after 1 day
// after
.withTempDirectory("gs://my-bucket/fhir-import/" + jobId + "/") // no lifecycle deletion

Defensive patterns

Strategy: validation

Validate before calling

boolean allPresent = FileSystems.matchResources(tempDestinations)
    .stream().allMatch(r -> r.status() == MatchResult.Status.OK);
if (!allPresent) { /* re-stage files before running import */ }

Try / catch

try {
  // run FhirIO import/write
} catch (IllegalStateException e) {
  if (e.getMessage().contains("temporary files are present")) {
    // re-run the staging step and retry the import
  } else throw e;
}

Prevention

Give each run a unique temp directory to avoid cross-job cleanup
Disable GCS lifecycle rules that delete objects before the import completes
Verify service-account read access on the temp bucket

When it happens

Trigger: Running FhirIO.Write with a tempDirectory whose staged NDJSON files were deleted or expired (GCS lifecycle rules), or where the Beam worker lacks read permission on the bucket; also on retry when the original staging files were cleaned up between attempts.

Common situations: GCS bucket lifecycle policy deleting staged files before the import fires; temp dir shared/overwritten by concurrent pipelines; typo'd or mismatched temp path prefix; service account missing storage.objects.get permission.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/785ef9039edda56f. Report an issue: GitHub.