apache/beam · error · IOException

This filename is already in use.

Error message

This filename is already in use.

What it means

AzureBlobStoreFileSystem.create() checks whether the target blob already exists before opening an output stream, because Azure's getBlobOutputStream silently overwrites existing blobs. To prevent silent data loss, it throws this IOException when the destination filename is already taken. It is a deliberate safety guard against overwrite-on-create semantics.

Solutions

1. Delete the existing blob first (via FileSystem.delete or Azure portal/CLI) if overwrite is intended.
2. Choose a new destination filename or add a unique suffix/timestamp to the output path.
3. Copy the existing blob aside (backup) before recreating at the same name.
4. Check existence beforehand with FileSystem.match() and branch on the result.

Example fix

// before
FileSystem fnFs = FileSystems.matchNewResource("azfs://container/existing/blob.txt", false);
FileSystems.create(fnFs, CreateOptions.StandardCreateOptions.builder().build()); // throws if exists
// after
ResourceId resourceId = FileSystems.matchNewResource("azfs://container/unique-run-2026-09-12/blob.txt", false);
FileSystems.create(resourceId, CreateOptions.StandardCreateOptions.builder().build());

ResourceId dst = FileSystems.matchNewResource(path, false);
MatchResult m = FileSystems.match(path);
if (m.status() == MatchResult.Status.OK && !m.metadata().isEmpty()) {
  throw new IllegalStateException("Destination already exists: " + path);
}

try {
  FileSystems.create(dst, createOptions);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("already in use")) {
    // switch to a unique name or handle overwrite explicitly
  } else { throw e; }
}

When it happens

Trigger: Calling FileSystem.create() (or Match/Write via the Beam filesystem layer) on an azfs:// path whose blob name already exists in the target container.

Common situations: Re-running a pipeline or job that writes to the same azfs output path without unique naming (e.g. missing timestamp/shard in output prefix); two concurrent writers racing to create the same blob; a user assuming create() behaves like put with overwrite.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• The copy source does not exist.
• The requested file doesn't exist.
• The resource to delete does not exist.
• Basepath %r must be an Azure Blob Storage path.
• delete does not delete containers.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/947315f6e358219d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/azure/src/main/java/org/apache/beam/sdk/io/azure/blobstore/AzureBlobStoreFileSystem.java:282

                blobProperties.getContentEncoding(),
                blobProperties.getETag())));
  }

  @Override
  protected WritableByteChannel create(AzfsResourceId resourceId, CreateOptions createOptions)
      throws IOException {
    BlobContainerClient blobContainerClient =
        client.get().getBlobContainerClient(resourceId.getContainer());
    if (!blobContainerClient.exists()) {
      throw new FileNotFoundException(
          "This container does not exist. Creating containers is not supported.");
    }

    BlobClient blobClient = blobContainerClient.getBlobClient(resourceId.getBlob());
    // The getBlobOutputStream method overwrites existing blobs,
    // so throw an error in this case to prevent data loss
    if (blobClient.exists()) {
      throw new IOException("This filename is already in use.");
    }

    OutputStream outputStream;
    try {
      outputStream = blobClient.getBlockBlobClient().getBlobOutputStream();
    } catch (BlobStorageException e) {
      throw (IOException) e.getCause();
    }
    return newChannel(outputStream);
  }

  @Override
  protected ReadableByteChannel open(AzfsResourceId resourceId) throws IOException {
    BlobContainerClient containerClient =
        client.get().getBlobContainerClient(resourceId.getContainer());
    if (!containerClient.exists()) {
      throw new FileNotFoundException("The requested file doesn't exist.");
    }

View on GitHub (pinned to 12126d8942)