What causes "Reference already exists" in apache/druid?

Use a unique blob name (include partition/timestamp/uuid) for the write

apache/druid · error · RE

Reference already exists

Error message

Reference already exists

What it means

AzureStorage.getBlockBlobOutputStream refuses to open an output stream when the target blob already exists, throwing "Reference already exists". This enforces no-overwrite semantics for druid storage writes, though the TODO notes StorageConnector#write documents overwrite behavior, so this may change.

Solutions

Use a unique blob name (include partition/timestamp/uuid) for the write
Delete the existing blob first if overwrite is intended
Route writes through an API that documents overwrite semantics, or wait for the noted behavior change
Handle the RE and treat as duplicate-write in caller logic

Example fix

// before
connector.write("segments/idx.zip", data); // fails if exists
// after
String key = "segments/" + id + "-" + System.nanoTime() + ".zip";
connector.write(key, data); // unique reference

Defensive patterns

Strategy: try-catch

Validate before calling

if (blockBlobClient.exists()) { throw new IllegalStateException("blob " + blobName + " already exists"); } // mirror of the library check

Try / catch

try { connector.write(key, data); } catch (RE e) { if ("Reference already exists".equals(e.getMessage())) { /* choose new key or delete existing */ } else throw e; }

Prevention

Generate unique blob keys (uuid/timestamp/partition) for every write
Never reuse output paths across retries without deleting first
Serialize concurrent writers to the same key with external coordination
Prefer an overwrite-capable API if rewrites are legitimate

When it happens

Trigger: Calling getBlockBlobOutputStream (e.g. via StorageConnector.write) for a blobName that already exists in the container — e.g. pushing a segment to an already-used path or writing a report/key twice.

Common situations: Re-running an ingestion/push with the same segment id and version; non-unique output paths (missing unique suffix/timestamp); concurrent writers racing on the same key.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/23f1cbcd17ebda92. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/azure-extensions/src/main/java/org/apache/druid/storage/azure/AzureStorage.java:165

   *
   * @return An OutputStream for writing the blob.
   */
  public OutputStream getBlockBlobOutputStream(
      final String containerName,
      final String blobName,
      @Nullable final Long blockSize,
      @Nullable final Integer maxAttempts
  ) throws BlobStorageException
  {
    final BlockBlobClient blockBlobClient = azureClientFactory
        .getBlobServiceClient(maxAttempts, defaultStorageAccount)
        .createBlobContainerIfNotExists(containerName)
        .getBlobClient(blobName)
        .getBlockBlobClient();

    // TODO based on the usage here, it might be better to overwrite the existing blob instead; that's what StorageConnector#write documents it does
    if (blockBlobClient.exists()) {
      throw new RE("Reference already exists");
    }

    final BlockBlobOutputStreamOptions options = new BlockBlobOutputStreamOptions();
    if (blockSize != null) {
      options.setParallelTransferOptions(new ParallelTransferOptions().setBlockSizeLong(blockSize));
    }

    return blockBlobClient.getBlobOutputStream(options);
  }

  /**
   * Gets the length of the specified block blob.
   *
   * @param containerName The name of the storage container.
   * @param blobName      The name of the blob within the container.
   *
   * @return The length of the blob in bytes.
   */

View on GitHub (pinned to 9b90983fd2)