apache/iceberg · critical · ValidationException

Can't index multiple DVs for

Error message

Can't index multiple DVs for %s: %s and %s

What it means

DeleteFileIndex.Builder.add refuses to index more than one deletion vector (DV) for the same referenced data file path. Iceberg allows at most one DV per data file; `putIfAbsent` returns the previously stored DV when a second one maps to the same path, and the builder throws ValidationException naming both files. This guards the commit path against producing a snapshot whose DV index is ambiguous.

Solutions

1. Inspect the reported DV descriptors (path, content offsets) and remove the duplicate delete file so only one DV per data file remains before committing.
2. Ensure the writing engine (Spark/Flink) does not re-run the same rewrite/compaction plan concurrently against the same data files; use retries with snapshot conflict re-planning.
3. If a DV was written twice at different offsets, keep the newest one and expire/delete the stale DV file, then re-run the operation.
4. Check for double-adding the same DeleteFile in custom catalog/commit code that calls DeleteFileIndex.Builder.

Example fix

// before: appending a DV without checking the data file already has one
indexBuilder.add(dv);
// after: guard by tracking DVs per referenced data file
if (dvPaths.add(dv.referencedDataFile())) {
  indexBuilder.add(dv);
} else {
  throw new IllegalStateException("DV already indexed for " + dv.referencedDataFile());
}

java.util.Set<String> seen = new HashSet<>();
for (DeleteFile dv : candidateDvs) {
  if (!seen.add(dv.referencedDataFile())) {
    throw new IllegalStateException("Duplicate DV for data file: " + dv.referencedDataFile());
  }
}

try {
  append.commit();
} catch (ValidationException e) {
  if (e.getMessage().startsWith("Can't index multiple DVs")) {
    // re-plan the operation against the latest snapshot, dropping the duplicate DV
  } else throw e;
}

When it happens

Trigger: Adding two deletion vectors whose `referencedDataFile()` returns the same path, e.g. building a snapshot where an overwrite/rewrite operation re-registered a DV for a data file that already has one, or re-adding the same DV delete file to the index builder twice via different AddDeleteFiles calls.

Common situations: Concurrent or duplicated compaction/rewrite jobs both producing DVs for the same data file; a buggy writer emitting the same DV path twice; replaying or re-applying delete files when constructing a snapshot; manually assembled table metadata referencing duplicate DVs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Cannot commit because Glue encountered a validation…
• Bitmap decoding has not been implemented
• Cannot add fields to map keys:
• Cannot alter map keys:
• Cannot build StorageCredential, some of required attributes…

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/8682bf54dac0dacc. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/DeleteFileIndex.java:572

            throw new UnsupportedOperationException("Unsupported content: " + file.content());
        }
        ScanMetricsUtil.indexedDeleteFile(scanMetrics, file);
      }

      return new DeleteFileIndex(
          globalDeletes.isEmpty() ? null : globalDeletes,
          eqDeletesByPartition.isEmpty() ? null : eqDeletesByPartition,
          posDeletesByPartition.isEmpty() ? null : posDeletesByPartition,
          posDeletesByPath.isEmpty() ? null : posDeletesByPath,
          dvByPath.isEmpty() ? null : dvByPath,
          specsById.isEmpty() ? null : specsById);
    }

    private void add(Map<String, DeleteFile> dvByPath, DeleteFile dv) {
      String path = dv.referencedDataFile();
      DeleteFile existingDV = dvByPath.putIfAbsent(path, dv);
      if (existingDV != null) {
        throw new ValidationException(
            "Can't index multiple DVs for %s: %s and %s",
            path, ContentFileUtil.dvDesc(dv), ContentFileUtil.dvDesc(existingDV));
      }
    }

    private void add(
        Map<String, PositionDeletes> deletesByPath,
        PartitionMap<PositionDeletes> deletesByPartition,
        DeleteFile file) {
      String path = ContentFileUtil.referencedDataFileLocation(file);

      PositionDeletes deletes;
      if (path != null) {
        deletes = deletesByPath.computeIfAbsent(path, ignored -> new PositionDeletes());
      } else {
        int specId = file.specId();
        StructLike partition = file.partition();
        deletes = deletesByPartition.computeIfAbsent(specId, partition, PositionDeletes::new);

View on GitHub (pinned to 86d9c8fc54)