apache/iceberg · error · RuntimeException

Metadata file might have been modified. Encryption key id %s

Error message

Metadata file might have been modified. Encryption key id %s differs from HMS value %s

What it means

During refresh, HiveTableOperations compares the Iceberg table's encryption key id (from the metadata file properties) against the value stored in the Hive metastore table parameters. If they differ, the metadata file may have been modified or swapped outside of Iceberg's optimistic concurrency protocol, and a RuntimeException is thrown to prevent reading from untrusted metadata. This integrity check protects Iceberg's encryption key tracking from being bypassed by direct HMS edits.

Source

Thrown at hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:596

      HMSTablePropertyHelper.verifyMetadataHash(metadata, metadataHashFromHMS);
      return;
    }

    LOG.warn(
        "Full metadata integrity check skipped because no metadata hash was recorded in HMS for table {}."
            + " Falling back to encryption property based check.",
        tableName);

    Map<String, String> propertiesFromMetadata = metadata.properties();

    String encryptionKeyIdFromMetadata =
        propertiesFromMetadata.get(TableProperties.ENCRYPTION_TABLE_KEY);
    if (!Objects.equals(encryptionKeyIdFromHMS, encryptionKeyIdFromMetadata)) {
      String errMsg =
          String.format(
              "Metadata file might have been modified. Encryption key id %s differs from HMS value %s",
              encryptionKeyIdFromMetadata, encryptionKeyIdFromHMS);
      throw new RuntimeException(errMsg);
    }

    String dekLengthFromMetadata =
        propertiesFromMetadata.get(TableProperties.ENCRYPTION_DEK_LENGTH);
    if (!Objects.equals(dekLengthFromHMS, dekLengthFromMetadata)) {
      String errMsg =
          String.format(
              "Metadata file might have been modified. DEK length %s differs from HMS value %s",
              dekLengthFromMetadata, dekLengthFromHMS);
      throw new RuntimeException(errMsg);
    }
  }

  @VisibleForTesting
  HiveLock lockObject(TableMetadata metadata) {
    if (hiveLockEnabled(metadata, conf)) {
      return new MetastoreLock(conf, metaClients, catalogName, database, tableName);
    } else {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the HMS table parameters and the current Iceberg metadata file; identify which side was modified and restore consistency via a proper Iceberg commit (never by hand-editing).
If a stale/mismatched metadata file was manually restored, re-commit through Iceberg so HMS metadata_location and encryption properties are rewritten together.
Audit for concurrent writers or external tools (ETL scripts, CLI tools) modifying HMS parameters directly and stop them.
If encryption properties are intentionally not used, ensure they are consistently absent or equal on both sides before refreshing.

Example fix

// before: manually patched metadata file left HMS params stale
// after: refresh through Iceberg after verifying both sides match
Table table = catalog.loadTable(identifier);
table.refresh(); // fails while key id differs
// Fix by re-applying the property through Iceberg:
table.updateProperties().set(TableProperties.ENCRYPTION_TABLE_KEY, correctKeyId).commit();

Defensive patterns

Strategy: validation

Validate before calling

TableMetadata tm = table.operations().current();
String keyIdInMetadata = tm.properties().get(TableProperties.ENCRYPTION_TABLE_KEY);
String keyIdInHms = hmsTable.getParameters().get(TableProperties.ENCRYPTION_TABLE_KEY);
if (!Objects.equals(keyIdInMetadata, keyIdInHms)) {
  throw new IllegalStateException("Encryption key id mismatch between metadata and HMS; reconcile before refresh");
}

Try / catch

try {
  table.refresh();
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("Encryption key id")) {
    // reconcile metadata vs HMS through a proper Iceberg commit, not manual edits
  } else { throw e; }
}

Prevention

Never edit HMS table parameters or Iceberg metadata files manually.
Change encryption properties only through Table.updateProperties().
Audit for external tools writing HMS parameters directly on Iceberg tables.
Keep all writers on consistent Iceberg versions and encryption configuration.

When it happens

Trigger: Calling any table refresh (e.g., table.refresh() or a load that triggers doRefresh) on a table where TableProperties.ENCRYPTION_TABLE_KEY in the Iceberg metadata JSON does not equal the encryption key id recorded in the HMS table parameters, typically because someone modified the metadata file or the HMS parameters out-of-band.

Common situations: Manual edits to HMS table parameters or metadata JSON files; concurrent or divergent writes to encryption properties from different tools; restoring an old metadata file without restoring HMS properties; using non-Iceberg tooling to mutate encrypted tables.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3424186a1e73e36e. Report an issue: GitHub.