apache/iceberg · warning

Optimistic locking is not available in the environment. Usin

Error message

Optimistic locking is not available in the environment. Using in-memory lock manager. To ensure atomic transaction, please configure a distributed lock manager such as the DynamoDB lock manager.

What it means

A warning from GlueCatalog.initializeLockManager: no custom lock-impl was configured and the Glue client cannot perform optimistic locking (the SET_VERSION_ID operation is a no-op in this environment), so the catalog falls back to an in-memory lock manager. In-memory locking is only process-local, so concurrent commits from different jobs/engines are not safely serialized against each other.

Source

Thrown at aws/src/main/java/org/apache/iceberg/aws/glue/GlueCatalog.java:159

    initialize(
        name,
        properties.get(CatalogProperties.WAREHOUSE_LOCATION),
        new AwsProperties(properties),
        new S3FileIOProperties(properties),
        awsClientFactory.glue(),
        initializeLockManager(properties),
        PropertyUtil.propertyAsBoolean(
            properties,
            CatalogProperties.UNIQUE_TABLE_LOCATION,
            CatalogProperties.UNIQUE_TABLE_LOCATION_DEFAULT));
  }

  private LockManager initializeLockManager(Map<String, String> properties) {
    if (properties.containsKey(CatalogProperties.LOCK_IMPL)) {
      return LockManagers.from(properties);
    } else if (SET_VERSION_ID.isNoop()) {
      LOG.warn(
          "Optimistic locking is not available in the environment. Using in-memory lock manager."
              + " To ensure atomic transaction, please configure a distributed lock manager"
              + " such as the DynamoDB lock manager.");
      return LockManagers.defaultLockManager();
    } else {
      LOG.debug("Using optimistic locking for Glue Data Catalog tables.");
    }
    return null;
  }

  @VisibleForTesting
  void initialize(
      String name,
      String path,
      AwsProperties properties,
      S3FileIOProperties s3Properties,
      GlueClient client,
      LockManager lock,

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Set warehouse/catalog property lock.impl to org.apache.iceberg.aws.dynamodb.DynamoDbLockManager with a DynamoDB lock table for distributed locking
2. Check why SET_VERSION_ID is no-op: verify AWS SDK version supports Glue table version locking and that no environment flag disabled it
3. If truly single-process, the in-memory manager is acceptable — silence the warning by accepting the risk or configuring a lock explicitly
4. Ensure IAM permissions permit the Glue calls that back optimistic locking

Example fix

// before
Map<String, String> props = Map.of("type", "glue");
GlueCatalog catalog = new GlueCatalog();
catalog.initialize("glue", props); // falls back to in-memory lock manager
// after
Map<String, String> props = Map.of(
    "type", "glue",
    CatalogProperties.LOCK_IMPL, "org.apache.iceberg.aws.dynamodb.DynamoDbLockManager",
    "lock.table", "iceberg_lock_table");
catalog.initialize("glue", props);

// warn on startup if lock-impl missing in multi-writer deployments
if (!props.containsKey(CatalogProperties.LOCK_IMPL) && multiWriter) {
  LOG.warn("Distributed lock manager not configured for Glue catalog");
}

When it happens

Trigger: Initializing GlueCatalog without catalog_properties.lock_impl while the AWS SDK/Glue environment does not support the version-id conditional update used for optimistic locking (e.g. Glue without LockTable support or restricted permissions).

Common situations: Multi-cluster or multi-engine deployments where two Spark/Flink jobs commit to the same Glue table concurrently; missing IAM permissions causing the version-id operation to be unavailable; forgetting to configure a DynamoDB LockManager in production.

Understand the failure class

Background: "missing required config value" errors: why libraries refuse to start when a configuration key is empty, unset, or blank — this error's family across 48 libraries.

Related errors

• Cannot find Glue table %s after refresh, maybe another proce
• Cannot commit %s due to unexpected exception
• Fail to acquire lock %s to commit new metadata at %s
• Cannot commit %s because base metadata location '%s' is not
• Cannot commit %s because Glue detected concurrent update

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/b1c81d9cdc146b2a. Report an issue: GitHub.