{"record":{"id":"1d6e1195de8d1ead","repo":"apache/iceberg","slug":"failed-to-unlock-even-on-2nd-attempt","errorCode":null,"errorMessage":"Failed to unlock even on 2nd attempt {}.{}","messagePattern":"Failed to unlock even on 2nd attempt (.+?)\\.(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java","lineNumber":417,"sourceCode":"          id = lockInfo.lockId;\n        } else {\n          LOG.warn(\"Could not find lock with HMSClient {}\", HiveVersion.current());\n          return;\n        }\n      } else {\n        id = lockId.get();\n      }\n\n      doUnlock(id);\n    } catch (InterruptedException ie) {\n      if (id != null) {\n        // Interrupted unlock. We try to unlock one more time if we have a lockId\n        try {\n          Thread.interrupted(); // Clear the interrupt status flag for now, so we can retry unlock\n          LOG.warn(\"Interrupted unlock we try one more time {}.{}\", databaseName, tableName, ie);\n          doUnlock(id);\n        } catch (Exception e) {\n          LOG.warn(\"Failed to unlock even on 2nd attempt {}.{}\", databaseName, tableName, e);\n        } finally {\n          Thread.currentThread().interrupt(); // Set back the interrupt status\n        }\n      } else {\n        Thread.currentThread().interrupt(); // Set back the interrupt status\n        LOG.warn(\"Interrupted finding locks to unlock {}.{}\", databaseName, tableName, ie);\n      }\n    } catch (Exception e) {\n      LOG.warn(\"Failed to unlock {}.{}\", databaseName, tableName, e);\n    }\n  }\n\n  private void doUnlock(long lockId) throws TException, InterruptedException {\n    metaClients.run(\n        client -> {\n          client.unlock(lockId);\n          return null;\n        });","sourceCodeStart":399,"sourceCodeEnd":435,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/hive-metastore/src/main/java/org/apache/iceberg/hive/MetastoreLock.java#L399-L435","documentation":"During MetastoreLock.unlock, if the thread was interrupted and an unlock retry (using the known lockId) also fails, the code logs this warning and finally restores the thread's interrupt status. It indicates a lock may have been left behind in HMS after two failed unlock attempts.","triggerScenarios":"Thread interrupted while unlocking (executor shutdown, task cancellation) AND the immediate doUnlock retry throws again (HMS unreachable, TException, InterruptedException again).","commonSituations":"Cancelling Spark/Flink jobs mid-commit; pod termination in Kubernetes; thread pools shutting down with in-flight commits.","solutions":["Check HMS for leaked locks (SHOW LOCKS / agentInfo) and manually unlock if present","Avoid cancelling tasks during commit; drain executors gracefully","Retry the whole operation — commit path will re-acquire locks appropriately","Investigate root interrupt source (shutdown hooks, timeouts) if recurring"],"exampleFix":"// before\nexecutor.shutdownNow(); // interrupts commit threads mid-unlock\n// after\nexecutor.shutdown();\nif (!executor.awaitTermination(60, TimeUnit.SECONDS)) {\n  executor.shutdownNow(); // interrupt only after graceful drain\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { operation.cancel(false); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); /* preserve status */ }","preventionTips":["Avoid shutdownNow/cancellation during table commits","Drain executors gracefully before interrupting","Preserve interrupt status in catch blocks","Check HMS for orphaned locks after forced shutdowns"],"tags":["hive","lock","interrupt","threading"],"backgroundTag":"operation-not-supported","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}