{"record":{"id":"0d1bbd18bb23fb27","repo":"prestodb/presto","slug":"hive-concurrent-modification-detected-0d1bbd","errorCode":"HIVE_CONCURRENT_MODIFICATION_DETECTED","errorMessage":"Partition %s was added or modified during INSERT","messagePattern":"Partition (.+?) was added or modified during INSERT","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/PartitionUpdate.java","lineNumber":197,"sourceCode":"\n    public static List<PartitionUpdate> mergePartitionUpdates(Iterable<PartitionUpdate> unMergedUpdates)\n    {\n        ImmutableList.Builder<PartitionUpdate> partitionUpdates = ImmutableList.builder();\n        for (Collection<PartitionUpdate> partitionGroup : Multimaps.index(unMergedUpdates, PartitionUpdate::getName).asMap().values()) {\n            PartitionUpdate firstPartition = partitionGroup.iterator().next();\n\n            ImmutableList.Builder<FileWriteInfo> allFileWriterInfos = ImmutableList.builder();\n            long totalRowCount = 0;\n            long totalInMemoryDataSizeInBytes = 0;\n            long totalOnDiskDataSizeInBytes = 0;\n            boolean containsNumberedFileNames = true;\n            for (PartitionUpdate partition : partitionGroup) {\n                // verify partitions have the same new flag, write path and target path\n                // this shouldn't happen but could if another user added a partition during the write\n                if (partition.getUpdateMode() != firstPartition.getUpdateMode() ||\n                        !partition.getWritePath().equals(firstPartition.getWritePath()) ||\n                        !partition.getTargetPath().equals(firstPartition.getTargetPath())) {\n                    throw new PrestoException(HIVE_CONCURRENT_MODIFICATION_DETECTED, format(\"Partition %s was added or modified during INSERT\", firstPartition.getName()));\n                }\n                allFileWriterInfos.addAll(partition.getFileWriteInfos());\n                totalRowCount += partition.getRowCount();\n                totalInMemoryDataSizeInBytes += partition.getInMemoryDataSizeInBytes();\n                totalOnDiskDataSizeInBytes += partition.getOnDiskDataSizeInBytes();\n                containsNumberedFileNames &= partition.containsNumberedFileNames();\n            }\n\n            partitionUpdates.add(new PartitionUpdate(firstPartition.getName(),\n                    firstPartition.getUpdateMode(),\n                    firstPartition.getWritePath(),\n                    firstPartition.getTargetPath(),\n                    allFileWriterInfos.build(),\n                    totalRowCount,\n                    totalInMemoryDataSizeInBytes,\n                    totalOnDiskDataSizeInBytes,\n                    containsNumberedFileNames));\n        }","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/PartitionUpdate.java#L179-L215","documentation":"When finishing a Hive INSERT, Presto merges per-worker PartitionUpdate records grouped by partition name. Each update in a group must agree on update mode, write path, and target path. A mismatch means the partition's layout changed while the query was writing — typically another writer added or modified the same partition concurrently — so Presto aborts with HIVE_CONCURRENT_MODIFICATION_DETECTED to avoid committing inconsistent data.","triggerScenarios":"Two queries (or jobs) writing the same partition of the same table simultaneously so their PartitionUpdate entries for one partition disagree on updateMode/writePath/targetPath; a concurrent INSERT/ALTER/ADD PARTITION against the target partition during a long-running INSERT.","commonSituations":"Scheduled ETL jobs overlapping; multiple analysts INSERTing into the same daily partition at the same time; a backfill job racing a streaming write to one partition; stale metastore state letting two writers pick the same target partition.","solutions":["Serialize writers to the same partition: ensure only one INSERT writes a given partition at a time (scheduling/locking).","Re-run the failed INSERT after the concurrent writer completes.","Use INSERT OVERWRITE or write to distinct partitions/table versions to avoid overlap.","Check for duplicate/overlapping scheduled jobs and stagger or partition their targets."],"exampleFix":"// before: two jobs both run at 00:00 writing dt=today\njobA: INSERT INTO events PARTITION (dt='2026-09-04') ...\njobB: INSERT INTO events PARTITION (dt='2026-09-04') ...\n// after: stagger or partition differently\njobB: INSERT INTO events PARTITION (dt='2026-09-04', src='jobB') ...  // or run jobB after jobA completes","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    executeInsert(sql);\n} catch (PrestoException e) {\n    if (e.getErrorCode() == HiveErrorCode.HIVE_CONCURRENT_MODIFICATION_DETECTED.toErrorCode().getId()) {\n        // wait for the competing writer, then retry the insert\n        retryAfterDelay(sql);\n    }\n}","preventionTips":["Serialize writes to the same partition via job scheduling or an external lock.","Ensure only one INSERT targets a given partition at a time in your ETL orchestration.","Use distinct partitions or INSERT OVERWRITE when parallel writers are unavoidable."],"tags":["hive","concurrency","insert","partition","conflict"],"backgroundTag":"concurrent-modification-detected","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}