pinpoint-apm/pinpoint · critical · IllegalStateException

Corrupted schema change logs. Duplicate order for change set

Error message

Corrupted schema change logs. Duplicate order for change set : 

What it means

Thrown by SchemaChangeLogServiceImpl.getSchemaChangeLogs when two stored change log records carry the same execOrder value. Records are ordered by execOrder (orderedSchemaChangeLogs.put returns a previous entry), and a collision means the log table content is corrupted since each change set must have a distinct execution order. Reading logs is aborted rather than returning an ambiguous ordering.

Source

Thrown at hbase/hbase-schema/src/main/java/com/navercorp/pinpoint/hbase/schema/service/SchemaChangeLogServiceImpl.java:121

                .execOrder(executionOrder)
                .checkSum(checkSum)
                .value(changeSet.getValue())
                .build();
        schemaChangeLogDao.insertChangeLog(namespace, schemaChangeLog);
        return schemaChangeLog;
    }

    @Override
    public List<SchemaChangeLog> getSchemaChangeLogs(String namespace) {
        List<SchemaChangeLog> schemaChangeLogs = schemaChangeLogDao.getChangeLogs(namespace);
        Map<Integer, SchemaChangeLog> orderedSchemaChangeLogs = new TreeMap<>();
        Set<String> schemaChangeLogIds = new HashSet<>();
        for (SchemaChangeLog schemaChangeLog : schemaChangeLogs) {
            Integer execOrder = schemaChangeLog.getExecOrder();
            String id = schemaChangeLog.getId();
            SchemaChangeLog previousLog = orderedSchemaChangeLogs.put(execOrder, schemaChangeLog);
            if (previousLog != null) {
                throw new IllegalStateException("Corrupted schema change logs. Duplicate order for change set : " + id);
            }
            if (!schemaChangeLogIds.add(schemaChangeLog.getId())) {
                throw new IllegalStateException("Corrupted schema change logs. Duplicate change set : " + id);
            }
        }
        return new ArrayList<>(orderedSchemaChangeLogs.values());
    }

    @Override
    public SchemaChangeLog getSchemaChangeLog(String namespace, String id) {
        Assert.hasLength(id, "namespace must not be empty");
        return schemaChangeLogDao.getChangeLog(namespace, id);
    }

}

View on GitHub (pinned to 744c3d3075)

Solutions

1. Identify the duplicate rows (SELECT id, execOrder ORDER BY execOrder) and fix/remove the corrupt record so orders are unique and contiguous.
2. Restore the change log table from a known-good backup taken before the corruption.
3. Ensure only one node/process performs schema updates at a time (distributed lock) to prevent concurrent writers from colliding.
4. If corruption cannot be repaired safely, back up data, drop managed tables and the log table, and re-initialize.

Example fix

// before: two rows with execOrder=3
// after: renumber duplicates
UPDATE schema_change_log SET execOrder = 4 WHERE id = 'changeSet-7';

// Detect duplicates in the log table before processing
Map<Integer, Long> byOrder = logs.stream()
    .collect(Collectors.groupingBy(SchemaChangeLog::getExecOrder, Collectors.counting()));
boolean corrupt = byOrder.values().stream().anyMatch(c -> c > 1);

try { logs = schemaChangeLogService.getSchemaChangeLogs(namespace); } catch (IllegalStateException e) {
    if (e.getMessage().contains("Duplicate order")) { alertOps("Corrupt change log: execOrder collision"); }
}

When it happens

Trigger: Reading change logs where two rows in the schema change log table share an identical execOrder column value.

Common situations: Concurrent schema updates on multiple nodes writing logs without proper locking/ordering; manual edits or partial backup restores of the log table; a bug or race in an older writer version that assigned the same order twice.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Corrupted schema change logs. Duplicate change set :
• Duplicate ColumnFamily name :
• Connection already closed
• Invalid namespace : <namespace>
• Already closed

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/3dc77ae2bd7c2509. Report an issue: GitHub.