pinpoint-apm/pinpoint · critical · IllegalStateException

Current table schema does not match the schema change log re

Error message

Current table schema does not match the schema change log records.

What it means

Thrown by HbaseSchemaServiceImpl.updateExistingSchemas when replaying all previously executed change sets against an empty namespace produces a table layout (schema snapshot) that does not match the live HBase table descriptors (currentHtds). It means the change log history and the actual table schemas have diverged — someone altered tables manually, or logs were lost/corrupted. Update is aborted to prevent applying further changes on an inconsistent base.

Source

Thrown at hbase/hbase-schema/src/main/java/com/navercorp/pinpoint/hbase/schema/service/HbaseSchemaServiceImpl.java:211

    }

    private boolean updateExistingSchemas(String namespace, String compression, List<ChangeSet> changeSets, List<TableDescriptor> currentHtds, List<SchemaChangeLog> executedLogs) {
        logger.info("[{}] Updating hbase schema.", namespace);

        List<String> executedChangeLogIds = executedLogs.stream()
                .map(SchemaChangeLog::getId).collect(Collectors.toList());
        logger.info("[{}] Executed change logs : {}", namespace, executedChangeLogIds);

        ChangeSetManager changeSetManager = new ChangeSetManager(changeSets);

        // Check if the current table schema matches the expected schema specified by the executed schema change logs.
        HbaseSchemaCommandManager initCommandManager = new HbaseSchemaCommandManager(namespace, compression);
        List<ChangeSet> executedChangeSets = changeSetManager.getExecutedChangeSets(executedLogs);
        for (ChangeSet executedChangeSet : executedChangeSets) {
            initCommandManager.applyChangeSet(executedChangeSet);
        }
        if (!hbaseSchemaVerifier.verifySchemas(initCommandManager.getSchemaSnapshot(), currentHtds)) {
            throw new IllegalStateException("Current table schema does not match the schema change log records.");
        }

        List<ChangeSet> changeSetsToApply = changeSetManager.filterExecutedChangeSets(executedLogs);
        if (changeSetsToApply.isEmpty()) {
            logger.info("[{}] Hbase schema already at latest version", namespace);
            return false;
        }
        HbaseSchemaCommandManager updateCommandManager = new HbaseSchemaCommandManager(namespace, compression, currentHtds);
        return applyChangeSets(updateCommandManager, changeSetsToApply, executedLogs);
    }


    private boolean applyChangeSets(HbaseSchemaCommandManager commandManager, List<ChangeSet> changeSets, List<SchemaChangeLog> executedLogs) {
        if (CollectionUtils.isEmpty(changeSets)) {
            return false;
        }

        String namespace = commandManager.getNamespace();

View on GitHub (pinned to 744c3d3075)

Solutions

  1. Compare the current table descriptors with the replayed snapshot (hbase shell 'list' / describe) and revert manual table modifications so they match the change log.
  2. If change sets were edited after execution, restore the original definitions that were actually executed.
  3. As a last resort for a corrupted history, back up data, drop the schema change log table and the managed tables, and re-initialize from scratch (fresh initialization).
  4. Ensure the configured namespace/cluster is the intended one before re-running update().

Example fix

// before: silently ignoring mismatch
// after: detect drift early in a health check
SchemaSnapshot snapshot = initCommandManager.getSchemaSnapshot();
if (!hbaseSchemaVerifier.verifySchemas(snapshot, currentHtds)) {
    logger.error("Schema drift detected for namespace {}. Tables: {}", namespace, currentHtds.keySet());
    throw new IllegalStateException("..." );
}
Defensive patterns

Strategy: try-catch

Validate before calling

// Prohibit manual table changes and verify drift regularly
if (!hbaseSchemaVerifier.verifySchemas(snapshot, currentHtds)) {
    alert("Schema drift detected for namespace " + namespace);
}

Try / catch

try { schemaService.update(); } catch (IllegalStateException e) {
    if (e.getMessage().contains("does not match the schema change log")) {
        // page an admin: manual table drift or corrupted history
    }
}

Prevention

When it happens

Trigger: Calling update() when the schema change log table has records, but HbaseSchemaVerifier.verifySchemas fails after rebuilding the snapshot from executedChangeSets vs the current HTableDescriptors fetched from HBase.

Common situations: An admin manually created/modified/dropped tables or column families in HBase outside of the schema tooling; the change log table was partially restored from a backup; a changeSet was edited after it was already executed (history no longer reproduces reality); namespace mismatch pointing update at another cluster's tables.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of pinpoint-apm/pinpoint@744c3d3075 (2026-09-07). Data as JSON: /api/errors/d22c9e3be5c8235f. Report an issue: GitHub.