apache/cassandra · error · CoordinatorBehindException

CoordinatorBehindException (read command serialized at later

Error message

CoordinatorBehindException (read command serialized at later schema epoch than replica)

What it means

CoordinatorBehindException is thrown on a replica when deserializing an incoming read command whose schema was serialized at a later cluster metadata epoch (schemaVersion) than the replica's current local epoch. It means the coordinator is 'behind': it has a newer schema view than this replica, so the replica cannot safely serve the request. The metric coordinatorBehindSchema is marked before throwing.

Source

Thrown at src/java/org/apache/cassandra/db/ReadCommand.java:1488

            int digestVersion = isDigest(flags) ? in.readUnsignedVInt32() : 0;
            TableId tableId = TableId.deserialize(in);

            Epoch schemaVersion = Epoch.EMPTY;
            if (version >= MessagingService.VERSION_60)
                schemaVersion = Epoch.serializer.deserialize(in);
            TableMetadata tableMetadata;
            try
            {
                tableMetadata = schema.getExistingTableMetadata(tableId);
            }
            catch (UnknownTableException e)
            {
                ClusterMetadata metadata = ClusterMetadata.current();
                Epoch localCurrentEpoch = metadata.epoch;
                if (schemaVersion != null && localCurrentEpoch.isAfter(schemaVersion))
                {
                    TCMMetrics.instance.coordinatorBehindSchema.mark();
                    throw new CoordinatorBehindException(e.getMessage());
                }
                throw e;
            }
            long nowInSec = version >= MessagingService.VERSION_50 ? CassandraUInt.toLong(in.readInt()) : in.readInt();
            return deserialize(kind.selectionDeserializer, flags, schemaVersion, digestVersion, nowInSec, tableMetadata, in, version);
        }

        public ReadCommand deserializeForAccord(Seekable key, TableMetadatas tables, DataInputPlus in, int version) throws IOException
        {
            Kind kind = Kind.fromOrdinal(in.readByte());
            int flags = in.readByte();
            if (isDigest(flags) || isForThrift(flags) || acceptsTransient(flags))
                throw new IllegalStateException("Received an Accord command with a digest/thrift/transient flag set.");

            TableMetadata tableMetadata = tables.deserialize(in);

            return deserialize(kind.accordSelectionDeserializer.apply(key), flags, tableMetadata.epoch, 0, 0, tableMetadata, in, version);
        }

View on GitHub (pinned to 88fd0f6a0e)

Solutions

Wait for this replica to catch up on the cluster metadata log (check `nodetool cms show` / cluster epoch); the coordinator typically retries and the condition is transient
Verify connectivity from this replica to the CMS and peers so TCM log fetches are not stalled
Check TCMMetrics.instance.coordinatorBehindSchema rate; if persistent, investigate replica-to-CMS network issues or stuck log replay
Restart the lagging replica if metadata replay is stuck

Defensive patterns

Strategy: retry

Try / catch

// client/coordinator side
try {
    result = session.execute(readQuery);
} catch (CoordinatorBehindException | ReplicaUnavailableException e) {
    // replica behind on schema epoch; brief backoff then retry
    Thread.sleep(backoffMs);
    result = session.execute(readQuery);
}

Prevention

After schema changes, allow all nodes to converge epochs before running load
Monitor TCMMetrics.coordinatorBehindSchema for persistent lag
Keep reliable connectivity between all nodes and the CMS
Upgrade nodes in a controlled rolling fashion without mixing epochs under load

When it happens

Trigger: ReadCommand.deserializePostV5 (public deserialization path) receives a read command over the wire whose serializedAtEpoch is after the local ClusterMetadata.current().epoch; thrown when the underlying deserialization raises because the schema version cannot be resolved locally.

Common situations: Coordinator rejoined or was upgraded faster than this replica; this replica is lagging in applying TCM log entries (network partition with CMS, paused gossip/log replay); rolling restart where the coordinator resumes traffic before the replica catches up.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/e3979b564754ca6a. Report an issue: GitHub.