t8y2/dbx · error · IllegalArgumentException

Kafka topic has more than ${MAX_PEEK_SCAN_RECORDS} readable

Error message

Kafka topic has more than ${MAX_PEEK_SCAN_RECORDS} readable partitions; select a partition to browse latest messages

What it means

This guard fires when a Kafka topic has more readable partitions than MAX_PEEK_SCAN_RECORDS while browsing latest messages across all partitions. Scanning every partition's latest offset requires one query per partition, so the agent caps the partition count to keep the peek operation bounded; past the cap it refuses and asks the caller to select a single partition.

Source

Thrown at agents/drivers/kafka/src/main/java/com/dbx/agent/kafka/KafkaAgent.java:2167

        boolean legacyOffsetRequest,
        long beginningOffset,
        long endOffset
    ) {
        return switch (startPosition) {
            case LATEST -> endOffset > beginningOffset ? beginningOffset : null;
            case OFFSET -> offset;
            case EARLIEST -> legacyOffsetRequest ? offset : beginningOffset;
        };
    }

    static int peekScanLimit(int count, int readablePartitionCount) {
        return peekScanLimit(count, readablePartitionCount, PeekStartPosition.EARLIEST);
    }

    static int peekScanLimit(int count, int readablePartitionCount, PeekStartPosition startPosition) {
        if (startPosition == PeekStartPosition.LATEST
            && readablePartitionCount > MAX_PEEK_SCAN_RECORDS) {
            throw new IllegalArgumentException(
                "Kafka topic has more than " + MAX_PEEK_SCAN_RECORDS
                    + " readable partitions; select a partition to browse latest messages"
            );
        }
        return MAX_PEEK_SCAN_RECORDS;
    }

    /**
     * Latest is a topic-level query. Below the scan budget, every partition contributes the
     * requested count so the global merge is exact. Above it, the fixed budget is shared across
     * partitions and the response is marked incomplete.
     */
    static int latestPeekMessagesPerPartition(int count, int readablePartitionCount) {
        int safePartitionCount = Math.max(1, readablePartitionCount);
        if (safePartitionCount > MAX_PEEK_SCAN_RECORDS) {
            throw new IllegalArgumentException(
                "Kafka topic has more than " + MAX_PEEK_SCAN_RECORDS
                    + " readable partitions; select a partition to browse latest messages"

View on GitHub (pinned to c0390bff16)

Solutions

Browse a specific partition instead of all partitions (pass a partition selection in the browse request).
Reduce the topic's partition count (topic reassignment/recreation) if full-topic latest browsing is truly needed.
Read the latest messages via a different path (e.g. subscribe with seek-to-end) rather than the peek browse API.
Raise MAX_PEEK_SCAN_RECORDS in the agent build if your deployment can afford the scan cost.

Example fix

// before
agent.call("kafka.peek_latest", {"topic": "events"})
// after
agent.call("kafka.peek_latest", {"topic": "events", "partition": 3})

Defensive patterns

Strategy: validation

Validate before calling

int readablePartitions = getReadablePartitionCount(topic);
if (readablePartitions > MAX_PEEK_SCAN_RECORDS) {
    // browse a single partition instead
    request.setPartition(0);
}

Try / catch

try {
    agent.browseLatest(topic);
} catch (IllegalArgumentException e) {
    if (e.getMessage().contains("readable partitions")) {
        agent.browseLatest(topic, partition); // fall back to single partition
    } else { throw e; }
}

Prevention

Track partition counts of topics you browse and prefer per-partition browsing on wide topics.
For latest-message checks on large topics, use a consumer with seekToEnd instead of the peek API.
Keep test topics small; avoid pointing browse tooling at high-partition production topics.

When it happens

Trigger: Calling the peek/latest-messages browse tool (peekScanLimit with PeekStartPosition.LATEST) on a topic whose readablePartitionCount exceeds MAX_PEEK_SCAN_RECORDS.

Common situations: High-partition topics (hundreds of partitions) created for throughput; auto-created topics with default partition counts; browsing 'latest' against a busy production topic instead of a small test topic.

Related errors

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/f12b89512058a541. Report an issue: GitHub.