t8y2/dbx · error · IllegalArgumentException

duplicate partition in offsets: " + partition

Error message

duplicate partition in offsets: " + partition

What it means

Within one commit request, each partition of the topic may appear only once. The parser accumulates entries into a Map<TopicPartition, OffsetAndMetadata>; if put() returns a previous value for the same TopicPartition it throws IllegalArgumentException "duplicate partition in offsets: <n>" (KafkaAgent.java:1532).

Source

Thrown at agents/drivers/kafka/src/main/java/com/dbx/agent/kafka/KafkaAgent.java:1532

        }
        JsonElement offsetsElement = params.get("offsets");
        if (!offsetsElement.isJsonArray()) {
            throw new IllegalArgumentException("offsets must be an array");
        }
        JsonArray offsetArray = offsetsElement.getAsJsonArray();
        if (offsetArray.isEmpty()) {
            throw new IllegalArgumentException("offsets must contain at least one partition offset");
        }
        for (JsonElement element : offsetArray) {
            if (!element.isJsonObject()) {
                throw new IllegalArgumentException("each offset must be an object");
            }
            JsonObject value = element.getAsJsonObject();
            int partition = nonNegativeExactInt(value, "partition");
            long offset = nonNegativeExactLong(value, "offset");
            TopicPartition topicPartition = new TopicPartition(topic, partition);
            if (offsets.put(topicPartition, new OffsetAndMetadata(offset)) != null) {
                throw new IllegalArgumentException("duplicate partition in offsets: " + partition);
            }
        }
        return offsets;
    }

    private static int nonNegativeExactInt(JsonObject object, String name) {
        long value = nonNegativeExactLong(object, name);
        if (value > Integer.MAX_VALUE) {
            throw new IllegalArgumentException(name + " is outside the supported integer range");
        }
        return (int) value;
    }

    private static long nonNegativeExactLong(JsonObject object, String name) {
        JsonElement element = object.get(name);
        if (element == null || !element.isJsonPrimitive() || !element.getAsJsonPrimitive().isNumber()) {
            throw new IllegalArgumentException(name + " must be a non-negative integer");
        }

View on GitHub (pinned to c0390bff16)

Solutions

1. Deduplicate by partition before sending, keeping the latest offset entry
2. Use a Map keyed by partition when building the offsets list so duplicates are impossible
3. Check upstream code for repeated appends of the same partition

Example fix

// before
offsets.add(Map.of("partition",0,"offset",10));
offsets.add(Map.of("partition",0,"offset",20));
// after
Map<Integer,Long> byPartition = new HashMap<>();
byPartition.put(0, 20L); // one entry per partition, latest wins

const parts = offsets.map(o => o.partition); if (new Set(parts).size !== parts.length) throw new Error('duplicate partition');

try { agent.execute(req); } catch (e) { if (String(e.message).startsWith('duplicate partition in offsets')) { /* dedupe and retry once */ } else throw e; }

When it happens

Trigger: The offsets array contains two objects with the same "partition" value for the same topic, e.g. [{"partition":0,"offset":10},{"partition":0,"offset":20}].

Common situations: Merging offset lists from multiple consumers without deduplication; retry logic re-appending an entry; concatenating per-partition results that overlap.

Related errors

• offsets must contain at least one partition offset
• each offset must be an object
• " + name + " is outside the supported integer range
• " + name + " must be a non-negative integer
• timestampMs is required when position is timestamp

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/99574bb0fbe2b238. Report an issue: GitHub.