apache/flink · error · NoSuchElementException

Reached the end of the collection. This could be caused by i

Error message

Reached the end of the collection. This could be caused by issues with the serializer or by calling the map() function more times than there are elements in the collection. Make sure that you set the number of records to be produced by the DataGeneratorSource equal to the number of elements in the collection.

What it means

Thrown as NoSuchElementException by tryDeserialize in IndexLookupGeneratorFunction during buildLookup() when deserialization hits EOF before all expected numElements are deserialized. The message explains this is caused by serializer issues or by requesting more records than there are elements in the collection.

Source

Thrown at flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/functions/IndexLookupGeneratorFunction.java:147

        LOG.info("Serializing elements using  {}", serializer);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        DataOutputViewStreamWrapper wrapper = new DataOutputViewStreamWrapper(baos);

        try {
            for (OUT element : elements) {
                serializer.serialize(element, wrapper);
            }
        } catch (Exception e) {
            throw new IOException("Serializing the source elements failed: " + e.getMessage(), e);
        }
        this.elementsSerialized = baos.toByteArray();
    }

    private OUT tryDeserialize() throws IOException {
        try {
            return serializer.deserialize(input);
        } catch (EOFException eof) {
            throw new NoSuchElementException(
                    "Reached the end of the collection. This could be caused by issues with the "
                            + "serializer or by calling the map() function more times than there "
                            + "are elements in the collection. Make sure that you set the number "
                            + "of records to be produced by the DataGeneratorSource equal to the "
                            + "number of elements in the collection.");
        } catch (Exception e) {
            throw new IOException(
                    "Failed to deserialize an element from the source. "
                            + "If you are using user-defined serialization (Value and Writable "
                            + "types), check the serialization functions.\nSerializer is "
                            + serializer,
                    e);
        }
    }

    private void buildLookup() throws IOException {
        for (long i = 0; i < numElements; i++) {
            lookupMap.put(i, tryDeserialize());

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the TypeSerializer round-trips correctly: serialize N elements, then deserialize exactly N elements.
2. Ensure the TypeInformation used during construction matches the element types exactly.
3. Test with a simple known type (e.g., String or Integer) to isolate whether the issue is in the serializer.
4. Check if the numElements counter (incremented in checkIterable) matches the actual number of serialized elements.

// Verify round-trip: serialize then deserialize exactly N elements
TypeSerializer<OUT> serializer = typeInfo.createSerializer(config.getSerializerConfig());
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputViewStreamWrapper out = new DataOutputViewStreamWrapper(baos);
int count = 0;
for (OUT element : elements) { serializer.serialize(element, out); count++; }
DataInputViewStreamWrapper in = new DataInputViewStreamWrapper(
    new ByteArrayInputStream(baos.toByteArray()));
for (int i = 0; i < count; i++) {
    serializer.deserialize(in); // throws EOFException if mismatch
}

try {
    IndexLookupGeneratorFunction<OUT> fn =
        new IndexLookupGeneratorFunction<>(typeInfo, elements);
    fn.open(readerContext);
} catch (NoSuchElementException e) {
    if (e.getMessage().startsWith("Reached the end of the collection")) {
        // serializer mismatch — test round-trip separately
    }
    throw e;
}

When it happens

Trigger: During open(), buildLookup deserializes numElements items. If the serialized byte array is exhausted early (EOFException), the exception fires. Causes include a TypeSerializer that writes variable-length data inconsistently, a numElements count that does not match the actual serialized content, or a serializer that skips null-valued fields.

Common situations: Using a custom serializer that does not round-trip correctly (serialize writes fewer bytes than deserialize reads); TypeInformation mismatch between construction-time serialization and open()-time deserialization after a serialization format change.

Related errors

• Failed to deserialize an element from the source. If you are
• Serializing the source elements failed: {e.getMessage()}
• {e.getMessage()}
• Could not skip {numBytes} bytes.
• Could not skip {numBytes} bytes.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/63f250e8b1222352. Report an issue: GitHub.