apache/flink · error · IndexOutOfBoundsException

offset=%d, numBytes=%d, address=%d

Error message

offset=%d, numBytes=%d, address=%d

What it means

copyFromUnsafe(offset, source, sourcePointer, numBytes) copies numBytes bytes from arbitrary unsafe memory into the segment at offset. It is an explicitly unchecked fast path with a single destination guard: if address + offset + numBytes exceeds addressLimit it throws IndexOutOfBoundsException formatted as 'offset=%d, numBytes=%d, address=%d'. Source bounds are not validated at all.

Source

Thrown at flink-core/src/main/java/org/apache/flink/core/memory/MemorySegment.java:1497

        }
        UNSAFE.copyMemory(this.heapMemory, thisPointer, target, targetPointer, numBytes);
    }

    /**
     * Bulk copy method. Copies {@code numBytes} bytes from source unsafe object and pointer. NOTE:
     * This is an unsafe method, no check here, please be careful.
     *
     * @param offset The position where the bytes are started to be write in this memory segment.
     * @param source The unsafe memory to copy the bytes from.
     * @param sourcePointer The position in the source unsafe memory to copy the chunk from.
     * @param numBytes The number of bytes to copy.
     * @throws IndexOutOfBoundsException If this segment can not contain the given number of bytes
     *     (starting from offset).
     */
    public void copyFromUnsafe(int offset, Object source, int sourcePointer, int numBytes) {
        final long thisPointer = this.address + offset;
        if (thisPointer + numBytes > addressLimit) {
            throw new IndexOutOfBoundsException(
                    String.format(
                            "offset=%d, numBytes=%d, address=%d", offset, numBytes, this.address));
        }
        UNSAFE.copyMemory(source, sourcePointer, this.heapMemory, thisPointer, numBytes);
    }

    // -------------------------------------------------------------------------
    //                      Comparisons & Swapping
    // -------------------------------------------------------------------------

    /**
     * Compares two memory segment regions.
     *
     * @param seg2 Segment to compare this segment with
     * @param offset1 Offset of this segment to start comparing
     * @param offset2 Offset of seg2 to start comparing
     * @param len Length of the compared memory region
     * @return 0 if equal, -1 if seg1 < seg2, 1 otherwise

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Before the call, validate offset >= 0 && offset + numBytes <= segment.size() and move to the next segment when it does not fit.
2. If numBytes derives from a length prefix, validate it against the record/segment budget before copying.
3. Decode the formatted message: offset + numBytes vs segment size tells you exactly by how much the write overshot; use it to size the rollover logic.

Example fix

// before
segment.copyFromUnsafe(offset, source, srcPtr, numBytes); // IndexOutOfBoundsException

// after
if (offset + numBytes > segment.size()) {
    throw new IndexOutOfBoundsException("record does not fit in current segment");
}
segment.copyFromUnsafe(offset, source, srcPtr, numBytes);

if (offset < 0 || numBytes < 0 || offset + numBytes > segment.size()) {
    throw new IndexOutOfBoundsException(
            "offset=" + offset + ", numBytes=" + numBytes + ", size=" + segment.size());
}
segment.copyFromUnsafe(offset, source, srcPtr, numBytes);

try {
    segment.copyFromUnsafe(offset, source, srcPtr, numBytes);
} catch (IndexOutOfBoundsException e) {
    // message prints offset/numBytes/address; use them to fix rollover math
    throw e;
}

When it happens

Trigger: Calling copyFromUnsafe with an offset near the end of the segment such that offset + numBytes overflows the segment size, e.g. writing a record larger than the space left in the current segment instead of rolling over to the next one.

Common situations: Record-building loops (normalized key / sort buffers) that forget to request the next segment when the remaining bytes are insufficient; numBytes computed from a serialized length field larger than expected; miscomputed source pointers corrupting downstream size math.

Related errors

• Memory segment does not represent off heap memory
• offset1=%d, offset2=%d, len=%d, bufferSize=%d, address1=%d,
• Could not write {numBytes} bytes. Buffer overflow.
• MemorySegment can be freed only once!
• Memory segment does not represent heap memory

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/dd3b8ff1b188b494. Report an issue: GitHub.