stamparm/maltrail · error · ValueError

trail bin too small

Error message

trail bin too small

What it means

open_bin() rejects the file before any parsing happens because its size is below _HEADER_SIZE, so the fixed-size header (magic, cap, n, blob_len) cannot even be unpacked. This fires when the path points to an empty file, a partially written/truncated trail bin, or a non-trail file that happens to be tiny. It is an input-validation guard: the caller passed a file that is not a complete trail binary. The mmap is closed first so the mapping does not leak. Fix by pointing open_bin at a valid, fully written trail bin produced by the trail writer.

Solutions

1. Rebuild or re-download the trail bin; verify it is complete
2. Check os.path.getsize(path) >= expected minimum before opening
3. Catch ValueError from open_bin and fall back to regenerating the bin
4. Ensure the producer writes the header atomically (write to temp then rename)

Example fix

// before
trails = open_bin("trails.bin")
// after
import os
if os.path.getsize("trails.bin") >= 16:
    trails = open_bin("trails.bin")
else:
    trails = rebuild_bin("trails.bin")

import os
def bin_header_plausible(path, min_size=16):
    return os.path.isfile(path) and os.path.getsize(path) >= min_size

try:
    trails = open_bin(path)
except ValueError:
    trails = rebuild_bin(path)  # too small / invalid

When it happens

Trigger: Opening a 0-byte or few-byte file (failed/aborted build, touch-created placeholder) via open_bin(path).

Common situations: trail bin build crashed before writing the header, empty file created by a failed download, mount/network issue returning an empty file.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• not a Maltrail provenance sidecar (bad magic)
• bad trail bin magic
• truncated trail bin (have , need )
• [x] invalid IP address
• provenance sidecar is truncated

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/9326f3a0bca96d89. Report an issue: GitHub.

Appendix: source

Thrown at core/trailsbin.py:169

    return memoryview(buf)[offset:offset + 4 * n].cast("I")

def open_bin(path):
    """
    Memory-maps a binary trail file and returns a dict of read handles:
    {mmap, hi, lo, val, pair_list, collisions, regex, length}. The 'hi'/'lo'/'val' views read directly from the
    shared mapping. Raises ValueError on a bad/truncated/foreign file.
    """

    f = open(path, "rb")
    try:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    finally:
        f.close()

    if mm.size() < _HEADER_SIZE:
        mm.close()
        raise ValueError("trail bin too small")

    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])
    if magic != _MAGIC:
        mm.close()
        raise ValueError("bad trail bin magic")

    off = _HEADER_SIZE
    expected = off + 12 * cap + blob_len
    if mm.size() < expected:
        mm.close()
        raise ValueError("truncated trail bin (have %d, need %d)" % (mm.size(), expected))

    hi = _u32_view(mm, off, cap); off += 4 * cap
    lo = _u32_view(mm, off, cap); off += 4 * cap
    val = _u32_view(mm, off, cap); off += 4 * cap

    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode("utf-8"))
    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects

View on GitHub (pinned to 77cfb06d76)