stamparm/maltrail · error · ValueError

bad trail bin magic

Error message

bad trail bin magic

What it means

open_bin() unpacks the file header and compares the magic field against _MAGIC; a mismatch means the file has a valid size but was not produced by this trail-bin format (wrong file type, different/foreign version, or corrupt first bytes). This is a format-identity guard that protects readers from interpreting arbitrary bytes as header-derived capacity, counts, and blob lengths, which would yield garbage views or out-of-bounds accesses. The mmap is closed before raising so no mapping leaks. Fix by supplying a genuine trail binary file with the expected magic.

Solutions

1. Point open_bin at the actual trails.bin produced by the matching library version
2. Rebuild the bin with the current version to guarantee the magic matches
3. Verify file integrity (checksum) after transfer
4. Inspect the first bytes manually to identify what file was actually passed

Example fix

// before
trails = open_bin(config.get("bin", "trails.prov"))
// after
trails = open_bin(config.get("bin", "trails.bin"))

def header_magic_ok(path):
    import struct
    with open(path, "rb") as f:
        hdr = f.read(16)
    magic = struct.unpack_from("<4s", hdr, 0)[0]
    return magic == b"TRLS"  # replace with actual _MAGIC

try:
    trails = open_bin(path)
except ValueError as e:
    if "magic" in str(e):
        raise WrongFileFormatError(path) from e

When it happens

Trigger: Opening a non-bin file (plain text, other sidecar, tarball) with open_bin(); a bin written by an incompatible format version; byte-swapped or corrupted header.

Common situations: Config points at the provenance sidecar or a log file instead of trails.bin; bin produced by a different Maltrail version after a format change; corrupted transfer.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• not a Maltrail provenance sidecar (bad magic)
• provenance sidecar is truncated
• trail bin too small
• [x] invalid IP address
• packet too short for header-protection sample

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/8ee40adddd3ee035. Report an issue: GitHub.

Appendix: source

Thrown at core/trailsbin.py:174

    Memory-maps a binary trail file and returns a dict of read handles:
    {mmap, hi, lo, val, pair_list, collisions, regex, length}. The 'hi'/'lo'/'val' views read directly from the
    shared mapping. Raises ValueError on a bad/truncated/foreign file.
    """

    f = open(path, "rb")
    try:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    finally:
        f.close()

    if mm.size() < _HEADER_SIZE:
        mm.close()
        raise ValueError("trail bin too small")

    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])
    if magic != _MAGIC:
        mm.close()
        raise ValueError("bad trail bin magic")

    off = _HEADER_SIZE
    expected = off + 12 * cap + blob_len
    if mm.size() < expected:
        mm.close()
        raise ValueError("truncated trail bin (have %d, need %d)" % (mm.size(), expected))

    hi = _u32_view(mm, off, cap); off += 4 * cap
    lo = _u32_view(mm, off, cap); off += 4 * cap
    val = _u32_view(mm, off, cap); off += 4 * cap

    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode("utf-8"))
    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects
    collisions = dict((_native_str(k), (_native_str(v[0]), _native_str(v[1]))) for k, v in raw_collisions.items())
    regex = _native_str(regex)

    return {"mmap": mm, "hi": hi, "lo": lo, "val": val, "mask": cap - 1,
            "pair_list": pair_list, "collisions": collisions, "regex": regex, "length": n + len(collisions)}

View on GitHub (pinned to 77cfb06d76)