stamparm/maltrail · error · ValueError

truncated trail bin (have , need )

Error message

truncated trail bin (have %d, need %d)

What it means

open_bin() computes the expected layout size (header + 12 bytes per capacity entry + JSON blob length) and requires the mapping to be at least that large. When the file is smaller, it reports how many bytes it has versus needs — the bin is incomplete or the header lies about its contents.

Solutions

1. Rebuild or re-download the complete trails.bin
2. Verify the bin is not currently being rewritten (use atomic rename on publish)
3. Check file size against the expected layout before/after transfer
4. Catch ValueError and fall back to the previous known-good bin

Example fix

// before
trails = open_bin("trails.bin")
// after
try:
    trails = open_bin("trails.bin")
except ValueError:
    trails = open_bin("trails.bin.bak")

import os
def bin_fully_written(path):
    return os.path.isfile(path) and not os.path.exists(path + ".tmp") and os.path.getsize(path) > 16

try:
    trails = open_bin(path)
except ValueError:
    trails = open_bin(fallback_bin)  # previous known-good copy

When it happens

Trigger: Partial download/rsync of trails.bin; build interrupted after writing header but before all arrays/blob; header cap/blob_len from a different (larger) version of the file.

Common situations: Interrupted scheduled update while the bin was being written; mismatched header from a corrupted write; restoring only part of a backup.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• provenance sidecar is truncated
• trail bin too small
• [x] invalid IP address
• not a Maltrail provenance sidecar (bad magic)
• packet too short for header-protection sample

AI-assisted analysis of stamparm/maltrail@77cfb06d76 (2026-09-13). Data as JSON: /api/errors/f2082a75fa65876d. Report an issue: GitHub.

Appendix: source

Thrown at core/trailsbin.py:180

    try:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    finally:
        f.close()

    if mm.size() < _HEADER_SIZE:
        mm.close()
        raise ValueError("trail bin too small")

    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])
    if magic != _MAGIC:
        mm.close()
        raise ValueError("bad trail bin magic")

    off = _HEADER_SIZE
    expected = off + 12 * cap + blob_len
    if mm.size() < expected:
        mm.close()
        raise ValueError("truncated trail bin (have %d, need %d)" % (mm.size(), expected))

    hi = _u32_view(mm, off, cap); off += 4 * cap
    lo = _u32_view(mm, off, cap); off += 4 * cap
    val = _u32_view(mm, off, cap); off += 4 * cap

    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode("utf-8"))
    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects
    collisions = dict((_native_str(k), (_native_str(v[0]), _native_str(v[1]))) for k, v in raw_collisions.items())
    regex = _native_str(regex)

    return {"mmap": mm, "hi": hi, "lo": lo, "val": val, "mask": cap - 1,
            "pair_list": pair_list, "collisions": collisions, "regex": regex, "length": n + len(collisions)}

def lookup(handles, key, default=None):
    """
    Looks up a key in the opened handles, returning its (info, reference) tuple or 'default'. This is the read hot
    path: a tiny side-dict check (empty in practice) then a linear-probe of the open-addressing table starting at
    (hash & mask) - terminating at the matching slot or the first empty one (a couple of probes at load < 0.5).

View on GitHub (pinned to 77cfb06d76)