{"record":{"id":"9326f3a0bca96d89","repo":"stamparm/maltrail","slug":"trail-bin-too-small","errorCode":null,"errorMessage":"trail bin too small","messagePattern":"trail bin too small","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"core/trailsbin.py","lineNumber":169,"sourceCode":"\n    return memoryview(buf)[offset:offset + 4 * n].cast(\"I\")\n\ndef open_bin(path):\n    \"\"\"\n    Memory-maps a binary trail file and returns a dict of read handles:\n    {mmap, hi, lo, val, pair_list, collisions, regex, length}. The 'hi'/'lo'/'val' views read directly from the\n    shared mapping. Raises ValueError on a bad/truncated/foreign file.\n    \"\"\"\n\n    f = open(path, \"rb\")\n    try:\n        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)\n    finally:\n        f.close()\n\n    if mm.size() < _HEADER_SIZE:\n        mm.close()\n        raise ValueError(\"trail bin too small\")\n\n    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])\n    if magic != _MAGIC:\n        mm.close()\n        raise ValueError(\"bad trail bin magic\")\n\n    off = _HEADER_SIZE\n    expected = off + 12 * cap + blob_len\n    if mm.size() < expected:\n        mm.close()\n        raise ValueError(\"truncated trail bin (have %d, need %d)\" % (mm.size(), expected))\n\n    hi = _u32_view(mm, off, cap); off += 4 * cap\n    lo = _u32_view(mm, off, cap); off += 4 * cap\n    val = _u32_view(mm, off, cap); off += 4 * cap\n\n    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode(\"utf-8\"))\n    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects","sourceCodeStart":151,"sourceCodeEnd":187,"githubUrl":"https://github.com/stamparm/maltrail/blob/77cfb06d7606506d101bbcec0786c77166c4255e/core/trailsbin.py#L151-L187","documentation":"open_bin() rejects the file before any parsing happens because its size is below _HEADER_SIZE, so the fixed-size header (magic, cap, n, blob_len) cannot even be unpacked. This fires when the path points to an empty file, a partially written/truncated trail bin, or a non-trail file that happens to be tiny. It is an input-validation guard: the caller passed a file that is not a complete trail binary. The mmap is closed first so the mapping does not leak. Fix by pointing open_bin at a valid, fully written trail bin produced by the trail writer.","triggerScenarios":"Opening a 0-byte or few-byte file (failed/aborted build, touch-created placeholder) via open_bin(path).","commonSituations":"trail bin build crashed before writing the header, empty file created by a failed download, mount/network issue returning an empty file.","solutions":["Rebuild or re-download the trail bin; verify it is complete","Check os.path.getsize(path) >= expected minimum before opening","Catch ValueError from open_bin and fall back to regenerating the bin","Ensure the producer writes the header atomically (write to temp then rename)"],"exampleFix":"// before\ntrails = open_bin(\"trails.bin\")\n// after\nimport os\nif os.path.getsize(\"trails.bin\") >= 16:\n    trails = open_bin(\"trails.bin\")\nelse:\n    trails = rebuild_bin(\"trails.bin\")","handlingStrategy":"validation","validationCode":"import os\ndef bin_header_plausible(path, min_size=16):\n    return os.path.isfile(path) and os.path.getsize(path) >= min_size","typeGuard":null,"tryCatchPattern":"try:\n    trails = open_bin(path)\nexcept ValueError:\n    trails = rebuild_bin(path)  # too small / invalid","preventionTips":["Publish bins atomically via rename so partial files are never seen","Confirm producer finished (exit code / done marker) before reading","Check file size before mmap-ing","Alert on zero-byte bin files in monitoring"],"tags":["python","mmap","binary-header"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"77cfb06d7606506d101bbcec0786c77166c4255e","analyzedAt":"2026-09-13T03:50:16.010Z","contentChangedAt":"2026-09-13T03:50:16.010Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}