{"record":{"id":"f2082a75fa65876d","repo":"stamparm/maltrail","slug":"truncated-trail-bin-have-d-need-d","errorCode":null,"errorMessage":"truncated trail bin (have %d, need %d)","messagePattern":"truncated trail bin \\(have (.+?), need (.+?)\\)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"core/trailsbin.py","lineNumber":180,"sourceCode":"    try:\n        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)\n    finally:\n        f.close()\n\n    if mm.size() < _HEADER_SIZE:\n        mm.close()\n        raise ValueError(\"trail bin too small\")\n\n    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])\n    if magic != _MAGIC:\n        mm.close()\n        raise ValueError(\"bad trail bin magic\")\n\n    off = _HEADER_SIZE\n    expected = off + 12 * cap + blob_len\n    if mm.size() < expected:\n        mm.close()\n        raise ValueError(\"truncated trail bin (have %d, need %d)\" % (mm.size(), expected))\n\n    hi = _u32_view(mm, off, cap); off += 4 * cap\n    lo = _u32_view(mm, off, cap); off += 4 * cap\n    val = _u32_view(mm, off, cap); off += 4 * cap\n\n    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode(\"utf-8\"))\n    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects\n    collisions = dict((_native_str(k), (_native_str(v[0]), _native_str(v[1]))) for k, v in raw_collisions.items())\n    regex = _native_str(regex)\n\n    return {\"mmap\": mm, \"hi\": hi, \"lo\": lo, \"val\": val, \"mask\": cap - 1,\n            \"pair_list\": pair_list, \"collisions\": collisions, \"regex\": regex, \"length\": n + len(collisions)}\n\ndef lookup(handles, key, default=None):\n    \"\"\"\n    Looks up a key in the opened handles, returning its (info, reference) tuple or 'default'. This is the read hot\n    path: a tiny side-dict check (empty in practice) then a linear-probe of the open-addressing table starting at\n    (hash & mask) - terminating at the matching slot or the first empty one (a couple of probes at load < 0.5).","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/stamparm/maltrail/blob/77cfb06d7606506d101bbcec0786c77166c4255e/core/trailsbin.py#L162-L198","documentation":"open_bin() computes the expected layout size (header + 12 bytes per capacity entry + JSON blob length) and requires the mapping to be at least that large. When the file is smaller, it reports how many bytes it has versus needs — the bin is incomplete or the header lies about its contents.","triggerScenarios":"Partial download/rsync of trails.bin; build interrupted after writing header but before all arrays/blob; header cap/blob_len from a different (larger) version of the file.","commonSituations":"Interrupted scheduled update while the bin was being written; mismatched header from a corrupted write; restoring only part of a backup.","solutions":["Rebuild or re-download the complete trails.bin","Verify the bin is not currently being rewritten (use atomic rename on publish)","Check file size against the expected layout before/after transfer","Catch ValueError and fall back to the previous known-good bin"],"exampleFix":"// before\ntrails = open_bin(\"trails.bin\")\n// after\ntry:\n    trails = open_bin(\"trails.bin\")\nexcept ValueError:\n    trails = open_bin(\"trails.bin.bak\")","handlingStrategy":"validation","validationCode":"import os\ndef bin_fully_written(path):\n    return os.path.isfile(path) and not os.path.exists(path + \".tmp\") and os.path.getsize(path) > 16","typeGuard":null,"tryCatchPattern":"try:\n    trails = open_bin(path)\nexcept ValueError:\n    trails = open_bin(fallback_bin)  # previous known-good copy","preventionTips":["Write bins to a temp path then atomically rename","Retain the previous good bin as a fallback","Verify transfer completeness (size or checksum) before swapping in","Schedule updates so readers never observe a partially built file"],"tags":["python","mmap","truncated-file"],"backgroundTag":"file-read-failed","analyzedSha":"77cfb06d7606506d101bbcec0786c77166c4255e","analyzedAt":"2026-09-13T03:50:16.010Z","contentChangedAt":"2026-09-13T03:50:16.010Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}