{"record":{"id":"8ee40adddd3ee035","repo":"stamparm/maltrail","slug":"bad-trail-bin-magic","errorCode":null,"errorMessage":"bad trail bin magic","messagePattern":"bad trail bin magic","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"core/trailsbin.py","lineNumber":174,"sourceCode":"    Memory-maps a binary trail file and returns a dict of read handles:\n    {mmap, hi, lo, val, pair_list, collisions, regex, length}. The 'hi'/'lo'/'val' views read directly from the\n    shared mapping. Raises ValueError on a bad/truncated/foreign file.\n    \"\"\"\n\n    f = open(path, \"rb\")\n    try:\n        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)\n    finally:\n        f.close()\n\n    if mm.size() < _HEADER_SIZE:\n        mm.close()\n        raise ValueError(\"trail bin too small\")\n\n    magic, cap, n, blob_len = _HEADER.unpack(mm[:_HEADER_SIZE])\n    if magic != _MAGIC:\n        mm.close()\n        raise ValueError(\"bad trail bin magic\")\n\n    off = _HEADER_SIZE\n    expected = off + 12 * cap + blob_len\n    if mm.size() < expected:\n        mm.close()\n        raise ValueError(\"truncated trail bin (have %d, need %d)\" % (mm.size(), expected))\n\n    hi = _u32_view(mm, off, cap); off += 4 * cap\n    lo = _u32_view(mm, off, cap); off += 4 * cap\n    val = _u32_view(mm, off, cap); off += 4 * cap\n\n    raw_pairs, raw_collisions, regex = json.loads(mm[off:off + blob_len].decode(\"utf-8\"))\n    pair_list = [(_native_str(p[0]), _native_str(p[1])) for p in raw_pairs]   # JSON lists -> the (info, reference) tuples the rest of the code expects\n    collisions = dict((_native_str(k), (_native_str(v[0]), _native_str(v[1]))) for k, v in raw_collisions.items())\n    regex = _native_str(regex)\n\n    return {\"mmap\": mm, \"hi\": hi, \"lo\": lo, \"val\": val, \"mask\": cap - 1,\n            \"pair_list\": pair_list, \"collisions\": collisions, \"regex\": regex, \"length\": n + len(collisions)}","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/stamparm/maltrail/blob/77cfb06d7606506d101bbcec0786c77166c4255e/core/trailsbin.py#L156-L192","documentation":"open_bin() unpacks the file header and compares the magic field against _MAGIC; a mismatch means the file has a valid size but was not produced by this trail-bin format (wrong file type, different/foreign version, or corrupt first bytes). This is a format-identity guard that protects readers from interpreting arbitrary bytes as header-derived capacity, counts, and blob lengths, which would yield garbage views or out-of-bounds accesses. The mmap is closed before raising so no mapping leaks. Fix by supplying a genuine trail binary file with the expected magic.","triggerScenarios":"Opening a non-bin file (plain text, other sidecar, tarball) with open_bin(); a bin written by an incompatible format version; byte-swapped or corrupted header.","commonSituations":"Config points at the provenance sidecar or a log file instead of trails.bin; bin produced by a different Maltrail version after a format change; corrupted transfer.","solutions":["Point open_bin at the actual trails.bin produced by the matching library version","Rebuild the bin with the current version to guarantee the magic matches","Verify file integrity (checksum) after transfer","Inspect the first bytes manually to identify what file was actually passed"],"exampleFix":"// before\ntrails = open_bin(config.get(\"bin\", \"trails.prov\"))\n// after\ntrails = open_bin(config.get(\"bin\", \"trails.bin\"))","handlingStrategy":"validation","validationCode":"def header_magic_ok(path):\n    import struct\n    with open(path, \"rb\") as f:\n        hdr = f.read(16)\n    magic = struct.unpack_from(\"<4s\", hdr, 0)[0]\n    return magic == b\"TRLS\"  # replace with actual _MAGIC","typeGuard":null,"tryCatchPattern":"try:\n    trails = open_bin(path)\nexcept ValueError as e:\n    if \"magic\" in str(e):\n        raise WrongFileFormatError(path) from e","preventionTips":["Version-stamp bins and check compatibility on load","Keep sidecars and bins in separate directories","Checksum after transfer to detect corruption","Regenerate bins after any library format change"],"tags":["python","file-format","binary-header"],"backgroundTag":"checksum-mismatch","analyzedSha":"77cfb06d7606506d101bbcec0786c77166c4255e","analyzedAt":"2026-09-13T03:50:16.010Z","contentChangedAt":"2026-09-13T03:50:16.010Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}