{"record":{"id":"9e4f652bc6c7910d","repo":"stamparm/maltrail","slug":"provenance-sidecar-is-truncated","errorCode":null,"errorMessage":"provenance sidecar is truncated","messagePattern":"provenance sidecar is truncated","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"core/provenance.py","lineNumber":91,"sourceCode":"        self._file = open(path, \"rb\")\n        try:\n            self._map = mmap.mmap(self._file.fileno(), 0, access=mmap.ACCESS_READ)\n        except Exception:\n            self._file.close()\n            raise\n\n        magic, count, table_len = _HEADER.unpack_from(self._map, 0)\n        if magic != _MAGIC:\n            self.close()\n            raise ValueError(\"not a Maltrail provenance sidecar (bad magic)\")\n\n        self.count = count\n        self._pairs = json.loads(self._map[_HEADER.size:_HEADER.size + table_len].decode(\"utf8\"))\n        self._base = _HEADER.size + table_len\n\n        if self._base + count * _ENTRY_SIZE > len(self._map):\n            self.close()\n            raise ValueError(\"provenance sidecar is truncated\")\n\n    def _hash_at(self, i):\n        offset = self._base + i * _ENTRY_SIZE\n        return struct.unpack_from(\"<Q\", self._map, offset)[0]\n\n    def lookup(self, trail):\n        \"\"\"(reference, source_path) for `trail`, or None.\n\n        The order matches what core/httpd.py's on-demand scan returned, so the caller does not care\n        which of the two answered.\n        \"\"\"\n\n        target = trail_hash(trail)\n        lo, hi = 0, self.count\n        while lo < hi:\n            mid = (lo + hi) // 2\n            if self._hash_at(mid) < target:\n                lo = mid + 1","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/stamparm/maltrail/blob/77cfb06d7606506d101bbcec0786c77166c4255e/core/provenance.py#L73-L109","documentation":"The sidecar header declares `count` entries, but the mapped file ends before the full entry table (header + table + count*_ENTRY_SIZE), so the file is rejected as truncated. This guards reads of hash entries past the end of the mapping.","triggerScenarios":"Incomplete download/copy of the sidecar; writer crashed or was killed mid-flush; disk full during generation; file truncated by log rotation/cleanup tools.","commonSituations":"rsync/scp interrupted, partial cache restore from backup, sidecar written while the process was OOM-killed.","solutions":["Regenerate or re-download the sidecar completely","Compare file size against the header-declared size before opening","Validate with the producing tool's integrity command/checksum","Catch ValueError on open and fall back to a fresh build"],"exampleFix":"// before\nprov = ProvenanceSidecar(path)\n// after\ntry:\n    prov = ProvenanceSidecar(path)\nexcept ValueError:\n    prov = rebuild_sidecar(path)","handlingStrategy":"validation","validationCode":"import os\ndef sidecar_size_ok(path):\n    return os.path.getsize(path) > 64  # at least header + table for nonzero entries","typeGuard":null,"tryCatchPattern":"try:\n    prov = ProvenanceSidecar(path)\nexcept ValueError:\n    prov = rebuild_sidecar(path)  # truncated -> rebuild","preventionTips":["Write sidecars atomically (tmp file + os.rename)","Verify size/checksum after download or copy","Avoid truncating sidecars during rotation/cleanup","Check producer exit status before consuming its output"],"tags":["python","file-format","truncated-file"],"backgroundTag":"file-read-failed","analyzedSha":"77cfb06d7606506d101bbcec0786c77166c4255e","analyzedAt":"2026-09-13T03:50:16.010Z","contentChangedAt":"2026-09-13T03:50:16.010Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}