{"record":{"id":"2057fa681f79504b","repo":"commaai/openpilot","slug":"data-to-short-to-contain-nal-unit-header","errorCode":null,"errorMessage":"data to short to contain nal unit header","messagePattern":"data to short to contain nal unit header","errorType":"exception","errorClass":"VideoFileInvalid","httpStatus":null,"severity":"error","filePath":"openpilot/tools/lib/vidindex.py","lineNumber":180,"sourceCode":"\n  # length of NAL unit is byte count up to next NAL unit start index\n  nal_unit_len = (pos if pos != -1 else len(dat)) - nal_unit_start\n  if DEBUG:\n    print(\"  nal_unit_len:\", nal_unit_len)\n  return nal_unit_len\n\ndef get_hevc_nal_unit_type(dat: bytes, nal_unit_start: int) -> HevcNalUnitType:\n  # 7.3.1.2 NAL unit header syntax\n  # nal_unit_header( ) {    // descriptor\n  #   forbidden_zero_bit    f(1)\n  #   nal_unit_type         u(6)\n  #   nuh_layer_id          u(6)\n  #   nuh_temporal_id_plus1 u(3)\n  # }\n  header_start = nal_unit_start + NAL_UNIT_START_CODE_SIZE\n  nal_unit_header = dat[header_start:header_start + NAL_UNIT_HEADER_SIZE]\n  if len(nal_unit_header) != 2:\n    raise VideoFileInvalid(\"data to short to contain nal unit header\")\n  nal_unit_type = HevcNalUnitType((nal_unit_header[0] >> 1) & 0x3F)\n  if DEBUG:\n    print(\"  nal_unit_type:\", nal_unit_type.name, f\"({nal_unit_type.value})\")\n  return nal_unit_type\n\ndef get_hevc_slice_type(dat: bytes, nal_unit_start: int, nal_unit_type: HevcNalUnitType) -> tuple[int, bool]:\n  # 7.3.2.9 Slice segment layer RBSP syntax\n  # slice_segment_layer_rbsp( ) {\n  #   slice_segment_header( )\n  #   slice_segment_data( )\n  #   rbsp_slice_segment_trailing_bits( )\n  # }\n  # ...\n  # 7.3.6.1 General slice segment header syntax\n  # slice_segment_header( ) {                                             // descriptor\n  #   first_slice_segment_in_pic_flag                                     u(1)\n  #   if( nal_unit_type >= BLA_W_LP && nal_unit_type <= RSV_IRAP_VCL23 )\n  #     no_output_of_prior_pics_flag                                      u(1)","sourceCodeStart":162,"sourceCodeEnd":198,"githubUrl":"https://github.com/commaai/openpilot/blob/516ec1e68203439a73f340f1d0b3b91eabc626ee/openpilot/tools/lib/vidindex.py#L162-L198","documentation":"Raised by get_hevc_nal_unit_type() in vidindex.py when fewer than 2 bytes remain after the NAL start code, so the 2-byte HEVC NAL unit header (forbidden_zero_bit, nal_unit_type, nuh_layer_id, nuh_temporal_id_plus1 per ITU-T H.265 7.3.1.2) cannot be read. A VideoFileInvalid, it almost always means the file ends with a bare start code or the NAL length computation ran past the end of a truncated file.","triggerScenarios":"hevc_index() encounters a NAL unit at the very end of the buffer where dat[header_start:header_start+2] yields fewer than 2 bytes — i.e. the file ends with 00 00 00 01 and nothing after it, or a truncated final NAL. Also reachable by calling get_hevc_nal_unit_type(dat, i) directly with an i pointing into the last 1-5 bytes.","commonSituations":"Route files cut off mid-upload, segments whose loggerd write was interrupted, or trailing start-code padding bytes at the end of a stream. Anyone batch-indexing old routes with incomplete final segments hits this.","solutions":["Pass allow_corrupt=True to hevc_index so the truncated tail is skipped instead of raising.","Confirm the file is complete (size matches the uploader log / rlog-derived camera path); re-pull the segment if truncated.","Strip trailing zero padding / dangling start codes before indexing if you control the pipeline.","As a diagnostic, hexdump the last 32 bytes of the file to confirm it ends with a dangling 00 00 00 01."],"exampleFix":"// before\nframes, prefix, prefix_dat = hevc_index(f'{seg}/camera.hevc')\n\n// after\nframes, prefix, prefix_dat = hevc_index(f'{seg}/camera.hevc', allow_corrupt=True)","handlingStrategy":"try-catch","validationCode":"import os\nSTART_CODE = b'\\x00\\x00\\x00\\x01'\nsize = os.path.getsize(path)\nwith open(path, 'rb') as f:\n    f.seek(size - 4)\n    tail = f.read()\n# file ending with a dangling start code will fail the header read\nends_with_dangling_sc = tail == START_CODE or (size >= 4 and tail.lstrip(b'\\x00') == b'')","typeGuard":null,"tryCatchPattern":"try:\n    frames, prefix, prefix_dat = hevc_index(path)\nexcept VideoFileInvalid as e:\n    if 'nal unit header' in str(e):\n        frames, prefix, prefix_dat = hevc_index(path, allow_corrupt=True)  # skip truncated tail\n    else:\n        raise","preventionTips":["Treat segments whose size doesn't match the route manifest as incomplete and re-download.","Pass allow_corrupt=True for any file that may have an interrupted final NAL.","Log file sizes when indexing in batch so truncation patterns are visible."],"tags":["video","hevc","parsing","truncation","openpilot"],"backgroundTag":null,"analyzedSha":"516ec1e68203439a73f340f1d0b3b91eabc626ee","analyzedAt":"2026-08-15T00:17:37.461Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}