Freika/dawarich · error · Csv::Detector::DetectionError

Could not detect required columns: latitude, longitude, time

Error message

Could not detect required columns: latitude, longitude, timestamp. Found headers must include recognized aliases for all three.

What it means

Raised as Csv::Detector::DetectionError when the detector cannot map the sample's headers onto the three mandatory columns: latitude, longitude, timestamp. Headers are matched first via Imports::FieldAliases (exact recognized aliases) then a downcased substring fallback ('latitude', 'longitude', 'timestamp'); if any of the three still has no column index, validation fails. It fires during import setup, before any rows are parsed.

Source

Thrown at app/services/csv/detector.rb:91

      # Fallback: substring matching for non-standard headers like "LATITUDE N/S"
      columns[:latitude]  ||= find_header_by_substring(headers, 'latitude')
      columns[:longitude] ||= find_header_by_substring(headers, 'longitude')
      columns[:timestamp] ||= find_header_by_substring(headers, 'timestamp')

      columns
    end

    def find_header_by_substring(headers, keyword)
      normalized = headers.map { |h| h.to_s.downcase.strip }
      normalized.index { |h| h.include?(keyword) }
    end

    def validate_columns!(columns)
      missing = %i[latitude longitude timestamp].reject { |f| columns[f] }
      return if missing.empty?

      raise DetectionError,
            I18n.t('services.csv.detector.required_columns_missing')
    end

    def parse_data_rows(lines, delimiter)
      lines.first(DATA_SAMPLE_SIZE).filter_map do |line|
        row = CSV.parse_line(line, col_sep: delimiter)&.map { |v| v&.strip }
        row if row&.any?(&:present?)
      end
    end

    def detect_coordinate_format(data_rows, columns)
      lat_idx = columns[:latitude]
      lon_idx = columns[:longitude]
      return :decimal_degrees unless lat_idx && lon_idx

      lat_values = data_rows.filter_map { |row| row[lat_idx] }
      lon_values = data_rows.filter_map { |row| row[lon_idx] }
      all_coords = lat_values + lon_values

View on GitHub (pinned to 97fad417c5)

Solutions

1. Open the file and confirm row 1 contains headers including recognized aliases for latitude, longitude, and timestamp (e.g. 'latitude'/'lat', 'longitude'/'lon'/'lng', 'timestamp'/'time' as aliased in Imports::FieldAliases).
2. Rename the offending columns to standard names (Latitude, Longitude, Timestamp) — the substring fallback accepts anything containing 'latitude'/'longitude'/'timestamp' after downcasing.
3. If the file has no header row, add one; if the delimiter is ; or tab, keep it consistent on every line so delimiter detection locks on.
4. Check for a mangled first line (BOM without utf-8 handling, stray quotes) by parsing the header in a console.

Example fix

# before (file headers)
# position_lat,position_lon,time
# -> DetectionError: required columns missing

# after (rename headers)
# latitude,longitude,timestamp
# 52.5200,13.4050,2024-05-01T10:00:00Z

sample = File.foreach(path, chomp: true).lazy.reject(&:empty?).first(12)
headers = CSV.parse_line(sample.first || '')&.map { |h| h.to_s.downcase.strip } || []
%w[latitude longitude timestamp].all? { |k| headers.any? { |h| h.include?(k) } } || raise('headers missing required columns')

def csv_with_required_headers?(path)
  header = CSV.parse_line(File.foreach(path, chomp: true).next || '')
  norm = (header || []).map { |h| h.to_s.downcase.strip }
  %w[latitude longitude timestamp].all? { |k| norm.any? { |h| h.include?(k) } }
rescue StandardError
  false
end

begin
  Csv::Detector.new(path).call
rescue Csv::Detector::DetectionError => e
  import.update!(status: :failed, error_message: 'CSV must contain columns recognizable as latitude, longitude, and timestamp')
end

When it happens

Trigger: A CSV whose headers use unrecognized names for the required fields (e.g. 'lat_dd'/'lon_dd' with no 'latitude' substring, 'time' instead of a 'timestamp'-containing header), a first data row mistaken for headers (headerless CSV), a wrong delimiter guess that merges all headers into one cell, or headers hidden behind a BOM/quotes that break parsing.

Common situations: Users exporting from a tracker app whose column names differ (e.g. 'Latitude N/S' works via substring but 'position_lat' does not), Excel exports saving semicolon-delimited while detection picked comma, Ukrainian/localized header names, a JSON or KML file mislabeled .csv.

Related errors

• zip has no entries
• GPX parse error: %{message}
• Import source cannot be nil
• entry exceeds #{MAX_EXTRACTED_SIZE} bytes
• blank token

AI-assisted analysis of Freika/dawarich@97fad417c5 (2026-08-21). Data as JSON: /api/errors/e2594b39e92c2b2e. Report an issue: GitHub.