DMS False Alarms per Hour: An Event-Based Evaluation Guide
Evaluate a driver monitoring system (DMS) at the level of the warning a driver receives, alongside the behaviors it misses. A useful acceptance report combines false alerts per eligible driving hour, event recall, and time to the first valid alert. Frame accuracy alone cannot answer those questions.
This guide proposes an evaluation protocol for training-data buyers and model teams. It is not a regulatory test, a safety certification, or a measured result for any product. The matching rules below are design choices to agree before testing. Start with a documented driver and occupant monitoring data specification.
Key Takeaways - Define a warning event and the eligible observation time before calculating an alert rate. - Match alerts to reference events one to one; retain late, duplicate and unmatched alerts in the audit log. - Freeze thresholds and temporal post-processing on validation data before evaluating the held-out test set. - Report raw counts, exposure hours and excluded time with every rate.
Why frame accuracy does not measure warning burden
A long normal-driving interval can contribute many more frames than a brief target behavior. Google's classification metrics guide explains why accuracy can conceal poor detection when class frequencies are unequal. Its definitions also distinguish precision, recall and false positive rate.
A false alert per hour has a different denominator from a classification false positive rate. The former divides unmatched alert events by eligible observation hours; the latter divides false-positive classifications by all negative classifications. Do not label one as the other or compare them without a common evaluation unit.
Step 1: Define eligible time and keep an exclusion ledger
For this proposed protocol, eligible time is the union of reviewed intervals in which the system is expected to monitor the driver and the reference behavior can be judged. Record interval boundaries, session identifier and exclusion reason. Do not sum overlapping windows as if they were independent hours.
Agree how parked vehicles, absent drivers, missing frames and uncertain reference labels are handled. Exclusion from behavior scoring must not erase a system-availability problem: report total recording time, eligible time and excluded time separately. Keep warning events during excluded intervals in a separate availability review rather than silently deleting them.
Short positive clips are useful for behavior coverage but cannot establish a normal-driving false-alert rate. Purchase sustained negative exposure as well as rare events. Keep staged, synthetic and naturalistic recordings in separate result tables; their mixture is a sampling decision, not an estimate of deployment prevalence. The synthetic-versus-real evaluation guide explains the dataset distinction.
Step 2: Turn model output into auditable alert events
Specify the score threshold, required persistence, release condition, minimum gap for merging and any suppression period. Save both the candidate signal and the emitted-warning log. Evaluate the released warning policy, not an unreported choice that makes the score look better.
For example, adjacent positive frames may become one warning after persistence and merging. Extending suppression can reduce the number of repeated alerts while also delaying a later valid warning. Therefore, changing suppression creates a new system configuration and requires a fresh comparison, even if the neural model is unchanged.
Step 3: Match alerts once, with a declared time window
Store reference event start and end times using the behavior-boundary annotation protocol. Define an acceptable alert window for each reference event. A simple project rule can allow the first alert from event onset until an agreed deadline; an early-warning task needs a different window. This guide supplies no universal deadline.
Use a deterministic one-to-one matcher. Process alerts by emission time and assign each to the earliest-starting compatible, unmatched reference event whose window contains it; break equal starts by reference identifier. Compatibility includes the monitored person and behavior class. Log the chosen match. One alert cannot satisfy several reference events.
Under this proposed strict counting rule, additional alerts for an already matched event remain unmatched and count in the false-alert numerator. Label them as duplicates so readers can distinguish them from alerts during normal behavior. Late alerts remain unmatched too. If a different acceptance contract excludes duplicates from its false-alert rate, name that alternative and publish the duplicate count alongside it; the rates are not interchangeable.
Step 4: Calculate counts before percentages
The following numbers are entirely fictional and demonstrate arithmetic, not performance. Suppose six eligible driving hours contain four reference events. The warning log contains five emitted alerts: two valid matches, one duplicate, one late alert and one unrelated alert.
| Quantity | Illustrative calculation | Interpretation |
|---|---|---|
| Event recall | 2 matched references / 4 references = 50% | Two reference events were missed under the declared window |
| Event precision | 2 matched alerts / 5 alerts = 40% | Uses the strict unmatched-alert rule above |
| False alerts per eligible hour | 3 unmatched alerts / 6 hours = 0.5 per hour | Includes duplicate, late and unrelated alerts |
| First-alert delay | Matched alert time minus reference onset | Summarize matched events; disclose misses separately |
If there are no eligible hours, report the alert rate as unavailable. If there are no positive reference events, event recall is unavailable. If no alert is emitted, precision is unavailable. Preserve the zero denominator rather than replacing the result with a perfect score. Zero observed false alerts over a finite recording is not evidence of a zero deployment rate.
Step 5: Select the policy without tuning on the test set
A score curve is useful before selecting the final warning policy. The scikit-learn precision–recall example describes threshold-dependent precision and recall, and distinguishes average precision from trapezoidal area. A frame-level curve still does not include your event merging or eligible-hour denominator.
Choose the threshold and temporal rules on validation sessions. Then freeze the full configuration and run the event protocol on held-out sessions. The scikit-learn threshold-tuning guide separates score estimation from the decision rule and warns against reusing model-training data for threshold tuning. Keep people and recording sessions separated as described in the leakage audit.
Compare policies on the same eligible intervals. Break down results by relevant, documented conditions such as daylight, darkness, eyewear and camera view. Attach positive-event counts and hours to each slice; a thin slice is a coverage gap, not a stable ranking. Adjacent frames from one drive are not independent test drives. Any uncertainty interval needs a declared method that respects the recording structure.
What to request in the acceptance package
Ask for an eligibility ledger, reference-event file, raw score export, emitted-alert log, match table and a configuration manifest. The manifest should identify model version, dataset version, threshold, temporal rules and matcher version. Include the review decisions for uncertain annotations and a list of excluded intervals.
A reviewer should be able to reproduce each numerator and denominator without guessing how warnings were merged. Use the annotation quality checklist to assign evidence and rework ownership. Agree acceptable alert burden, missed-event cost and required coverage with the actual deployment team; a generic metric guide cannot set those limits.
Sources and editorial scope
The linked Google and scikit-learn documents support the general metric and threshold concepts. The hourly denominator, matching procedure and acceptance files are editorial recommendations, not rules attributed to those sources. Documentation was checked on November 2, 2026. No original experiment, customer result or compliance claim is made.
Prepared by VWU.AI's website editorial team for training-data procurement discussions. See company information or contact the team about a data specification.