Driver Distraction Datasets: Labels and Hard Negatives
A driver distraction dataset is easier to evaluate when each label identifies an actor, an observable action and the conditions needed to judge it. “Phone present,” “driver holding a phone” and “driver looking at a phone” describe different evidence. Combining them into one undocumented class creates ambiguity before model training starts.
This guide helps buyers define distraction-related video annotations and the confusing examples that should accompany them. It extends the DMS and OMS procurement checklist with an annotation-focused review method. The guide pairs hypothetical labelling decisions with one worked example from a VWU.AI video.
Key Takeaways
- Define labels from visible evidence and identify the person performing the action.
- Distinguish a negative example from an example that cannot be judged.
- Include transitions, overlapping actions and similar non-target activities.
- Validate the exported labels and review errors by class and condition.
Choose the action your model actually needs to recognize
Define the intended output before assembling a list of distracting behaviors. An object detector needs object labels. An action recognizer needs action evidence over time. A gaze estimator needs a definition of gaze target and uncertainty. A system that combines these outputs needs a stated rule for combining them.
The Drive&Act project separates long-running tasks, semantic actions and object interactions. This hierarchy is a useful reminder that a training label can describe different levels of behavior. Its research-only usage terms also need to be distinguished from a commercial project's licensing requirements. The dataset's structure is a reference, not a ready-made purchasing specification.
For every proposed class, write the observable condition, target actor, required context and allowed co-occurring labels. Avoid assuming that a visible action measures a person's internal attention. A driver can appear to face forward while the evidence needed for a cognitive-state judgment is absent.
Build a confusion table, not just a positive class list
A hard negative resembles the target but does not meet its definition. It is useful only in relation to a specific label. A passenger using a phone can be negative for “driver phone interaction” and positive for “passenger phone interaction” at the same time.
An unknown example is different: the evidence is insufficient to decide. Keep that distinction in the annotation policy and in evaluation. Counting unobservable cases as confident negatives can distort what the dataset teaches and what the test measures.
| Target definition to agree | Candidate negative | Candidate unknown or review case |
|---|---|---|
| Driver holds and operates a handheld phone | Passenger operates a phone | Driver's hands and object are hidden |
| Driver reaches toward a defined rear-seat region | Driver changes gear | Camera crop excludes the destination |
| Driver visibly drinks from a container | Container rests in a holder | Container obscures the mouth and contact is unclear |
| Driver turns head toward the passenger | Driver looks toward a mirror | Face is out of view during the turn |
| Driver manipulates an infotainment control | Hand rests near the control | Touch target cannot be identified |
These distinctions need project-specific evidence. For example, a hand near a screen does not by itself prove a touch. The DMD paper describes driver-monitoring material covering distraction, gaze, hand interaction and temporal actions. Its separate recording domains support keeping these concepts explicit rather than reducing them to a single unexplained distraction flag.
Preserve actor roles when several people share the frame
Store both a stable track identity and the role needed by the task. “Person 2” can be stable within a clip yet fail to explain whether that person is the driver. Seat, role and track identity should be separate fields when the dataset uses all three.
Specify how roles are assigned when the camera view changes, a person leaves the image or a clip contains an ambiguous seating arrangement. Do not assume a left-side face is always the driver. Record steering configuration only when it is known, and make uncertainty reviewable.
For interactions, name the actor and target. “Passing an object” is incomplete if the project needs to distinguish driver-to-passenger from passenger-to-driver movement. When the target is occluded, record that limitation rather than inventing a relationship. Review a few multi-occupant examples before agreeing the full annotation schema.
Define transitions and overlapping labels
Decide when an action starts and ends, how interruptions are handled, and whether multiple labels can coexist. A driver can hold a phone while looking forward, then look down while continuing to hold it. A single clip-level class cannot express that sequence without a rule that discards information.
Consider a fictional eight-second recording. From 1.0 to 2.5 seconds the driver picks up a phone. From 2.5 to 5.0 seconds the phone remains in the driver's hand. The driver's gaze direction is only assessable from 3.0 to 4.0 seconds. The project could annotate separate hand-object and gaze events rather than label the entire clip “looking at phone.”
| Fictional interval | Evidence available | Example annotation decision |
|---|---|---|
| 1.0 to 2.5 seconds | Visible pickup motion | phone_pickup, driver actor |
| 2.5 to 5.0 seconds | Phone visibly held | phone_held, driver actor |
| 3.0 to 4.0 seconds | Eyes and target relation assessable | Apply the agreed gaze label after review |
| Remaining gaze intervals | Insufficient gaze evidence | gaze_unknown, not “looking forward” |
These times are illustrative. A real protocol needs its own boundary definitions and must allow a reviewer to reject an asserted gaze label even when a phone action is clear. Keep the original context so adjudicators can inspect the transition.
Worked example: when does holding the phone begin?
The phone is visible on the driver’s lap in the first frame. If the target is “driver holding a phone”, labelling the whole clip positive would count the time before the driver picks it up. The four intervals below follow the same phone through the shot.
Watch the 6-second cabin clip · Download the four annotated intervals (CSV)
| Time (s) | Phone state in the CSV | Visible evidence |
|---|---|---|
| 0–0.75 | phone_on_lap | The phone rests flat on the driver’s lap. It is visible before the hand reaches for it. |
| 0.75–1.75 | reaching | A hand moves toward the phone and partly covers it. The exact moment a firm grip begins is unclear. |
| 1.75–2.75 | lifting | The phone rises and tilts with the hand, moving away from its original flat position. |
| 2.75–6.041667 | held | The driver holds the phone in one hand with their head angled down. The clip ends with the phone still in hand. |
The clip contains 145 frames at 24 fps. The CSV counts frames from zero, includes each start frame and excludes the end frame. Frame 144 is the last visible frame. Seconds are rounded to six decimal places; use frame numbers for exact alignment.
Screen activity is marked unknown throughout. The screen content and individual taps cannot be resolved, and head angle alone does not establish what the driver is looking at. The boundaries describe this clip and can be reviewed against the original frames.
Reproduce the four intervals in ASAM OpenLABEL
OpenLABEL includes both ends of a frame interval. For example, the CSV gives the reaching motion a start of 18 and an exclusive end of 42. In OpenLABEL, the same motion spans frames 18 through 41. Frame 42 belongs to the lifting motion.
Download the OpenLABEL annotations (JSON) · Download the CSV, conversion script and instructions (ZIP)
To check your importer, compare its label for frames 41 and 42 with the video. The screen activity remains unknown in both frames.
Prevent the background from becoming a label shortcut
Build the coverage plan across relevant subjects, views, lighting, clothing, object appearance and vehicle interiors. Check whether every positive class appears with a unique background, actor or recording session. A model may then have a convenient cue unrelated to the behavior the buyer intended to teach.
Construct paired review cases where feasible: similar camera conditions with different actions, or the same action under different conditions. This is a proposed diagnostic design, not a guarantee that paired examples eliminate shortcuts. Avoid declaring a dataset diverse merely because it has many frames from the same session.
Keep related people and source sequences grouped during splitting. The scikit-learn cross-validation guide describes subject-related dependence and group isolation. Apply the generalization question consistently to generated variants and near-duplicate clips as well.
Review the batch by error type and usable evidence
Use a pilot to test the label manual, export format and data loader together. Ask reviewers to classify a shared subset independently and retain disagreements. A disagreement may expose an unclear rule, a difficult view or a genuine annotation error; those causes require different fixes.
Break review findings into missing events, extra events, wrong classes, actor confusion, timing errors and unsupported labels. Report which examples were evaluable. A high overall agreement figure can conceal a weak rare class or a systematic error on a particular view.
For model evaluation, distinguish event-level behavior detection from frame-level classification. Agree how predicted and reference events are matched, how repeated alerts count and which normal-driving time is eligible for a false-alert rate. Do not compare two reported numbers unless their task definitions and denominators align.
Before batch acceptance, confirm:
- Every class has an inclusion rule, exclusions and an unknown policy.
- Actor roles and interaction targets survive export without losing meaning.
- Confusing negatives and action transitions appear in the pilot.
- Concurrent labels and time conventions are tested in the training reader.
- Source groups, near-duplicates and evaluation boundaries are checked.
- Defects have a correction owner, rule version and reinspection record.
Use existing in-cabin examples to discuss the visual task, then describe the required distinctions in the dataset requirements form. Request sample confirmation before treating any proposed label or scenario as an agreed delivery capability.
Editorial note
Published by VWU.AI on October 29, 2026. This guide is a sourced annotation-planning guide, not a new benchmark, customer case study or performance report. The four phone intervals were reviewed against the footage on November 18, 2026. The separate eight-second example is hypothetical. Public sources were checked on the review date; send corrections to chan@vwu.ai.