VWU.AI · Published

Driver Distraction Datasets: Labels and Hard Negatives

Driver Distraction Datasets: Labels and Hard Negatives — Actor → Object → Action evidence
Original concept diagram, not measured data. Review the evidence at each step before expanding the dataset.

A driver distraction dataset is easier to evaluate when each label identifies an actor, an observable action and the conditions needed to judge it. “Phone present,” “driver holding a phone” and “driver looking at a phone” describe different evidence. Combining them into one undocumented class creates ambiguity before model training starts.

This guide helps buyers define distraction-related video annotations and the confusing examples that should accompany them. It extends the DMS and OMS procurement checklist with an annotation-focused review method. The guide pairs hypothetical labelling decisions with one worked example from a VWU.AI video.

Key Takeaways

Choose the action your model actually needs to recognize

Define the intended output before assembling a list of distracting behaviors. An object detector needs object labels. An action recognizer needs action evidence over time. A gaze estimator needs a definition of gaze target and uncertainty. A system that combines these outputs needs a stated rule for combining them.

The Drive&Act project separates long-running tasks, semantic actions and object interactions. This hierarchy is a useful reminder that a training label can describe different levels of behavior. Its research-only usage terms also need to be distinguished from a commercial project's licensing requirements. The dataset's structure is a reference, not a ready-made purchasing specification.

For every proposed class, write the observable condition, target actor, required context and allowed co-occurring labels. Avoid assuming that a visible action measures a person's internal attention. A driver can appear to face forward while the evidence needed for a cognitive-state judgment is absent.

Build a confusion table, not just a positive class list

A hard negative resembles the target but does not meet its definition. It is useful only in relation to a specific label. A passenger using a phone can be negative for “driver phone interaction” and positive for “passenger phone interaction” at the same time.

An unknown example is different: the evidence is insufficient to decide. Keep that distinction in the annotation policy and in evaluation. Counting unobservable cases as confident negatives can distort what the dataset teaches and what the test measures.

Target definition to agree Candidate negative Candidate unknown or review case
Driver holds and operates a handheld phone Passenger operates a phone Driver's hands and object are hidden
Driver reaches toward a defined rear-seat region Driver changes gear Camera crop excludes the destination
Driver visibly drinks from a container Container rests in a holder Container obscures the mouth and contact is unclear
Driver turns head toward the passenger Driver looks toward a mirror Face is out of view during the turn
Driver manipulates an infotainment control Hand rests near the control Touch target cannot be identified

These distinctions need project-specific evidence. For example, a hand near a screen does not by itself prove a touch. The DMD paper describes driver-monitoring material covering distraction, gaze, hand interaction and temporal actions. Its separate recording domains support keeping these concepts explicit rather than reducing them to a single unexplained distraction flag.

Preserve actor roles when several people share the frame

Store both a stable track identity and the role needed by the task. “Person 2” can be stable within a clip yet fail to explain whether that person is the driver. Seat, role and track identity should be separate fields when the dataset uses all three.

Specify how roles are assigned when the camera view changes, a person leaves the image or a clip contains an ambiguous seating arrangement. Do not assume a left-side face is always the driver. Record steering configuration only when it is known, and make uncertainty reviewable.

For interactions, name the actor and target. “Passing an object” is incomplete if the project needs to distinguish driver-to-passenger from passenger-to-driver movement. When the target is occluded, record that limitation rather than inventing a relationship. Review a few multi-occupant examples before agreeing the full annotation schema.

Driver Distraction Datasets: Labels and Hard Negatives — Actor → Object → Action evidence
Original concept diagram, not measured data. Review the evidence at each step before expanding the dataset.

Define transitions and overlapping labels

Decide when an action starts and ends, how interruptions are handled, and whether multiple labels can coexist. A driver can hold a phone while looking forward, then look down while continuing to hold it. A single clip-level class cannot express that sequence without a rule that discards information.

Consider a fictional eight-second recording. From 1.0 to 2.5 seconds the driver picks up a phone. From 2.5 to 5.0 seconds the phone remains in the driver's hand. The driver's gaze direction is only assessable from 3.0 to 4.0 seconds. The project could annotate separate hand-object and gaze events rather than label the entire clip “looking at phone.”

Fictional interval Evidence available Example annotation decision
1.0 to 2.5 seconds Visible pickup motion phone_pickup, driver actor
2.5 to 5.0 seconds Phone visibly held phone_held, driver actor
3.0 to 4.0 seconds Eyes and target relation assessable Apply the agreed gaze label after review
Remaining gaze intervals Insufficient gaze evidence gaze_unknown, not “looking forward”

These times are illustrative. A real protocol needs its own boundary definitions and must allow a reviewer to reject an asserted gaze label even when a phone action is clear. Keep the original context so adjudicators can inspect the transition.

Worked example: when does holding the phone begin?

The phone is visible on the driver’s lap in the first frame. If the target is “driver holding a phone”, labelling the whole clip positive would count the time before the driver picks it up. The four intervals below follow the same phone through the shot.

Watch the 6-second cabin clip · Download the four annotated intervals (CSV)

Time (s)Phone state in the CSVVisible evidence
0–0.75phone_on_lapThe phone rests flat on the driver’s lap. It is visible before the hand reaches for it.
0.75–1.75reachingA hand moves toward the phone and partly covers it. The exact moment a firm grip begins is unclear.
1.75–2.75liftingThe phone rises and tilts with the hand, moving away from its original flat position.
2.75–6.041667heldThe driver holds the phone in one hand with their head angled down. The clip ends with the phone still in hand.

The clip contains 145 frames at 24 fps. The CSV counts frames from zero, includes each start frame and excludes the end frame. Frame 144 is the last visible frame. Seconds are rounded to six decimal places; use frame numbers for exact alignment.

Screen activity is marked unknown throughout. The screen content and individual taps cannot be resolved, and head angle alone does not establish what the driver is looking at. The boundaries describe this clip and can be reviewed against the original frames.

Reproduce the four intervals in ASAM OpenLABEL

OpenLABEL includes both ends of a frame interval. For example, the CSV gives the reaching motion a start of 18 and an exclusive end of 42. In OpenLABEL, the same motion spans frames 18 through 41. Frame 42 belongs to the lifting motion.

Download the OpenLABEL annotations (JSON) · Download the CSV, conversion script and instructions (ZIP)

To check your importer, compare its label for frames 41 and 42 with the video. The screen activity remains unknown in both frames.

Prevent the background from becoming a label shortcut

Build the coverage plan across relevant subjects, views, lighting, clothing, object appearance and vehicle interiors. Check whether every positive class appears with a unique background, actor or recording session. A model may then have a convenient cue unrelated to the behavior the buyer intended to teach.

Construct paired review cases where feasible: similar camera conditions with different actions, or the same action under different conditions. This is a proposed diagnostic design, not a guarantee that paired examples eliminate shortcuts. Avoid declaring a dataset diverse merely because it has many frames from the same session.

Keep related people and source sequences grouped during splitting. The scikit-learn cross-validation guide describes subject-related dependence and group isolation. Apply the generalization question consistently to generated variants and near-duplicate clips as well.

Review the batch by error type and usable evidence

Use a pilot to test the label manual, export format and data loader together. Ask reviewers to classify a shared subset independently and retain disagreements. A disagreement may expose an unclear rule, a difficult view or a genuine annotation error; those causes require different fixes.

Break review findings into missing events, extra events, wrong classes, actor confusion, timing errors and unsupported labels. Report which examples were evaluable. A high overall agreement figure can conceal a weak rare class or a systematic error on a particular view.

For model evaluation, distinguish event-level behavior detection from frame-level classification. Agree how predicted and reference events are matched, how repeated alerts count and which normal-driving time is eligible for a false-alert rate. Do not compare two reported numbers unless their task definitions and denominators align.

Before batch acceptance, confirm:

Use existing in-cabin examples to discuss the visual task, then describe the required distinctions in the dataset requirements form. Request sample confirmation before treating any proposed label or scenario as an agreed delivery capability.

Editorial note

Published by VWU.AI on October 29, 2026. This guide is a sourced annotation-planning guide, not a new benchmark, customer case study or performance report. The four phone intervals were reviewed against the footage on November 18, 2026. The separate eight-second example is hypothetical. Public sources were checked on the review date; send corrections to chan@vwu.ai.