DMS and OMS Training Data: Procurement and Acceptance Checklist
Before buying training data for a driver monitoring system (DMS) or occupant monitoring system (OMS), define the model task, behavior labels and acceptance method. A request for “distraction videos” does not explain which actions count, how to label occlusion or how the buyer will decide that delivery is complete.
This checklist is for algorithm, data and procurement teams planning captured or synthetic video projects. It offers practical recommendations grounded in public research and documentation, rather than an automotive certification standard. VWU.AI's in-cabin behavior data service follows requirements review, sample validation and project-specific delivery. The examples below are illustrative, not customer deliverables or reported performance results.
Key Takeaways
- Define labels through observable actions and include confusing negative examples.
- Test representative samples, annotations and your data loader before scaling production.
- Group related subjects, source sequences or generation sources before splitting data.
- Record dataset acceptance and model evaluation separately; valid files do not establish deployment readiness.
On this page: Task · Examples · Timing · Metadata · Splits · Acceptance · Checklist · Sources
1. Specify the prediction the model must make
Turn broad concepts such as fatigue, distraction and passenger interaction into observable events that reviewers can label consistently. “The driver is interacting with a phone” is a different task from “a phone is visible”: the first needs a relationship among an actor, an object and an action.
The Drive&Act project uses a hierarchy of activities, actions and object interactions. Its structure illustrates why annotation levels belong in dataset design. Its labels and usage terms do not automatically transfer to a commercial project. Drive&Act official project
| Requirement | Question to resolve |
|---|---|
| Prediction unit | Does the model classify a frame, clip, time interval or continuous event? |
| Target actor | How are the driver, front passenger and rear passengers identified? |
| Observability | When eyes, hands or objects are hidden, is the label unknown, ignored or another state? |
| Label relationships | Can turning around and phone interaction overlap, or does one label take precedence? |
| Deployment conditions | Which camera positions, imaging modalities, steering configurations and lighting conditions matter? |
Check: ask the model owner and annotation lead to label the same sample independently. Record disagreements and revise definitions. Do not turn an unobservable mental state into a certain label without stating what additional evidence supports it.
2. Pair positive events with boundaries and negative examples
Choose negative examples that resemble the target action but fail its definition. A review that only asks whether the desired action appears can miss ambiguous labels.
These examples are discussion aids, not universal annotation rules:
| Target event | Example positive definition | Negative or unknown cases to discuss |
|---|---|---|
| Phone interaction | The target actor holds a phone and performs the agreed interaction | A mounted phone; a passenger using a phone; the object is occluded |
| Turning around | Head motion toward the rear reaches the agreed visible state | Checking a mirror; eye movement alone; the face is not visible |
| Eye closure | Visible eyes remain closed during a defined event | Ordinary blinking; sunglasses; blur prevents a judgment |
| Occupant contact | A specified actor makes the defined contact with an identified target | Passing an object; adjusting clothing; occluded contact; uncertain target |
For each label, record inclusion, exclusion, unknown and review-needed cases. Observable eye closure or yawning should not automatically become a diagnosis of physiological drowsiness without the additional evidence the task requires.
Agree the sample mix using task risk, gaps in existing data and validation results. A single positive-to-negative ratio is not suitable for every project. Published scenario videos help discuss appearance; the project still needs its own label definitions.
3. Define event boundaries and video specifications
For temporal annotations, specify how an event starts, how it ends and how boundary error is measured. Labels can use milliseconds or frame indices, but the unit must be explicit. Frame-based labels also need a defined relationship between frame rate and timestamps.
The following fictional record demonstrates a possible format. Its values are examples, not agreed delivery specifications:
{
"clip_id": "example-001",
"source_group_id": "source-example-a",
"origin": "synthetic",
"camera_view": "front",
"duration_ms": 6000,
"events": [
{
"actor_id": "driver-01",
"label": "phone_interaction",
"start_ms": 1800,
"end_ms": 4200,
"visibility": "visible"
}
],
"label_schema_version": "example-v1"
}
The reader should validate 0 ≤ start_ms < end_ms ≤ duration_ms and check actor IDs and labels against the agreed dictionaries. Mark events truncated by a clip boundary instead of assuming the action ends with the file.
Record the container, codec, resolution, frame rate, duration, imaging modality and audio requirements separately. A grayscale appearance does not establish equivalence to a real near-infrared sensor. If the deployment uses that modality, test the imaging assumptions during sample validation. Both parties must confirm feasible formats, annotation fields and production scope.
4. Use metadata to audit coverage and provenance
A delivery manifest should explain where a clip came from, why it belongs in the dataset and how it may be used. Datasheets for Datasets proposes documenting motivation, composition, collection and recommended uses. That provides a public framework for discussing delivery documentation. Gebru and colleagues, revised 2021
Agree fields according to the project and retain unknown values explicitly:
- Identity and versions: clip ID, original sequence or source group, label-schema version and batch version.
- Scenario: actor and seat, view, lighting, vehicle configuration, target action and relevant distractors.
- Provenance: captured or synthetic origin, available source records, permissions and use restrictions.
- Delivery checks: filename, size, checksum, duration, annotation path and review status.
Do not infer unverified demographic attributes from appearance. For synthetic data, describe which generation-source records are available and what remains unknown. For captured data, agree the required permissions, privacy handling and retention arrangements. The project's compliance owner should resolve legal questions.
5. Set group boundaries before clipping and augmentation
Adjacent clips from one recording and repeated clips of one person can carry related information. scikit-learn's grouped cross-validation documentation explains that repeated subject samples can allow a model to learn person-specific features, motivating group-aware evaluation. scikit-learn grouped cross-validation
For an in-cabin project, turn that principle into auditable source groups:
- Choose grouping dimensions based on the generalization question: subjects, original recordings, capture sessions or generation sources.
- Assign clips, duplicates and augmented variants from the same source to a group before allocating training, validation and test sets.
- If several relationships must remain isolated, merge connected samples across those relationships. Simply concatenating IDs does not necessarily keep a person out of multiple groups.
- Check group intersections and duplicates, then check behavior and condition coverage in each split. Isolation and coverage need to be assessed together.
These are applications of the grouping principle, not results automatically guaranteed by a tool. Synthetic variants also need source tracking. Multiple outputs from one seed image should not be treated as independent evaluation evidence without examination.
6. Validate samples before agreeing batch release
Run representative samples through the actual data reader, annotation review and model-evaluation process. Then freeze a versioned acceptance checklist. Selecting only visually attractive examples will not test difficult boundaries.
| Acceptance layer | What to check | Evidence to retain |
|---|---|---|
| Files | Decoding, agreed specifications and manifest-to-label correspondence | Validation logs and missing-item list |
| Scenarios | Actors, seats, actions and targets match the request | Item-level decisions and rejection reasons |
| Labels | Classes, relationships, visibility and event boundaries follow the rules | Review records, disagreements and rule versions |
| Coverage | Agreed target conditions, negatives and rare combinations are represented | Counts broken down by condition |
| Independence | Source groups, duplicates and splits follow the evaluation design | Group-overlap and duplicate reports |
| Model | Task metrics on independent, relevant evaluation data | Condition-level results, limitations and reproducible settings |
Before the batch, agree the sampling procedure, defect categories, rework scope and reinspection process. Thresholds depend on the task and its risks; this guide does not prescribe an unvalidated universal pass rate.
Sign off dataset quality separately from model performance. Evaluate the contribution of synthetic training data using an independent test set representative of target conditions. Visual quality alone cannot establish deployment benefit. Model Cards for Model Reporting emphasizes intended use, evaluation and limitations; it is a reporting framework, not a DMS approval certificate. Mitchell and colleagues, 2019
Final checklist before requesting a quote
- The task, target actor and rules for unknown or occluded states are documented.
- Each target behavior has positive definitions, boundary cases and negatives.
- Video specifications, labels, time units and file organization are agreed.
- Provenance, permissions and use restrictions have explicit recording requirements.
- Grouped splits and duplicate checks have an owner and inspectable outputs.
- Sample acceptance, batch release, rework and model evaluation have separate criteria.
Use the dataset requirements form to describe missing scenarios and delivery needs. VWU.AI discusses sample feasibility and production scope for the specific request. Pricing, quantity, schedules and usage terms are agreed per project.
Editorial ownership and sources
Published by VWU.AI for training-data requirements discussions. This guide synthesizes public research and official documentation; it does not report a new experiment. Tables, fields and JSON are illustrative recommendations, not customer data, model scores or delivery promises. Technical sources were checked on October 26, 2026.
The in-cabin illustration is an existing synthetic visual from the website, not a customer capture. Send corrections or project questions to chan@vwu.ai, or read the VWU.AI service overview.