Driver Monitoring Dataset Splits: Prevent Data Leakage
A driver-monitoring dataset split should separate the information your evaluation claims to hold out. If the claim is performance on unfamiliar drivers, keep those drivers out of training and model-selection data. If the claim concerns new recordings of known drivers, define that different boundary explicitly. Randomly separating video frames does not establish either condition.
Consider a hypothetical delivery containing an original cabin clip, a cropped version and several overlapping windows. Different filenames could send related observations into both training and testing. This guide proposes a source-manifest and overlap-audit workflow for finding that problem before training. It is an engineering recommendation, not a report of a VWU.AI experiment or a guarantee of generalization.
Key Takeaways
- Define what must be unseen before choosing a splitting method.
- Preserve recording and transformation lineage before extracting windows.
- Check every prohibited grouping axis across training, validation and test sets.
- A passing identifier audit establishes recorded separation, not the absence of every possible leak.
1. Write the evaluation claim before assigning partitions
Start with one sentence describing the intended test population. “Unfamiliar drivers in familiar cabins” and “familiar drivers in a new camera view” test different conditions. A repeated vehicle is not automatically a protocol violation in the first case, and an unfamiliar-driver result does not establish performance in an unseen vehicle.
Translate that claim into explicit separation rules. For an unfamiliar-person protocol, identify every relevant person in multi-occupant recordings, rather than assuming the driver identifier covers passengers. Use consistent pseudonymous identifiers; do not place names or contact information in a public manifest.
Also separate validation from testing. Validation supports decisions such as model selection and threshold choice. Reserve an independent test set for the final declared evaluation. The scikit-learn guidance on data leakage explains why information from evaluation data must not enter learned preprocessing or model fitting. Apply that rule after the split, including feature selection and fitted normalization.
For the broader production context, see our driver and occupant monitoring data service. A service description cannot replace a project-specific evaluation protocol.
2. Build a manifest that preserves parent relationships
Use one row per delivered asset and keep identifiers stable across exports. The following fields are a proposed project format, not a required industry standard or a claim about a completed customer delivery.
| Field | What it records | What to verify |
|---|---|---|
sample_id |
Unique delivered asset | Exactly one partition per asset |
participant_groups |
Relevant people, as pipe-delimited pseudonyms | Every identity required by the protocol is represented |
recording_id |
Globally scoped continuous recording | IDs are not reused by different suppliers or sessions |
lineage_group_id |
Root source shared by derived variants | Crops and re-encodes retain their source link |
split |
train, validation or test |
Value matches the frozen split specification |
An operational manifest may also need vehicle, camera, session, parent asset, timestamps, transformation and label-version fields. Avoid an unscoped identifier such as session_01 when separate collections might reuse it. Define the namespace and document how missing identifiers will be resolved. Missing provenance should fail the relevant audit, rather than be treated as evidence that samples are independent.
3. Assign source groups before expanding video derivatives
Decide the partition of a root recording before producing overlapping windows, extracted frames or transformed variants. Propagate that assignment to its descendants. If derivatives already exist, reconstruct the parent links before evaluating the split.
Original workflow illustration created for VWU.AI; not experimental data. The diagram shows source-lineage grouping, not proof that every driver or environment is held out.
For a strict new-recording holdout, adjacent cuts from one continuous recording should not become independent simply because an export gave them new clip IDs. Keep a root identifier above the cut. For synthetic variants, record known shared reference assets or identity sources where applicable. A different prompt or seed alone does not prove independence; an identical prompt alone does not prove two samples share a source.
Exact file hashes help identify byte-identical copies. They do not establish that differently encoded, cropped or visually related samples are independent. Similarity screening can suggest records for review, but this guide provides no universal similarity threshold or detection guarantee. Keep uncertainty in the manifest and evaluation report. Our synthetic-versus-real evaluation guide discusses the separate question of which data population the test represents.
4. Enforce each required axis, not just a combined label
The same person can appear in several sessions. A combined key such as (person_A, session_1) differs from (person_A, session_2), so splitting those tuple groups can still place person A on both sides. A single grouping column only protects the grouping information you actually encoded.
When several axes must all be disjoint, one practical approach is to connect assets that share any prohibited identity, session or source root, and then assign entire connected components to partitions. This is our proposed grouping procedure, not an automatic feature or a guarantee supplied by scikit-learn. Include transitive connections: two recordings sharing one relevant occupant may link otherwise separate groups.
If that construction produces a very large component, the requested evaluation may be infeasible with the available collection. Gather a genuinely independent cohort, change the study design, or narrow the stated claim. Do not silently cut source links or repeatedly reshuffle until the result looks favorable.
5. Choose a splitter with the right expectations
The GroupKFold documentation describes non-overlapping supplied groups between training and testing within each fold. It does not know which unrecorded identities or parent recordings should have been grouped together.
GroupShuffleSplit supports randomized group holdouts. Its size parameters count groups, not individual clips; a fraction of groups need not produce the same fraction of video duration. Test groups can also overlap across repeated splits, so repeated splits are not automatically separate independent test cohorts.
StratifiedGroupKFold attempts to preserve class proportions while separating groups. Large or uneven groups can prevent the desired stratification. Report actual group counts, asset counts, durations and missing labels after splitting. Do not move individual frames out of a protected group merely to make a class-balance table look better.
6. Audit the manifest and freeze the result
Run overlap checks before training and whenever the source inventory changes. The following Python example checks a deliberately strict policy: all listed participants, recording roots and lineage groups must remain in one partition. Adapt the declared axes to your evaluation claim; it is not a universal rule for every longitudinal or personalization study.
import csv
from collections import defaultdict
AXES = ("participant_groups", "recording_id", "lineage_group_id")
PARTITIONS = {"train", "validation", "test"}
owners = {axis: defaultdict(set) for axis in AXES}
sample_ids = set()
seen_partitions = set()
with open("split_manifest.csv", newline="", encoding="utf-8") as handle:
reader = csv.DictReader(handle)
required = {"sample_id", "split", *AXES}
if not required.issubset(reader.fieldnames or []):
raise ValueError("Missing manifest columns")
for row in reader:
sample_id = (row.get("sample_id") or "").strip()
partition = (row.get("split") or "").strip()
if not sample_id or sample_id in sample_ids:
raise ValueError("Missing or repeated sample_id")
if partition not in PARTITIONS:
raise ValueError("Unknown partition")
sample_ids.add(sample_id)
seen_partitions.add(partition)
for axis in AXES:
value = row.get(axis) or ""
values = value.split("|") if axis == "participant_groups" else [value]
if any(not item.strip() for item in values):
raise ValueError(f"Missing required identifier: {axis}")
for item in values:
owners[axis][item.strip()].add(partition)
if seen_partitions != PARTITIONS:
raise ValueError("Expected a non-empty three-way split")
conflicts = {
axis: [key for key, splits in groups.items() if len(splits) > 1]
for axis, groups in owners.items()
}
if any(conflicts.values()):
raise ValueError("Cross-partition group overlap detected")
This audit checks all partition pairs implicitly: an identifier owned by more than one partition fails. It does not detect an incorrectly assigned identity, an omitted participant, an unknown near-duplicate or contamination in a pretrained model. Passing it is evidence about the recorded metadata and chosen rules only.
Keep the accepted manifest, its checksum, source inventory, grouping code, library versions, random seed and audit output together. A seed alone cannot preserve a split when inputs or group definitions change. Fit preprocessing using training data, keep the test inventory fixed during model selection, and record any later protocol revision explicitly.
What to include in a dataset handoff
A useful handoff contains the evaluation claim, identifier dictionary, root-to-derivative map, frozen split manifest, overlap report, distribution summary and unresolved provenance gaps. These artifacts let reviewers inspect the split without accepting an unsupported “leakage-free” label.
Use the video annotation quality checklist for complementary label and export checks. If you need a project-specific production brief, discuss the intended scenarios and acceptance requirements, including which people, recording conditions and source families must be held out.
Editorial ownership and corrections
VWU.AI publishes this guide as a proposed technical workflow. The linked primary documentation was checked on October 29, 2026. The diagrams and code examples are illustrative and do not describe customer results. Learn about VWU.AI, or send a correction with the affected passage and source.
Sources
Official scikit-learn documentation, retrieved October 29, 2026: GroupKFold, GroupShuffleSplit, StratifiedGroupKFold, and common pitfalls: data leakage. The manifest fields, diagrams and audit workflow above are illustrative recommendations, not a documented customer result.