Many review programmes use a fixed percentage of completed work. It is simple, predictable and easy to budget. But it assumes that error is distributed reasonably evenly across the population being sampled.
In annotation, video review and other specialist data work, that assumption often fails. New contributors, unfamiliar categories, difficult scenes, changing tool behaviour and recent guideline updates can create very different risk profiles.
Start with a useful baseline
Random sampling should remain part of the design. It protects against blind spots and gives an unbiased view of performance when implemented correctly. The problem begins when random sampling is the only signal used to allocate review effort.
A mature programme can combine a random baseline with additional sampling triggered by risk.
Allocate review to uncertainty and impact
Consider whether a task contains an unfamiliar object category, a rare scenario, an incomplete sequence or a disagreement between model outputs and human labels. Review intensity may also increase when a contributor is new, recent quality has fallen or a category is known to be difficult.
Risk should reflect more than the probability of an error. A minor metadata mistake and a missed safety relevant object may have very different consequences. Review allocation needs a view of severity as well.
Keep the scoring fair
Adaptive sampling has a catch. Contributors assigned harder work are more likely to be reviewed and may therefore appear to have worse performance. A simple acceptance rate can punish people for taking on difficult queues.
Contributor scoring should account for task difficulty, confidence in the estimate and sample size. It should distinguish a temporary performance dip from a stable pattern and allow people to recover when performance improves.
Use what the reviews teach you
If error concentration changes, sampling should change with it. When reviewers repeatedly correct the same class of mistake, the programme should ask whether instructions, interfaces or upstream systems need to change.
That is the shift from review as a cost centre to review as a decision system: one that helps protect quality while revealing where the production process needs to improve.