GOVERN-1.3 / MAP-5.1 / MEASURE-2.11 NIST AI Risk Management Framework

Human oversight for high-risk decisions

Demonstrate that high-risk automated decisions are subject to mandatory human review and approval before execution, with clear risk criteria, enforced workflow controls, and complete audit trails.

Description

What this control does

This control mandates that automated systems, including artificial intelligence and machine learning models, cannot independently execute decisions classified as high-risk without human review and approval. High-risk decisions typically include actions affecting safety, legal compliance, financial liability exceeding thresholds, access to sensitive data, or irreversible changes to production systems. Organizations must define risk thresholds, implement technical controls that pause automated workflows pending human authorization, and maintain audit trails of all human interventions. This control is critical for preventing algorithmic bias, erroneous automated actions, and ensuring accountability for consequential decisions.

Control objective

What auditing this proves

Demonstrate that high-risk automated decisions are subject to mandatory human review and approval before execution, with clear risk criteria, enforced workflow controls, and complete audit trails.

Associated risks

Risks this control addresses

Autonomous AI systems execute harmful decisions without human awareness, causing financial loss, reputational damage, or regulatory penalties
Algorithmic bias in automated decision-making leads to discriminatory outcomes in employment, credit, healthcare, or law enforcement contexts
Machine learning models operating on corrupted or poisoned training data make systematically flawed high-stakes decisions without human detection
Automated systems escalate privileges, provision access to sensitive systems, or approve financial transactions exceeding authorization thresholds without oversight
Runaway automation loops execute cascading destructive actions (data deletion, infrastructure changes, account terminations) before humans can intervene
Lack of human accountability creates liability exposure when automated decisions violate regulations such as GDPR, FCRA, or sector-specific compliance requirements
Insufficient audit trails prevent investigation and remediation when automated decisions cause incidents, impeding root cause analysis and regulatory reporting

Testing procedure

How an auditor verifies this control

Obtain and review the organization's formal policy defining high-risk decision categories, risk thresholds, and mandatory human oversight requirements.
Request inventory of all automated decision systems including AI/ML models, RPA workflows, and rule-based engines that could execute high-risk actions.
Select a representative sample of at least 5 automated systems spanning different risk categories for detailed testing.
Examine system architecture documentation and configuration settings to identify technical controls that enforce human-in-the-loop checkpoints for high-risk decisions.
Review access control matrices and role assignments to verify that only authorized personnel can approve high-risk automated recommendations.
Query audit logs and decision records for the past 90 days to identify instances where high-risk decisions were flagged for human review.
Select 10-15 high-risk decision records and trace each through workflow logs to confirm human approval occurred before execution, noting approver identity, timestamp, and decision rationale.
Attempt to simulate or review test cases where automated systems correctly escalated decisions to humans when risk thresholds were met, and verify fail-safe behavior if human approval is denied or times out.

Evidence required Collect the formal high-risk decision policy document with defined thresholds and approval workflows; configuration exports from automated decision systems showing human approval gates and escalation logic; access control lists designating authorized approvers; audit logs spanning 90 days containing decision IDs, risk classifications, human approver identities, approval timestamps, and final outcomes; workflow diagrams or system architecture documentation illustrating decision routing and fail-safe mechanisms; sample decision records with accompanying approval documentation and business justification.

Pass criteria All sampled high-risk automated decisions show documented human review and explicit approval before execution, with audit trails linking decisions to authorized approvers and no evidence of high-risk actions bypassing mandatory human oversight controls.

Where this control is tested

Audit programs including this control

AI Model Risk & Governance

Whether you train, fine-tune or just consume models, you need governance — inventory, risk classification, human oversight, evaluation.