Skip to main content
← All controls
CP-9 / SI-13 / SC-36 NIST SP 800-53 Rev 5

Upstream provider failover

Demonstrate that the organization has implemented, tested, and documented failover mechanisms for critical upstream provider dependencies that activate automatically or through predefined procedures to maintain service availability.

Description

What this control does

Upstream provider failover ensures that critical dependencies on third-party services, APIs, data feeds, or infrastructure providers automatically switch to alternative providers or redundant instances when the primary upstream source becomes unavailable. This control includes both automated failover mechanisms (load balancers, DNS failover, API gateway routing) and manual procedures with defined recovery time objectives. It protects organizational operations from single points of failure in the supply chain and ensures continuity when external dependencies experience outages, performance degradation, or compromise.

Control objective

What auditing this proves

Demonstrate that the organization has implemented, tested, and documented failover mechanisms for critical upstream provider dependencies that activate automatically or through predefined procedures to maintain service availability.

Associated risks

Risks this control addresses

  • Prolonged service outage when a critical upstream provider experiences downtime without redundant alternatives configured
  • Data loss or transaction failures during provider outages due to lack of failover or queuing mechanisms
  • Cascading failure across multiple internal systems dependent on a single external provider without circuit breaker patterns
  • Inability to detect upstream provider degradation or failure in time to trigger manual failover procedures
  • Untested failover configurations that fail during actual provider outages, extending recovery time
  • Provider lock-in preventing rapid migration to alternative suppliers during extended outages or compromise
  • Regulatory non-compliance or SLA breaches when upstream failures prevent critical business functions from operating

Live threat patterns this control mitigates:

HIGH DDoS Campaign Against Public Services Volumetric or application-layer attack aimed at taking a service offline. Demands edge mitigation (CDN / scrubbing), rate limiting,…

Testing procedure

How an auditor verifies this control

  1. Obtain and review the inventory of critical upstream providers including SaaS vendors, cloud infrastructure providers, payment processors, identity providers, and third-party APIs.
  2. Retrieve architecture diagrams and configuration documentation showing failover mechanisms for each critical upstream dependency including load balancer rules, DNS failover settings, and API gateway routing logic.
  3. Review incident response and business continuity procedures documenting manual failover steps, decision trees, and designated personnel authorized to trigger failover.
  4. Examine monitoring and alerting configurations to verify automated detection of upstream provider failures including health checks, synthetic transactions, and provider status page integrations.
  5. Select a representative sample of critical upstream dependencies and review recent failover test reports or disaster recovery exercise documentation showing actual execution of failover procedures.
  6. Inspect change management records for failover configuration updates to verify that changes are reviewed, approved, and validated through testing before production deployment.
  7. Request evidence of recent failover events (planned or unplanned) including incident tickets, runbooks executed, time-to-failover metrics, and post-incident reviews.
  8. Validate that recovery time objectives (RTOs) and recovery point objectives (RPOs) are defined for each critical upstream dependency and that failover mechanisms are designed to meet these targets.
Evidence required Configuration exports from load balancers, DNS providers, and API gateways showing active failover rules and health check parameters. Failover test reports and disaster recovery exercise documentation from the past 12 months including timestamps, test scenarios, observed failover times, and identified issues. Monitoring dashboards or log excerpts showing upstream health checks and alert triggers. Architecture diagrams annotating primary and secondary providers with failover logic. Incident records documenting actual provider outages and failover activation including communication logs and restoration timelines.
Pass criteria All critical upstream provider dependencies have documented and technically implemented failover mechanisms that have been successfully tested within the past 12 months, with monitoring in place to detect failures and trigger automated or manual failover procedures meeting defined RTOs.