Third-party API reliability is part architecture review, part product policy, and part supplier governance. A payment, identity, communications, mapping, tax, or AI provider sits outside the team’s deployment and incident controls, but its behavior still shapes the product promise. A logo on an architecture diagram and a contractual availability figure do not show what happens when the API is slow, rate-limited, inconsistent, or unreachable from one region.
A useful review follows one business transaction across the boundary. It identifies what must be true, how long the call may consume, which failures are safe to retry, what a fallback may claim, how operators distinguish provider failure from client defects, and how the company would migrate if the dependency no longer met its needs. The output is a decision record and funded controls, not a generic vendor score.
Define the business and technical dependency contract
Name each operation, user journey, data class, and required outcome. Mark whether the call is synchronous, asynchronous, optional, or irreversible. Capture regions, endpoints, authentication, quotas, payload limits, consistency, idempotency support, version policy, and retention. Different operations from one provider can have very different risk: creating a payment has stronger duplicate and reconciliation concerns than fetching a currency list.
Separate the provider’s SLA from your consumer-side SLO. Read the SLA’s exclusions, measurement point, maintenance treatment, claim process, and remedy. Then measure success and latency from your workload. Include DNS, TLS, token acquisition, network reachability, SDK behavior, and your configuration because users experience the entire call. Reconcile large differences with provider status and support records.
| Review area | Question | Evidence | Decision |
|---|---|---|---|
| Business criticality | Which user outcome stops or becomes unsafe? | Flow map and impact classification | Hard, soft, or removable dependency |
| Interface | Which operation, region, version, and quota apply? | Contract, API docs, and tested limits | Supported integration profile |
| Failure semantics | Can timeout hide a completed side effect? | Idempotency and reconciliation tests | Retry, query, or manual recovery |
| Operations | Can teams detect and escalate the fault? | Consumer telemetry and support exercise | Runbook and ownership |
| Exit | How are data, behavior, and history moved? | Adapter boundaries and migration rehearsal | Trigger and estimated migration effort |
Allocate an end-to-end timeout budget
Start with the user or workflow deadline and subtract time for every stage: ingress, internal processing, provider connection, response handling, and a fallback or retry if allowed. Use observed latency distributions by operation and region. A timeout should be long enough for normal tail latency but short enough to leave the caller useful choices. Arbitrary long defaults hold threads, connections, and memory while the result is no longer useful.
Cover connection establishment, DNS, TLS, reads, and total request duration. Some client settings apply only to socket reads and leave other phases unbounded. Warm-up behavior can differ after deployment or certificate rotation, so establish connections before receiving full traffic where appropriate. Propagate the remaining deadline through internal services and cancel downstream work when the caller gives up.
Design retries around operation semantics
Classify errors as permanent, transient, overloaded, ambiguous, and authentication or quota faults. Do not retry validation errors or permission failures. Retry transient errors only while deadline remains, with exponential backoff and jitter. Cap attempts and enforce a process- or service-level retry budget so a provider incident cannot turn one request into uncontrolled extra traffic. Where several layers could retry, select one owner.
For writes, an apparent timeout may occur after the provider committed the operation. Use provider-supported idempotency keys with a scope and retention window that match the business action. If no safe idempotency mechanism exists, query status or reconcile before repeating. Preserve the provider request identifier and your correlation identifier. Never treat a transport error as proof that no side effect happened.
Use a circuit breaker to stop calls that are unlikely to succeed and to protect both systems, but tune it per operation and failure mode. A single global breaker can disable healthy read operations because one write endpoint fails. Define open, probe, and recovery behavior; limit probe traffic; and expose state changes. A breaker is not a fallback by itself: callers still need an honest result when the circuit is open.
| Observed outcome | Retry? | Required control | Caller response |
|---|---|---|---|
| Invalid request or denied permission | No | Validate locally and alert on unexpected change | Return actionable permanent failure |
| Explicit overload with retry guidance | Maybe | Deadline, jitter, attempt and retry budgets | Delay or fail without synchronized retries |
| Connection failure before send | Usually for safe operations | Bounded attempts and alternate endpoint policy | Retry only within journey budget |
| Timeout after a write may have committed | Not blindly | Idempotency key or status reconciliation | Report pending or unknown state |
| Sustained provider errors | No per-request storm | Circuit breaker and degraded contract | Fail fast, queue, or use approved fallback |
Review quotas as shared production capacity
Document limits by account, tenant, region, operation, token, and time window. Test response headers and error behavior before production. Forecast normal and burst consumption, including retries, batch jobs, replay, and disaster failover. Reserve capacity for critical flows and keep nonurgent workloads from consuming the entire allowance. A second region may share the same global quota, so regional failover can increase demand without increasing supply.
Monitor remaining headroom where the provider exposes it, plus your own request rate and rejection counts. Set escalation thresholds well before exhaustion because quota increases may require support lead time. Protect credentials and token refresh paths; an authentication endpoint or secret store can become a hidden prerequisite for every provider call. Exercise key rotation without forcing simultaneous reconnects across the fleet.
Specify fallback, queuing, and reconciliation semantics
A fallback must preserve defined invariants. Cached tax rules need freshness and jurisdiction limits. A second email provider requires suppression lists, consent, and duplicate controls. Queued payments require expiry, status, and reconciliation; they must not appear completed at acceptance. Decide whether to fail open, fail closed, return stale data, queue, switch provider, or remove the feature for each operation.
Test fallback capacity and common dependencies. Two providers do not create independence if both use the same network egress, identity broker, DNS resolver, adapter defect, or source data. During recovery, limit queued replay and compare provider records with your ledger. Define who resolves ambiguous outcomes and how customers are notified. Data correctness often takes longer to restore than endpoint availability.
Build independent observability and escalation
Measure calls, latency, timeouts, retries, breaker state, quota rejection, result class, and fallback activation by operation and region. Keep label values bounded and protect sensitive payloads. Run a synthetic transaction from relevant network paths, but ensure it cannot create real side effects. Compare provider status with consumer evidence without suppressing your alert merely because the status page is green.
Create an escalation matrix with support channel, severity definitions, response targets, evidence package, and executive contact. Rehearse opening a ticket and retrieving request IDs before a major event. Incorporate incident notification, root-cause reporting, change notice, deprecation, and quota escalation into commercial review. Product communications should describe user impact and workaround rather than blaming a supplier.
Maintain a credible exit path
Put provider-specific objects behind a narrow adapter, but do not promise effortless portability. Authentication models, payment states, model outputs, webhooks, and historical identifiers differ. Inventory proprietary fields, stored data, event history, secrets, compliance evidence, and customer configuration. Estimate migration by operation and define triggers such as repeated SLO failure, unacceptable roadmap change, price risk, legal constraint, or deprecation.
Test exports, dual-running, reconciliation, and rollback on a limited cohort. A dormant alternate integration can decay as APIs change, so either exercise it or treat it as unproven. Sometimes the rational decision is to accept concentration risk because dual-provider complexity would create more defects; record that choice and its review date.
Key takeaways
- Review each external operation in the context of a named business outcome.
- Budget timeouts from the end-to-end deadline and cancel work that has become useless.
- Retry only safe transient failures with jitter, bounded attempts, and a shared retry budget.
- Treat quotas, fallback capacity, reconciliation, and recovery replay as first-class design concerns.
- Keep consumer-side evidence, escalation procedures, and an exercised exit plan.
Frequently asked questions
What SLA percentage is sufficient for a critical API?
No percentage is sufficient in isolation. Match the provider commitment and observed behavior to the journey target, failure correlation, recovery time, and fallback. The product may require independent controls even when the advertised percentage is high.
Does every critical API need two providers?
No. Dual providers add cost, semantic mismatch, data governance, and reconciliation risk. Use them when reduced correlated risk justifies that burden and traffic can be switched safely. Otherwise invest in degradation, queueing, or a clearer stop condition.
Are official SDK retry defaults safe?
Treat defaults as a starting point. Confirm which errors are retried, total elapsed time, jitter, idempotency assumptions, and interaction with retries in your services. Configure them against the journey deadline and test under sustained provider failure.
Conclusion
An external API becomes reliable enough only in the context of the product using it. Bound its time and traffic, make write ambiguity recoverable, reserve quota, define truthful fallback, and maintain evidence beyond the provider’s dashboard. A review that includes operations and exit cost lets leadership choose the dependency with eyes open rather than discover its real contract during an outage.