The OpenTelemetry Collector agent vs gateway decision is not mainly about installation preference. It decides where telemetry crosses a failure boundary, where credentials and policy live, whether processors can see complete data, and how much data can be buffered when a backend slows. An agent runs beside an application or on the same host; a gateway exposes one or more central OTLP endpoints. Many production systems deliberately use both.
Choose per signal and processor rather than declaring one company-wide shape. Host logs and host metrics naturally begin near the machine. Tail sampling needs all spans for a trace at one decision point. Central egress credentials belong in a controlled boundary. A topology that works for stateless batching may corrupt stateful aggregation or split a trace, so the component graph and routing keys matter as much as the number of Collector instances.
Start with signal origin and collection responsibility
Inventory traces, application metrics, host metrics, container logs, file logs, events, and externally scraped endpoints. Record how each source is discovered, whether it pushes or is scraped, which metadata is available locally, and what happens when the application, node, or network fails. An agent can collect node-local files and host data that a remote gateway cannot reach safely. An SDK can also export directly to a gateway when no local collection is required.
Define ownership. Application teams may own instrumentation and resource attributes while a platform team owns Collector distributions, policy, routing, and backend contracts. Avoid having both an agent and gateway scrape the same target or transform the same attribute without a reason. Duplicate collection increases cost and can violate metric identity assumptions. Document the intended single path for each source.
| Requirement | Agent fit | Gateway fit | Tiered fit |
|---|---|---|---|
| Read host files or host metrics | Strong: local access and metadata | Weak: remote access is awkward | Agent collects; gateway routes |
| Central backend credentials | Credentials copied widely | Strong: one controlled egress tier | Agents use internal OTLP; gateways hold secrets |
| Tail sampling | Poor across distributed traces | Possible with trace-affine routing | Agents route by trace ID to sampler tier |
| Network isolation | Buffers near source | Creates approved egress endpoint | Strong separation with an extra hop |
| Small simple deployment | Straightforward local instance | Straightforward central endpoint | Usually unnecessary complexity |
Use agents for local context and source isolation
An agent can run as a host service, Kubernetes DaemonSet, or application sidecar. It shortens the first network hop, enriches signals with local metadata, collects host-bound sources, and can apply memory limits or a small sending queue before telemetry leaves the node. Failure is contained: one agent usually affects one host or workload rather than every tenant using a central endpoint.
The tradeoff is fleet management. Configuration, upgrades, resource limits, and credentials may be replicated across many instances. A sidecar consumes resources per workload and can couple Collector restarts with application lifecycle. A DaemonSet shares one agent among pods on a node and needs appropriate tenancy controls. Agents are not automatically reliable buffers; in-memory queues vanish on restart, local disks can fill, and node loss removes both application and buffered telemetry.
Use gateways for policy, egress, and cross-source processing
A gateway provides a stable OTLP endpoint per cluster, data center, region, or security domain. It can centralize backend credentials, filtering, redaction, routing, batching, and vendor fan-out. Platform teams can change destinations without updating every SDK. Horizontal replicas can scale stateless pipelines behind a load balancer, and gateway capacity can be managed independently from application nodes.
Centralization also concentrates risk. A bad configuration, saturated queue, credential failure, or undersized gateway can affect many services. The extra hop adds latency and resource cost. Define failure domains by region and tenant rather than routing all telemetry through one global cluster. Protect gateways from untrusted or accidental high-volume producers with authentication, limits, bounded queues, and workload isolation.
A gateway is not one machine. It is a role in the topology. Deploy replicas, load balance appropriately, and isolate pipelines when signals or tenants have materially different risk. Logs may need larger buffers, traces may need trace-affine routing, and metric scrapers need stable ownership. Combining every signal in one process can make a log spike evict the traces needed to investigate it.
Place processors according to their state and data scope
Stateless transformations can run on agents or ordinary gateways as long as ordering and duplication are controlled. Stateful processors need all relevant data at the same instance. Tail sampling must see the spans for a trace. Cumulative-to-delta conversion needs each metric stream to have one consistent processor. Span-to-metrics aggregation may need service-affine routing. Ordinary round-robin balancing can break these assumptions even while every Collector is healthy.
Use the load-balancing exporter with a suitable routing key when a downstream tier performs stateful work. Trace-ID routing keeps spans of a trace together for tail sampling; service routing can support service-level aggregation. Scaling or changing the backend set can remap keys, so test incomplete state during rollout. Prefer a simpler single well-resourced stateful tier until traffic justifies the operational burden of sharded state.
| Processor or activity | Preferred placement | Routing requirement | Failure concern |
|---|---|---|---|
| Resource detection near host | Agent | Local source identity | Missing or wrong metadata after relocation |
| Memory limiting and initial batching | Every tier | None beyond pipeline isolation | Backpressure and dropped data |
| PII redaction before network exit | Agent or trust-boundary gateway | All egress must pass control | Bypass path leaks attributes |
| Tail sampling | Dedicated gateway tier | All spans by trace ID | Split traces and state loss on remap |
| Cumulative-to-delta conversion | Stable agent or routed gateway | One writer per metric stream | Resets, duplicate writers, inaccurate deltas |
Preserve metric identity and the single-writer principle
Metric streams require globally unique resource identity and a single writer. Multiple gateways exporting the same stream can create out-of-order points, gaps, or conflicting values. Scrape each target through one logical collector path, add stable resource attributes, and ensure failover does not produce concurrent writers. Stateful metric processors must receive the same stream consistently across collection cycles.
During migration, duplicate export may be intentional for backend comparison, but separate destinations and account for the doubled load. Do not merge two copies into one backend as though they were independent samples. Validate counters, temporality, timestamps, and resource identity through rolling Collector changes. A topology is incorrect if it preserves availability by silently damaging metric meaning.
Design buffering around real network outages
Map failure of application-to-agent, agent-to-gateway, and gateway-to-backend links. Configure bounded sending queues and retries for transient failures. Size them from incoming bytes or batches, expected outage duration, available memory or disk, and acceptable loss. Persistent queues can survive Collector restart but are not limitless archives; local storage needs capacity alerts, encryption and access controls where required, and a deliberate overflow policy.
Decide where backpressure should surface. An overloaded agent may refuse SDK exports; a gateway may fill while agents continue buffering; an unavailable backend may cause several tiers to retain copies. Observe accepted, refused, queued, failed, and sent data at each hop. Prevent synchronized retry storms and ensure applications do not block critical work indefinitely to preserve telemetry.
Choose the simplest topology that satisfies the contract
Use agent-only when sources are local, processing is simple, and each agent can reach the backend safely. Use gateway-only when applications push OTLP, centralized policy and credentials matter, and host-bound collection is unnecessary. Use agent-to-gateway when local collection and buffering must combine with controlled egress, multi-backend routing, or cross-source processing. Add a dedicated stateful tier only for a processor that requires it.
Pilot with representative volume and failure tests. Measure application overhead, hop latency, Collector CPU and memory, queue utilization, refusal and export failure, backend throttle, and data completeness. Test rolling upgrades, backend outage, loss of a gateway replica, node termination, configuration rollback, and a noisy tenant. Document capacity ownership and an emergency way to bypass a broken optional processor without bypassing security controls.
Key takeaways
- Choose topology per signal, source, trust boundary, and processor state.
- Use agents for node-local collection and gateways for centralized policy and egress.
- Route complete state to one processor instance for tail sampling and metric conversion.
- Preserve globally unique metric identity and one logical writer.
- Capacity-test every hop, queue, rollout, and noisy-tenant failure.
Frequently asked questions
Is a gateway required for OpenTelemetry?
No. SDKs or agents can export directly to an OTLP-capable backend. Add a gateway when centralized credentials, policy, routing, tenancy, buffering, or processors justify another production service and network hop.
Should Kubernetes use a sidecar or DaemonSet agent?
A sidecar offers workload isolation and direct lifecycle coupling but costs resources per pod. A DaemonSet is more efficient for host and container sources but shares capacity and trust across workloads. Choose from source access, isolation, and operating cost.
Can one gateway handle logs, metrics, and traces?
It can, but isolate pipelines and set signal-specific limits. Separate deployments when volume, stateful processing, credentials, tenant risk, or failure impact differ enough that one signal could starve the others.
Conclusion
Collector topology should preserve signal meaning and contain failure, not merely move configuration. Put local work near the source, central policy at a deliberate trust boundary, and stateful processing where routing can provide complete data. The right answer is often a modest tiered design whose queues, identities, and failure behavior the platform team can actually operate.