High cardinality metrics are created when a metric can produce many distinct attribute or label combinations. Every combination can become a time series with memory, CPU, network, storage, indexing, and query cost. A single unbounded user_id, request path, email, object key, or raw error message can multiply series far faster than traffic grows and can degrade monitoring during the incident operators need to investigate.
Cardinality is not simply the number of labels. It is the number of observed combinations of their values for a metric stream and resource identity. Ten regions, fifty endpoints, five status classes, and twenty tenant tiers can produce up to 50,000 combinations before instance or deployment dimensions are counted. Governance should preserve dimensions that support aggregate decisions and route individual-detail questions to traces, logs, exemplars, or an analytical store.
Design metrics from the decisions they support
For each metric, write the operational question, expected aggregation, alert or dashboard, owner, and retention need. A request counter might need method, normalized route, and status class to show traffic and errors. It rarely needs exact URL, user, request identifier, or free-form exception. If a label will not be used in a bounded aggregation or routing decision, omit it from the metric.
Choose a stable semantic event and a unit. Use counters for occurrences, histograms for distributions, and gauges for current state where appropriate. Follow naming conventions so one metric represents one logical unit and labels preserve consistent meaning. Do not encode dimensions into a family of dynamically generated metric names; that merely hides cardinality from label review and makes discovery and queries harder.
| Candidate dimension | Metric treatment | Reason | Alternative for detail |
|---|---|---|---|
| HTTP method | Keep from a fixed set | Small stable set useful for operations | Trace for exact request |
| Normalized route template | Keep after routing normalization | Groups equivalent handlers | Log raw path with access controls |
| User or account ID | Drop | Unbounded identity and privacy exposure | Trace or event query by authorized ID |
| Status class | Keep as bounded class | Supports error and success aggregation | Log exact application code if needed |
| Raw error text | Drop or map to owned category | Unbounded and unstable content | Structured error event with trace correlation |
Calculate a metric-specific cardinality budget
Estimate the product of bounded values, but also inspect real combinations because dimensions may be correlated. Include resource attributes such as service, environment, cluster, region, instance, and deployment. Multiply by histogram bucket series where the backend representation requires it, replicas, retention, and number of emitting services. Set a budget based on platform capacity, query expectations, and business value rather than one universal label count.
Define expected and hard ranges for each label. region may have a current set and an approved expansion process. route should come from a router template, not a URL string. error_type should use a maintained taxonomy with an other path. Treat labels supplied from request headers or user input as untrusted. Review how an attacker or accidental loop could create new values.
Enforce label contracts before export
Create instrumentation libraries or wrappers that expose approved dimensions and normalize values close to the source. Use OpenTelemetry Views or equivalent SDK configuration to retain only required attributes, rename streams, choose aggregation, or drop a metric. Filtering before aggregation is more efficient and, in the OpenTelemetry Metrics SDK model, occurs before cardinality-limit enforcement when configured through views.
Add static checks for obvious dangerous keys and dynamic tests that feed many representative values, then count resulting streams. Review new metrics and label changes in code ownership. Maintain a registry with metric name, type, unit, dimensions, expected bounds, dashboard consumers, retention, and owner. Expire unused instruments; a metric that no dashboard, alert, SLO, or investigation uses should not consume indefinite shared capacity.
Enforce backend or collector limits as a final guardrail, not the primary design. Hard dropping may protect the platform while hiding affected tenants or errors. Define which series survive, how overflow is represented, and how owners are alerted. Never increase a limit automatically in response to overflow without understanding the growth source.
| Control point | Preventive control | Detection | Failure behavior |
|---|---|---|---|
| Instrumentation API | Approved attributes and normalized routes | Unit tests count generated sets | Reject or map unknown values |
| SDK View | Attribute allowlist and aggregation choice | Compare emitted streams with contract | Drop nonessential attributes |
| Collector | Transform and tenant limits | Volume and unique-series estimates | Isolate or throttle offending pipeline |
| Backend | Series, ingest, and query budgets | Active-series growth and expensive queries | Protect shared service with explicit rejection |
| Governance | Registry, owner, review and expiry | Orphan and unused metric reports | Disable after notice and validation |
Understand OpenTelemetry cardinality limits and overflow
The OpenTelemetry Metrics SDK specification defines a configurable hard limit on unique attribute combinations for a metric during a collection cycle and a default of 2,000 when no view or reader limit is supplied. It reserves an overflow attribute set marked otel.metric.overflow=true to aggregate measurements that cannot receive their own aggregator after the limit. Implementation and version support should still be verified in each language SDK.
Overflow protects application memory and preserves a coarse total, but it loses the individual attributes that would explain which values exceeded the budget. Alert on the overflow series and treat any unexpected appearance as a contract breach. A planned overflow strategy can be acceptable for a best-effort dimension, but do not rely on it for SLO or billing dimensions whose separation is required for correctness.
Move high-cardinality detail to the right signal
Use traces for per-request context, logs or structured events for individual identifiers and errors, and exemplars to connect an aggregate metric point to a representative trace. Apply access, retention, and sampling controls appropriate to those signals. Moving an identity out of a metric does not eliminate privacy obligations; it makes storage and query behavior explicit and avoids paying a permanent per-series cost for rare investigations.
If operators need top tenants or endpoints, compute a bounded ranking in an analytical pipeline or emit a controlled aggregate rather than a series for every identity. For business analytics, use an event warehouse with suitable governance. Metrics should answer fast repeated aggregate questions. Trying to turn a time-series database into an arbitrary entity index usually harms both cost and operational reliability.
Respond to a cardinality incident without destroying evidence
When active series or ingestion spikes, identify the metric, label, service, deployment, and first-seen time. Preserve a bounded sample of offending values in a secure diagnostic channel before dropping them. Apply the narrowest reversible control: disable one instrument, remove one label through a Collector transform, lower one tenant’s ingest, or roll back the release. Broadly dropping all metrics can remove the evidence needed to protect the platform.
Expect churn after the source stops. Backends may retain inactive series until staleness and retention behavior take effect, and query indexes may remain expensive. Verify recovery in ingestion rate, active series, memory, query latency, and billable volume. Record why pre-production controls missed the change and add a test or contract rule. Do not expose raw offending identities in incident reports.
Operate label governance as a product capability
Publish a short label policy, approved semantic conventions, code examples, and a review path for justified exceptions. Provide teams with a local cardinality estimator and cost preview. Report budget use by metric owner, not as a public developer score. Platform teams should make the safe path easy and preserve a fast emergency control when a release threatens shared monitoring.
Review high-growth metrics after service fan-out, tenant growth, new regions, and semantic-convention changes. Track active series, series churn, overflow, unused metrics, ingestion bytes, query latency, and spend together. Lower cardinality that makes dashboards unusable is not success; the goal is sufficient operational resolution within predictable capacity and cost.
Key takeaways
- Give every metric a decision, bounded dimension contract, owner, and expiry path.
- Budget the combinations of labels and resource attributes, not the label count alone.
- Normalize and filter attributes before aggregation, with limits as a final guardrail.
- Monitor OpenTelemetry overflow and never treat it as lossless behavior.
- Use traces, logs, exemplars, and analytical stores for individual-detail questions.
Frequently asked questions
How high is high cardinality?
There is no backend-independent threshold. Evaluate combinations per metric, total active series, churn, query patterns, retention, and platform capacity. An unbounded label is a design risk even when today’s series count is small.
Can tenant ID ever be a metric label?
Only with a deliberately bounded tenant set, strong operational need, privacy review, and capacity budget. For large or growing populations, prefer tenant tiers, controlled top-N analytics, logs, or traces. Never accept arbitrary tenant values without limits.
Is dropping the bad label enough during an incident?
It stops new combinations if applied on every path, but backend series and indexes may persist. Verify ingestion and active-series recovery, preserve bounded diagnostic evidence, and repair the instrumentation contract before re-enabling the metric.
Conclusion
Cardinality is an information-design budget. Keep bounded dimensions that support repeated operational decisions, move identity-level detail to suitable signals, and enforce the contract before data reaches shared storage. With overflow detection, incident controls, and ownership, teams can expand observability without turning every new label into unpredictable infrastructure growth.