Getting telemetry under control in a large organisation
Large organisations do not emit telemetry from one clean place. It comes from many Kubernetes clusters, dozens of AWS services, vendor portals like Sentry, and even client devices reaching in over the public internet. Left ungoverned, that produces three chronic problems.
First, runaway cost: metric volume and especially high cardinality drive observability bills up fast, and a single integration can trend toward six-figure annual cost just to extract the data. Second, inconsistency: the same logical metric appears under different names across teams and frameworks, which breaks comparison and reliable alerting. Third, unclear ownership: something breaks and nobody knows who owns the affected service.
We fix all three at the pipeline layer, with one central point of control instead of coordination across dozens of slow-moving service teams.
Pipeline-level control with Vector and OpenTelemetry
We build a centralised telemetry platform around a Vector pipeline that handles collection, normalisation, sampling, content-based filtering and routing for metrics, logs and traces before anything reaches the backend.
Inconsistent naming is a recurring problem: dozens of applications share the same framework, yet the framework names the same metric differently depending on the emitting function. We fix it in the pipeline by extracting the meaningful part of the metric name, promoting it to a proper tag, and forwarding a clean, consistent metric downstream. Legacy custom formats are converted to the target platform format in the same layer, and OpenTelemetry metrics and traces are first-class citizens of the pipeline.
Observability cost control and cardinality protection
Cost control starts with an inventory: every metric the organisation collects, listed in one central place, with an owner. From there, governance becomes enforceable.
Dangerous high-cardinality tags are banned by default in all their variants and blocked at ingestion, before they ever reach the backend. The stakes are concrete: we have seen a single integration trending toward six-figure annual cost, and individual bad tags capable of generating tens of thousands of dollars of cost in a single week. Catching these in the pipeline means the problem never appears on the invoice.
Log routing that reacts in hours, not release cycles
Service teams typically ship on weekly or fortnightly release cycles. A noisy log line released on Monday would burn money for weeks before a fix ships, if the fix had to come from the service itself.
Our pipeline handles it instead: content-based filtering drops logs matching a specific string while everything else flows through, and sampling is applied selectively, including deliberately unsampled routes for services that need every log. A rule change in the pipeline takes effect in hours, independent of anyone's release calendar.
Service ownership metadata and cost attribution
In a large organisation, the most frequent incident-response question is not "what broke" but "who owns it". We maintain the mapping of teams to components and feed it into the observability platform, so on-call responders see the owner next to the alert instead of chasing it across wikis.
The same metadata powers cost attribution. There is one shared observability bill, but tagged resources and metrics make it possible to trace a spend spike back to the team that caused it, and to go talk to them with numbers.
Reference experience at global scale
Inside a large global gaming company, we led a New Relic to Datadog migration at scale, reworking how metrics are scraped, configured and ingested, and built the Vector-based central pipeline that now runs across many Kubernetes clusters, multiple AWS services and client devices over the public internet.
In an organisation-wide Datadog migration, the unified setup proved three times more cost-effective than the previous solution, cutting roughly $100K per month on log optimisation alone, and produced standardised telemetry guidelines that all teams adopted.
Our observability work extends beyond web backends. We performance-tested an XMPP cluster for hundreds of thousands of concurrent users with a dedicated Grafana dashboard covering XMPP, ErlangVM and RDBMS subsystems, and standardised Datadog monitoring for gaming login services handling over 15 million daily active users. For teams whose telemetry feeds analytics rather than alerting, see our data engineering practice.
How a telemetry engineering engagement runs
We start with an audit of the existing landscape: sources, volumes, formats, costs and pain points. Then we design a metrics, logs and traces architecture that ingests from heterogeneous sources, not just one clean environment.
Next comes governance: what may be collected, cardinality limits, default guards, sampling and filtering rules, and a taxonomy that keeps metrics comparable across teams. We build the pipelines, integrations, central metric inventory and ownership metadata layer, then optimise cost continuously, including working directly with vendor support on large-scale issues. Finally we operate, support and maintain the platform, with ongoing enablement for internal teams.
This model fits organisations where CTOs and platform leads need observability spend to be predictable, and it is battle-tested in gaming-scale environments where telemetry volume peaks with player traffic.
