Role Overview:The Senior Open Telemetry (OTel) Instrumentation Engineer will be responsible for end-to-end instrumentation of SaaS products, ensuring robust observability across multi-tenant surfaces and preventing data leakage. This role involves managing OTel Collector fleets across diverse cloud environments (AWS, GCP, Azure), gathering network telemetry, and owning Application Performance Monitoring (APM) instrumentation. The engineer will also build and maintain internal instrumentation libraries, drive the adoption of OTel semantic conventions, and collaborate on telemetry schemas for AIOps and ML models.
Key Responsibilities:- Instrument SaaS products utilizing OTel SDKs (Python, Node.js, Java, Go) to ensure spans, metrics, and structured logs are emitted at appropriate granularity.
- Define and enforce attribute naming conventions aligned strictly with OTel semantic standards.
- Instrument multi-tenant surfaces to guarantee robust tenant-level observability while preventing cross-tenant data leakage.
- Deploy and maintain the OTel Collector fleet across AWS, GCP, and Azure, including receiver configurations, processor pipelines, and exporter routing.
- Instrument serverless (Lambda, Cloud Run) and container runtimes (EKS, GKE, AKS) utilizing auto-instrumentation where feasible.
- Collect and normalize cloud provider metrics (CloudWatch, Cloud Monitoring, Azure Monitor) via OTel receiver plugins.
- Gather network flow data (sFlow, NetFlow/IPFIX), SNMP traps, and BGP state via OTel-native and bridged receivers.
- Correlate network events with application traces using consistent trace-context propagation across network boundaries.
- Partner with NetOps to define the MELT (Metrics, Events, Logs, Traces) strategy for on-prem, SD-WAN, and cloud interconnect segments.
- Own Application Performance Monitoring (APM) instrumentation, including distributed tracing, custom span attributes, database query capture, and error fingerprinting.
- Instrument async workloads (Kafka, SQS, Celery) ensuring W3C Trace Context propagation across message boundaries.
- Define and instrument Service Level Indicators (SLIs) and Service Level Objectives (SLOs) queryable directly from the Lakehouse.
- Build and maintain internal instrumentation libraries and language-specific wrappers that encode organizational conventions.
- Author and review instrumentation RFCs while driving the adoption of OTel semantic conventions across all engineering teams.
- Collaborate with the AIOps ML team to ensure telemetry schemas meet feature engineering and ML model requirements.
- Operate the Collector pipeline at scale, managing backpressure handling, sampling strategies, and cardinality budgets.
Required Skills:- Strong expertise in OpenTelemetry (SDKs, APIs, Collector, OTLP).
- Proficiency in distributed tracing, metrics, and logging principles (opentelemetry.io).
- Experience with instrumentation across languages including Java, Python, Go, Node.js, and .NET.
Qualifications:- Senior-level experience in instrumentation engineering.
