Role overview

Senior Observability Engineer – Kubernetes, Prometheus, Grafana & Thanos

ABOUT THE ROLE

We are seeking an experienced Observability Engineer to join an enterprise Kubernetes platform team.

You’ll own the complete observability stack across multiple production Kubernetes clusters, providing metrics, logging, tracing, and alerting capabilities that ensure exceptional reliability and performance for mission-critical applications.

This role combines deep technical expertise in modern observability tools with emerging AI/ML capabilities to build intelligent monitoring solutions, predictive alerting, and self-healing infrastructure.

WHAT YOU'LL DO

Design, deploy, and maintain enterprise-scale observability infrastructure, including Prometheus, Grafana, Thanos, Loki, and modern collection agents.
Manage observability deployments using GitOps principles and infrastructure as code.
Implement long-term metrics storage solutions with cloud object storage.
Maintain and upgrade observability components across development, QA, UAT, production, and disaster recovery environments.
Configure distributed observability architecture spanning multiple data centers and cloud providers.

METRICS & MONITORING

Design and implement Prometheus monitoring strategies for Kubernetes infrastructure and containerized applications.
Create Service Monitors and Pod Monitors for automated metrics collection.
Develop rules for intelligent alerting with minimal false positives.
Configure multi-cluster metrics federation and aggregation.
Optimize metrics cardinality, storage efficiency, and query performance.