Backend Engineer II

About the Company

Grafana Labs is the company behind Grafana Cloud, a fully managed observability platform trusted by more than 10,000 organizations to improve reliability, resolve incidents faster, and manage telemetry at scale. Built on open-source technologies and open standards, Grafana Cloud helps teams monitor and understand data across different systems and environments. Its customers include Anthropic, Bloomberg, NVIDIA, Microsoft, and Salesforce.

Grafana Labs is a 100% remote company with team members across more than 40 countries. The company values open-source collaboration, transparency, autonomy, and trust, supported by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, J.P. Morgan, CapitalG, and Lead Edge Capital.

About the Role

Grafana Labs is seeking a Senior Engineer to join the Tempo team, which develops Tempo, the open-source distributed tracing backend behind Grafana Cloud Traces and Grafana Enterprise Traces. Tempo enables users to search traces, generate metrics from spans, and connect tracing data with logs, metrics, and profiles across the Grafana ecosystem.

The role focuses on advancing Tempo's performance, scalability, reliability, and capabilities as a platform for next-generation observability products and AI-driven workflows. The successful candidate will take ownership of complex technical initiatives, shape architectural decisions, improve production operations, and collaborate with product and engineering teams to deliver reliable solutions for customers and downstream applications.

Key Responsibilities

• Lead significant, multi-month technical projects from problem definition and design through implementation and rollout, including trace aggregation APIs, autoscaling, customer limits, and query engine improvements.
• Own major components of Tempo across ingestion, storage, query execution, and metrics generation.
• Write and present design documents, participate in design reviews, make informed trade-offs involving performance, cost, and complexity, and document technical decisions.
• Develop structured, deterministic, and discoverable APIs that support Grafana products, AI-driven assistants, and external integrations.
• Improve operational excellence by owning outcomes against service-level objectives, including P99 write latency, incident recurrence, and total cost of ownership per ingested gigabyte.
• Reduce operational overhead through automation, parameterized rollouts, actionable alerts, and other improvements that support a Zero Ops approach.
• Collaborate with product managers and teams working on App Observability, Asserts, Traces Drilldown, and Grafana Assistant to understand requirements and remove technical barriers.
• Mentor and support engineers through code reviews, constructive design feedback, and pairing, particularly with junior and mid-level team members.
• Participate in on-call rotations for the services developed and maintained by the team, contribute to incident response, and support post-incident learning.
• Contribute to Tempo's open-source community by reviewing external contributions and participating in development discussions.
• Use AI coding assistants within company security guidelines to support prototyping, test generation, refactoring, documentation, and incident follow-ups while maintaining strong code quality and review standards.

Technical Focus Areas

• Develop trace aggregation capabilities, extend TraceQL metrics, and create APIs that support AI assistants and downstream products.
• Improve end-to-end autoscaling for customer limits and Tempo cells, including predictive scaling, handling traffic spikes, and safe scale-down.
• Build safeguards for bursty, high-cardinality workloads generated by AI agents.
• Improve query performance through new data formats, optimized query pipelines, and targeted enhancements for common Drilldown and Traces workflows.
• Support efficient queries across 30-day ranges and high ingestion volumes.
• Improve deployment and multi-cell operations through parameterized rollouts, streamlined deployments, and tooling that supports growth into triple-digit cell counts.
• Develop customer-facing configuration and observability capabilities to reduce support escalations.

Required Qualifications

• Demonstrated experience delivering complex, multi-month technical projects spanning design, implementation, and production operations.
• Significant hands-on experience building and operating distributed data systems in production, including ingestion pipelines, storage engines, query execution, or similar systems.
• Strong software engineering skills, with the ability to write clean, reliable, maintainable, and performant code.
• Strong experience with Go or relevant systems programming experience in languages such as Rust, C, or C++.
• Experience owning production services, participating in on-call rotations, reducing operational overhead, and managing service-level objectives.
• Ability to break complex problems into manageable steps, deliver initial solutions, gather feedback, and iterate.
• Strong collaboration and written communication skills, including experience with design documents, code reviews, and asynchronous teamwork.
• Ability to work independently, make sound technical decisions, and contribute to engineering direction in a remote environment.

Preferred Qualifications

• Experience with distributed tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages, SQL or TraceQL-like engines, or APIs intended for programmatic consumption by services or AI agents.
• Experience with columnar storage formats such as Parquet or specialized on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building systems for AI and large language model consumers, including structured APIs, metadata and discovery endpoints, deterministic outputs, and evaluation frameworks.
• Open-source contributions or maintainership experience and comfort collaborating with developer communities.
• Previous experience using Grafana, Prometheus, Loki, or Tempo in production or homelab environments.
• Experience working with fully remote, globally distributed teams.

Work Environment and Collaboration

• Remote position open to candidates in the United States and Canada.
• Remote-first working model with regular video meetings and primarily asynchronous, written communication.
• Collaborative environment emphasizing transparency, creativity, diverse perspectives, autonomy, and clear communication.
• In-person onboarding to help new team members understand the company, its products, and its working practices.
• Access to modern AI coding assistants and a company-funded usage budget, with access to frontier models from OpenAI, Anthropic, and Google, subject to security guidelines.

Compensation and Benefits

• U.S. on-target earnings and base compensation range: $154,445–$185,334, as stated in the job posting.
• Compensation may vary based on role level, experience, and skills assessed during the interview process.
• Restricted Stock Units (RSUs) are included with all roles, giving team members an ownership stake in Grafana Labs.
• Global annual leave policy of 30 days per year, including three days reserved for Grafana Shutdown Days, subject to applicable local legislation.
• Career development opportunities within a growing organization.
• Transparent communication, accessible leadership, and a collaborative, high-trust culture.

Compensation ranges are specific to each country. Candidates applying from other locations will receive details about the applicable local compensation range and benefits from their recruiter at the beginning of the process.

Equal Employment Opportunity

Grafana Labs is an equal opportunity employer. The company welcomes applicants regardless of race, colour, nationality, origin, caste, sex, gender reassignment, gender identity or expression, sexual orientation, age, religion or belief, disability, veteran status, genetic information, pregnancy, maternity, marital status, family or carer status, or any other characteristic protected by local law. Grafana Labs believes equality and diversity are essential to building a strong organization.

Recruitment and Data Privacy

Grafana Labs may use AI tools during recruitment to help match information provided in CVs with job requirements. The recruitment team will continue to review applications manually to assess alignment with current openings.

Candidates can refer to Grafana Labs' recruitment privacy information for details about how personal data is used after applying.