Distributed Telemetry Engineer – Backend/Spark

Job Title: Sr SW Engineer – Infrastructure Telemetry & Capacity Planning Platform

Location: San Jose, CA (Hybrid)

Duration: Contract to hire

Job Description:

We are seeking an experienced Technical Lead / Senior Software Engineer to contribute to the design and development of an Infrastructure Telemetry and Capacity Planning platform. This role combines backend engineering, large-scale data processing, distributed query technologies, and frontend development to transform infrastructure telemetry into reliable, queryable datasets and actionable capacity-planning insights.

The platform processes data related to server and GPU utilization, memory, thermal conditions, power consumption, and workload metrics across a large fleet. The ideal candidate brings strong hands-on engineering skills, experience building data-intensive applications, and the ability to translate ambiguous business and infrastructure challenges into scalable architectures and production-ready software.

Key Responsibilities:

Data Processing & Query Architecture:

• Design and develop scalable data ingestion, transformation, processing, and persistence pipelines for large telemetry datasets.
• Build and optimize SQL queries using distributed query engines such as Trino/Presto, Spark SQL, Amazon Athena, or similar technologies.
• Work with columnar data formats such as Apache Parquet, designing schemas, partitioning strategies, and storage layouts for efficient querying at scale.
• Evaluate streaming versus batch-processing patterns based on data volume, cardinality, performance, and latency requirements.
• Develop Spark-based processing jobs using AWS EMR or equivalent distributed data-processing platforms.

Backend Development:

• Build production-grade backend services and REST APIs using Java and/or Python.
• Implement reliable ingestion and processing workflows using technologies such as Kafka, Flink, and Spark.
• Design relational database schemas and data models using PostgreSQL, Amazon RDS, or similar databases.
• Develop scalable, maintainable services with appropriate error handling, testing, logging, and monitoring.
• Contribute to architecture discussions, code reviews, and engineering best practices.

Frontend & Data Visualization:

• Develop interactive data-intensive applications using React and TypeScript.
• Build dashboards and visualization layers for telemetry metrics, resource utilization, operational trends, and capacity planning.
• Implement data visualizations using libraries such as ECharts, D3.js, or Highcharts.
• Develop editable data grids and tabular interfaces using tools such as AG Grid.
• Collaborate across backend and frontend components to deliver responsive, efficient experiences for querying and analyzing large datasets.

Infrastructure Telemetry & Capacity Planning

• Design and implement solutions for collecting and processing server and GPU telemetry, including CPU utilization, memory, thermal, power, and workload metrics.
• Support end-to-end telemetry workflows covering ingestion, processing, storage, querying, monitoring, and visualization.
• Apply observability practices using tools such as Prometheus and Grafana.
• Translate telemetry data into actionable insights supporting utilization analysis, resource trending, and capacity-planning decisions.

Cloud-Native Engineering & Delivery

• Build highly available and extensible applications using cloud-native architecture, preferably on AWS.
• Work with containerized environments, including Docker and Kubernetes, and CI/CD pipelines.
• Use AI-assisted development tools such as Cursor, Windsurf, or equivalent to accelerate delivery while maintaining code quality.
• Operate independently in an ambiguous environment, converting business requirements into algorithms, technical designs, and working software.

Required Qualifications

• 10+ years of software development experience, including hands-on delivery of complex backend, data-intensive, or distributed applications.
• Strong SQL skills, including query optimization and working with large datasets.
• Hands-on experience with distributed query engines such as Trino/Presto, Spark SQL, Amazon Athena, or similar technologies.
• Experience with columnar data formats such as Parquet and large-scale data storage and querying.
• Strong proficiency in Java and/or Python, including backend development and REST API design.
• Experience with Apache Spark and distributed data processing; AWS EMR or equivalent platforms preferred.
• Experience with streaming data ingestion and processing technologies such as Kafka or Flink.
• Frontend development experience with React and TypeScript.
• Experience building data-heavy UIs with charting libraries such as ECharts, D3.js, or Highcharts, and editable data grids such as AG Grid.
• Relational database design experience with PostgreSQL, Amazon RDS, or similar technologies.
• Experience with NoSQL databases, such as DynamoDB or MongoDB, and/or lakehouse and distributed query architectures.
• Familiarity with infrastructure observability, telemetry, monitoring, and alerting using tools such as Prometheus and Grafana.
• Experience with cloud-native systems, Docker, Kubernetes, and CI/CD practices.
• Experience using AI-assisted development tools such as Cursor, Windsurf, or equivalent.
• Strong analytical, problem-solving, and communication skills, with the ability to independently turn ambiguous requirements into production-ready solutions.

Preferred Qualifications:

• Understanding of infrastructure capacity planning, utilization forecasting, resource trending, and capacity optimization.
• Experience building telemetry dashboards and operational analytics platforms.
• Experience optimizing query performance, data partitioning, and processing efficiency for large-scale datasets.
• Familiarity with Agile methodologies, DevOps practices, automated testing, and continuous delivery.
• Experience designing end-to-end solutions across backend services, data pipelines, databases, and frontend visualization layers.