QA Engineer - Work From Home

About the Company

BMC Software develops enterprise technology solutions that help organizations connect people, systems, and data to improve business performance. Its products support large organizations in accelerating innovation, improving operational efficiency, and managing complex technology environments.

The company's Intelligent Z Optimization & Transformation (IZOT) product line focuses on helping enterprises monitor and manage mainframe systems. Its solutions support mainframe modernization by improving developer experience, system integration, application development speed, code quality, and security while reducing operational costs and risks.

About the Role

BMC Software is seeking a Principal Quality Automation Engineer to develop and maintain quality evaluation systems for generative AI, retrieval-augmented generation (RAG), and agentic AI applications. This role focuses on ensuring AI-powered agents deliver reliable, consistent, safe, and production-ready results through structured evaluation frameworks, automated testing, and data-driven release decisions.

You will build evaluation datasets, scoring rubrics, automated evaluation pipelines, and quality metrics that help engineering teams identify issues such as hallucinations, model drift, unsafe responses, and inconsistent outputs. Working with data science, AI engineering, and product teams, you will turn evaluation results into actionable improvements and help establish clear standards for releasing AI features.

The role is remote within the United States and involves independently owning defined evaluation projects from requirements and planning through implementation and delivery.

Key Responsibilities

• Define and implement evaluation strategies and frameworks for generative AI, large language models (LLMs), RAG systems, and agentic AI applications.
• Create and maintain golden datasets, evaluation rubrics, and scoring methods, including LLM-as-a-judge approaches, to measure AI quality consistently.
• Apply statistical sampling, repeated trials, judge calibration, dataset versioning, and anti-contamination practices to improve evaluation reliability.
• Develop automated evaluation harnesses and integrate them into CI/CD pipelines so model, prompt, and agent changes are evaluated before release.
• Establish quality metrics and release gates to distinguish production-ready AI features from prototypes.
• Design testing strategies for multi-agent workflows and AI systems that interact with tools, APIs, Model Context Protocol (MCP), and enterprise applications.
• Use Playwright where appropriate to support automated testing of AI-driven workflows.
• Evaluate end-to-end agent behavior, including planning, tool usage, response generation, and task completion.
• Analyze observability data and execution traces to identify failure patterns and recommend improvements to AI system reliability.
• Collaborate with data science and AI engineering teams to translate evaluation findings into model, prompt, and workflow improvements.
• Promote responsible AI practices and human-in-the-loop principles while contributing to governance and model validation.
• Own defined evaluation projects independently, from problem definition through delivery.
• Produce decision-ready quality reports and integrate evaluation workflows into release processes to support informed launch decisions.

Required Qualifications

• Bachelor's or master's degree in Computer Science, Artificial Intelligence, or a related field, or equivalent experience.
• Experience working with generative AI, LLM applications, AI/ML systems, and quality evaluation frameworks.
• Understanding of common LLM, RAG, and agentic AI failure modes, including hallucinations and inconsistent outputs.
• Knowledge of vector databases and embedding-based retrieval appropriate to the role's requirements.
• Proficiency in Python and experience building evaluation datasets, scoring rubrics, and automated evaluation workflows.
• Experience integrating automated evaluations into CI/CD pipelines.
• Familiarity with LLM observability and tracing tools, such as Langfuse or OpenTelemetry (OTel), and analytics platforms such as OpenSearch or similar technologies.
• A rigorous, data-driven approach to quality assessment and problem-solving.
• Ability to collaborate effectively with data science, engineering, and product teams.
• Demonstrated experience owning a defined AI evaluation suite or quality workflow that influenced a release decision.
• Experience building repeatable evaluation datasets and rubrics rather than relying solely on manual spot checks.
• Ability to independently manage well-defined quality engineering problems from initial definition through delivery.

Preferred Qualifications

• Experience with evaluation frameworks such as Ragas, DeepEval, promptfoo, or similar tools.
• Knowledge of AI safety, governance, and model validation practices.
• Experience working in enterprise or regulated environments.
• Familiarity with OpenShift or AWS.
• Experience with performance, load, or reliability testing for AI applications.
• Knowledge of MLOps practices.
• Experience with open-weight models or local model serving.
• Familiarity with MCP and Agile development practices.
• Experience using Atlassian tools.

Work Arrangement and Travel

This position can be based remotely within the United States. Occasional business travel may be required.

Compensation

The estimated annual base salary range for this position is $152,925–$254,875. Actual compensation depends on factors including skills, experience, training, licensure, certifications, and business and organizational needs. The listed range represents one component of BMC's overall compensation package, which may also include a variable compensation plan and country-specific benefits.

Equal Employment Opportunity

BMC Software is committed to equal employment opportunity regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, or veteran status. Reasonable accommodations are available during the application and hiring process.

BMC Software maintains a strict policy against requesting payment in exchange for employment opportunities.