Boomi
Senior Software Quality Engineer - AI
Canada · Senior
Sponsorship not specified$90k-$112kDetected 18 hours ago
PythonJavaReactBackend DevelopmentCode ReviewGitCloud PlatformsCI/CDLinuxMachine LearningLLMsRAGAgentic AIAgileJiraConfluenceSeleniumPlaywrightpytestLoad TestingTest AutomationCommunicationCollaboration
About the role
- If this sounds like a good fit for you, check out boomi.com or visit our Boomi Careers page to learn more.
- How You'll Make An Impact As a Senior Software Quality Engineer, you will bridge traditional software quality engineering with cutting-edge AI quality practices.
Responsibilities
- Design and execute comprehensive test plans for backend services, APIs, and microservices architectures.
- Partner with Backend Engineers, AI Engineers, and Product teams to understand requirements and identify quality risks early in the development cycle.
- Develop automated testing frameworks using tools like Pytest, Playwright, unittest, and integration testing libraries.
- Build end-to-end test suites that validate the integration between backend systems, AI agents, databases & services.
- Design and implement automated evaluation frameworks for Generative AI features, including LLM / SLM model testing, prompt/ output validation, and behavioral assessment of agentic workflows.
- Develop quality metrics and evaluation methodologies for LLM-based applications, assessing accuracy, consistency, reliability, quality of AI/ML models.
- Create and maintain curated evaluation datasets and synthetic test data that cover edge cases, adversarial scenarios, and real-world variability.
- Implement quality gates and acceptance criteria and create test documentation including test plans, automation architecture, quality reports, and runbooks.
- Build strategies for testing in cloud environments & containerized execution.
- Implement performance testing, load testing, and reliability testing for production backend services and AI inference pipelines.
Requirements
- Strong understanding of CI/CD pipelines and experience integrating automated tests into deployment workflows.
- Familiarity with testing AI/ML applications or LLM-based systems, with understanding of unique quality challenges in non-deterministic systems.
- Knowledge of Generative AI concepts, agentic frameworks (ReAct, chain-of-thought, tool use), and LLM capabilities/limitations.
- Exposure to agentic AI patterns and knowledge of how to test tool orchestration, multi-turn conversations, and RAG systems.
- Experience with collaboration tools like Jira, Confluence, and Git for team workflows.
- Bachelor's degree in Compute
- 4+ years of experience in software quality engineering, test automation, or software development with strong QA focus.
- Experience with various testing methodologies: unit testing, integration testing, end-to-end testing, contract testing, performance testing, and test-driven development (TDD).
Skills
- About Boomi and What Makes Us Special
- Boomi, the data activation company for AI, powers the agentic enterprise by bringing data to life across the business.
- How You'll Make An Impact
Compensation
- $90k-$112k
This listing is sourced directly from Boomi's careers page and normalized into a canonical job model.