O.C. Tanner

O.C. Tanner

Software Engineer in Test - AI

USA - Utah-Salt Lake City-Headquarters

Sponsorship not specifiedDetected 7 days ago
PythonJavaRubyDistributed SystemsBackend DevelopmentSQLNoSQLAWSDockerKubernetesCI/CDKafkaLLMsRAGLangGraphCybersecuritySeleniumLoad TestingTest AutomationLeadershipCommunicationCollaboration

About the role

  • O.C. Tanner is the global leader in software and services that improve workplace culture through meaningful employee experiences.
  • Join us as we help people all over the world thrive at work.

Responsibilities

  • Design, develop, and maintain automated test frameworks for AI platform services, APIs, agents, prompt-based workflows, and RAG-enabled applications.
  • Develop automated AI evaluation suites and regression benchmarks that continuously measure model behavior and detect quality degradation before release.
  • Build and execute functional, integration, end-to-end, regression, performance, and reliability tests for AI-driven systems and services.
  • Validate responsible AI controls, permissions, guardrails, data handling practices, and business rules that protect customer trust.
  • Partner with Engineering, Product, and Support throughout the software lifecycle to drive risk-based testing strategies, influence quality to ensure production readiness.
  • Utilize observability tools, logs, metrics, and traces to investigate production issues and perform root cause analysis.

Requirements

  • 3+ years of experience in software testing, quality engineering, test automation, or software development.
  • Experience testing APIs, microservices, distributed systems, backend services, or event-driven architectures.
  • Strong experience testing AI-enabled applications using technologies such as LLMs, LangChain, LangGraph, or similar platforms.
  • Hands-on experience evaluating AI-generated outputs using datasets, scoring rubrics, golden test sets, benchmarking frameworks, or automated quality checks.
  • Experience testing RAG systems, including retrieval quality, embeddings, grounding, citations, context accuracy, and streaming responses.
  • Experience integrating automated testing into CI/CD pipelines and modern software delivery practices.
  • Experience investigating production issues and using incident and defect data to improve system quality and reliability.
  • Strong understanding of test automation, performance testing, risk-based testing, release validation, and responsible AI principles.
  • Excellent collaboration and communication skills, with the ability to clearly articulate quality risks and acceptance criteria.

Nice to have

  • Experience with AWS, Kubernetes, Docker, and cloud-native architectures.
  • Experience testing event-driven systems using Kafka or similar messaging platforms.
  • Familiarity with application security testing and OWASP Top 10 principles.
  • Proficiency in Python and at least one additional programming language such as Java, Ruby, or Go.
  • Experience with reliability engineering practices, including observability, production readiness reviews, incident analysis, SLOs, and continuous improvement.
  • Experience conducting performance, load, stress, scalability, or resilience testing.
Apply directly at O.C. Tanner →Create a free account for alerts like thisView O.C. Tanner immigration profile

This listing is sourced directly from O.C. Tanner's careers page and normalized into a canonical job model.