Sonatus

Sonatus

Senior Manager, Engineering - AI Validation

Sunnyvale, CA · Senior

Sponsorship not specified$220k-$265kDetected 4 days ago
PythonBashVector DatabasesCloud PlatformsJenkinsLinuxMachine LearningData EngineeringLLMsRAGAgentic AILLMOpsLangGraphSystems EngineeringElectrical EngineeringPlaywrightpytestManual Testing

About the role

  • Establish AI governance, Responsible AI practices, model lineage, safety guardrails, and compliance processes appropriate for automotive safety-critical systems and enterprise AI products.
  • Act as the quality gatekeeper for AI-enabled releases, partnering with engineering, product, safety, and OEM stakeholders to identify risks, define release criteria, and ensure production readiness.

Responsibilities

  • Define and drive Sonatus's AI validation strategy across embedded, in-vehicle, cloud-connected, and cloud-native AI systems, identifying gaps in model development, testing, deployment, and governance.
  • Lead, hire, mentor, and grow a high-performing AI Validation organization, establishing scalable engineering processes, technical direction, and execution excellence.
  • Own the end-to-end validation strategy for AI/ML models, LLMs, RAG pipelines, and agentic AI workflows-from data pipelines and model training through cloud services and in-vehicle deployment.
  • Design and maintain evaluation harnesses for RAG and agentic systems that measure retrieval quality, grounding, citation accuracy, factual consistency, context relevance, safety, latency, reliability, and execution correctness.
  • Evaluate, integrate, and optimize open-source and commercial AI validation technologies, driving build-versus-buy decisions for Sonatus's AI quality platform.
  • Collaborate across engineering teams to define validation strategies for emerging AI capabilities, rapidly prototype new evaluation approaches, and standardize successful practices into reusable frameworks.
  • Drive continuous improvement by tracking industry advances in AI evaluation, agentic AI, LLM validation, and RAG systems, translating them into scalable validation capabilities across Sonatus.

Requirements

  • 10+ years of experience in software or systems engineering-including embedded, cloud, networking, security, or automotive domains-with 3+ years leading high-performing engineering or QA organizations.
  • Proven experience designing and implementing scalable evaluation frameworks for AI systems, including multi-step agentic workflows, regression testing, benchmarking, and automated quality scoring.
  • Practical experience with RAG evaluation frameworks such as RAGAS, including evaluation tuning, embedding optimization, retrieval quality improvement, and production-scale LLM evaluation pipelines.
  • Experience validating hallucination, grounding, citation accuracy, bias, fairness, toxicity, and factual consistency in production LLM applications.
  • Experience designing systems that verify external knowledge claims and ensure responses are grounded in traceable citations and trusted data sources.
  • Strong experience testing cloud-native platforms and cloud-managed embedded products, including end-to-end system validation.
  • Experience establishing AI governance, safety, compliance, and Responsible AI practices for enterprise or safety-critical systems.
  • Proficiency in Python, Linux, shell scripting, modern test frameworks (PyTest, Playwright, Behave), and engineering productivity tools such as Jenkins and JIRA.
  • Hands-on experience developing, deploying, testing, or operating AI/ML systems, with strong expertise in modern ML workflows, neural networks, and MLOps.
  • Deep understanding of LLMs, RAG architectures, vector databases, embeddings, retrieval optimization, and agentic AI frameworks such as LangGraph or equivalent orchestration platforms.
  • Strong expertise with hybrid evaluation methodologies combining deterministic validation (citation grounding, structural validation, exact matching) and probabilistic LLM-as-a-Judge techniques (faithfulness, answer relevance, context precision, task completion).
  • Ways to Stand Out:

Nice to have

  • Bachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field required (MS preferred).

Skills

  • Experience validating AI or agentic systems in safety-critical or regulated industries (automotive, aerospace, medical).
  • Experience implementing enterprise-grade AI governance (auditability, monitoring, policy enforcement) in production systems.

Compensation

  • $220,000 - $265,000 USD

Benefits

  • Health care plan (Medical, Dental & Vision)
  • Flexible and Dependent Care Expense program
  • Retirement plan (401k)
  • Life Insurance (Basic, Voluntary & AD&D)
  • Unlimited paid time off per year, 14+ paid holidays
  • Wellness benefit allowance
  • Phone & Internet reimbursement

Company info

  • At Sonatus, we're driving the transformation to AI-enabled software-defined vehicles.
  • Traditional automotive software methods can't keep pace with consumer expectations shaped by the mobile industry-where features evolve rapidly, update seamlessly, and improve continuously.
  • That's why leading OEMs trust Sonatus to accelerate this shift.
  • Our technology is already in production across more than 8 million vehicles on the road today and rapidly expanding.
  • Headquartered in Sunnyvale, CA, with 250+ employees worldwide, Sonatus combines the agility of a fast-growing company with the scale and impact of an established partner.
  • Backed by strong funding and proven by global deployment, we're solving some of the most interesting and complex challenges in the industry.
  • Join us and help redefine what's possible as we shape the future of mobility.
Apply directly at Sonatus →Create a free account for alerts like thisView Sonatus immigration profile

This listing is sourced directly from Sonatus's careers page and normalized into a canonical job model.