Devoted
Senior Manager, AI Performance & Operations
Remote USA · Senior · Full-time
Sponsorship not specified$110k-$172kDetected 30 days ago
Machine LearningLLMsRAGA/B TestingIncident ResponsePatient CareLeadershipCommunicationPlain Language
About the role
- This person is the accountable operator for Guide's highest-volume and highest-risk AI workflows once they are live.
Responsibilities
- Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions.
- Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure.
- Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production.
- Deep personal accountability with a drive to own results rather than just producing recommendations.
- They maintain the portfolio view of where each workflow stands, detect degradation early, diagnose the root cause, drive corrective action, and verify that implemented changes improve outcomes.
- They also serve as the in-house expert for early-stage projects by helping teams define instrumentation, evaluation design, launch criteria, and safe hand off into general availability.
Requirements
- Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems.
- Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments.
Skills
- Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows.
Compensation
- $110,000 - $172,000 / year
- The pay range listed for this position is the range the organization reasonably and in good faith expects to pay for this position at the time of the posting.
- Once the interview process begins, your talent partner will provide additional information on the compensation for the role, along with additional information on our total rewards package.
- Our management team is dedicated to this policy with respect to recruitment, hiring, placement, promotion, transfer, training, compensation, benefits, employee activities and general treatment during employment.
Benefits
- Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met.
- Employer sponsored health, dental and vision plan with low or no premium
- Generous paid time off
- $100 monthly mobile or internet stipend
- Stock options for all employees
- Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
- Parental leave program
- Our Total Rewards package includes:
- Our total rewards package is for full time employees only.
- Healthcare equality is at the center of Devoted's mission to treat our members like family.
- At Devoted Health, we're on a mission to dramatically improve the health and well-being of older Americans by caring for every person like family.
Equal opportunity
- Devoted is an equal opportunity employer.
Apply directly at Devoted →Create a free account for alerts like thisView Devoted immigration profile
This listing is sourced directly from Devoted's careers page and normalized into a canonical job model.