Harper
Senior Member of Technical Staff, AI Quality
San Francisco · Staff+
Sponsorship not specified$176k-$253kDetected 50 days ago
Machine LearningLLMsLLMOpsCRMLogisticsCommunicationUnderwriting
About the role
- Turning judgment into compute only compounds if the company can tell whether the compute is getting better.
- Your job is to turn agent quality from a vibe into a number.
- Harper's agents handle intake, sales, service, voice, and submission packaging; every one needs to be evaluated, regression-tested, and monitored in production.
Responsibilities
- Build capability + regression eval suites for your assigned agents - intake, submissions, placements, renewals, CRM, or voice.
- Design graders. Deterministic first (string match, state check, tool-call assertions)
- 3-6 years building software, with hands-on production LLM/agent eval experience - capability + regression suite design, LLM-as-judge graders, golden datasets.
- You've designed an LLM-as-judge rubric that survived human calibration, and you debug a hallucination by reading transcripts, not aggregate dashboards.
- Design graders. Deterministic first (string match, state check, tool-call assertions); LLM-as-judge where deterministic fails; human calibration on samples.
Requirements
- You can describe a specific regression an eval suite you built caught - and exactly how it would have leaked otherwise.
Nice to have
- open-source eval-framework contributions
- red-team/adversarial testing
- voice eval (latency, interruption, transcription accuracy)
- ML eval/observability background.
- AI quality is the discipline that decides whether the whole bet holds, which means the work is scrutinized and the bar is high - your evals are what let everyone else ship fast without flying blind.
- The right person wants that leverage and that pace.
Compensation
- $176,000-$253,000 cash (base + target performance bonus), plus competitive equity.
Benefits
- Uber commuter benefits; breakfast, lunch, and dinner provided; snacks and coffee stocked; free gym membership; health, dental, and vision.
This listing is sourced directly from Harper's careers page and normalized into a canonical job model.