Sentry
Senior Software Engineer, AI Evals
San Francisco, California · Senior
Sponsorship not specified$240k-$280kDetected 175 days ago
TypeScriptPythonMachine LearningLLMsAgentic AIA/B TestingManual Testing
About the role
- This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale.
Responsibilities
- Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
- Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
- Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
- Partner closely with applied AI engineers and product leaders to define what "good" looks like and translate it into measurable criteria
- Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring
- Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
- Thrive in cross-functional environments and enjoy influencing model design through better evaluation
- You'll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.
- We strive to build a diverse team, with an inclusive culture where every teammate can thrive.
Requirements
- Comfort writing production-quality code (we use Python and TypeScript)
- Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
- Minimum 5+ years of professional experience with a Bachelor's degree in computer science, machine learning, or a related field
- Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Nice to have
- experience evaluating LLMs, agentic systems, or AI-assisted developer tools
- Sentry is an open-source company because we believe that everyone, everywhere, should have the ability and tools to make great software.
- Software should be accessible.
- That starts with making our industry accessible.
- Want to learn more about how Sentry handles applicant data?
Compensation
- The base salary range (or hourly wage range, if applicable) that Sentry reasonably expects to pay for this position is $240,000 to $280,000 USD.
Benefits
- See Sentry Benefits https://sentry.io/careers/ for more details about the Company's benefit plans/programs.
Equal opportunity
- Sentry is committed to providing equal employment opportunities to its employees and candidates for employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally-protected characteristic. This commitment includes the provision of reasonable accommodations to employees and candidates for employment with physical or mental disabilities who require such accommodations in order to (a) perform the essential functions of their jobs, or (b) seek employment with Sentry. We strive to build a diverse team, with an inclusive culture where every teammate can thrive. Sentry is an open-source company because we believe that everyone, everywhere, should have the ability and tools to make great software. Software should be accessible. That starts with making our industry accessible.
- If you need assistance or an accommodation due to a disability, you may contact us at accommodations@sentry.io.
- Sentry is committed to providing equal employment opportunities to its employees and candidates for employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally-protected characteristic.
- This commitment includes the provision of reasonable accommodations to employees and candidates for employment with physical or mental disabilities who require such accommodations in order to (a) perform the essential functions of their jobs, or (b) seek employment with Sentry.
Visa & Work Authorization
- candidates for employment regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally-protected
This listing is sourced directly from Sentry's careers page and normalized into a canonical job model.