Distyl AI
AI Engineer, Evaluation
San Francisco
Sponsorship not specifiedDetected 33 days ago
PythonLLMsA/B TestingSupply ChainResearch
About the role
- ABOUT DISTYL AI Distyl is an applied AI technology company partnering with the world's most ambitious institutions to rearchitect critical operations for the frontier of AI.
- As a result, Distyl's technologies affect some of the world's largest operations - from hundreds of millions of consumer interactions to tens of millions of supply chain transactions and millions of patient journeys.
- This role is for engineers who believe that AI systems only improve when measurement is tightly coupled to development-and who want to apply that philosophy directly to systems that matter.
Responsibilities
- Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
- Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases.
- Develop and maintain evaluation pipelines-offline and online-that integrate directly into system iteration loops.
- Evaluation results inform prompt design, agent logic, model selection, and release readiness, ensuring that system changes are driven by measurable improvements rather than intuition alone
- They investigate where evaluation signals diverge from real-world outcomes and refine grading approaches to maintain signal quality as systems and domains evolve
- Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments. You treat evaluation code with the same rigor as application code
- Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment. You design evaluation systems that support fast iteration while maintaining trust and safety in production
Skills
- You treat evaluation code with the same rigor as application code
- AI-Native Working
Company info
- a 100% production deployment success rate for our customers and one of the few enterprise AI companies to run a profitable business.
Apply directly at Distyl AI →Create a free account for alerts like thisView Distyl AI immigration profile
This listing is sourced directly from Distyl AI's careers page and normalized into a canonical job model.