Asapp
Lead Machine Learning Engineer, Evaluations
Mountain View
Sponsorship not specified$170k-$190kDetected 42 days ago
PythonAWSKubernetesCI/CDKafkaMachine LearningData EngineeringNLPLLMsAgentic AIA/B TestingResearchMentoring
Stay score
odds of building a lasting career here
59Sponsors, lottery-bound
Cap-exempt (no lottery)0
Sponsors this role80
Entry-level history0
PERM / green-card track0
Lottery odds (Level IV)94
Fits your clock70
Sponsors, but it's cap-subject — you still face the weighted lottery (~61% per draw at Level IV). Good if you win; have a cap-exempt backup on your list.
Lottery odds assume a STEM candidate.
Personalize to your clock →Community outcomes
No reports yet — be the first to help the next applicant.
About the role
- We value customer obsession, purposeful speed, ownership, and a relentless focus on outcomes.
- ASAPP's AI Engineering team is seeking an enterprising, talented and curious machine learning engineer.
- This role will play a key part in designing and owning evaluation frameworks that ensure quality, safety, and performance across complex agentic systems.
Responsibilities
- Help develop the technical roadmap and architecture for the evaluation platform, from offline benchmarking to online/production monitoring of agentic and LLM-based systems.
- Design eval methodologies appropriate to different stages of the pipeline: golden/regression test sets, human-in-the-loop review workflows, LLM-as-judge approaches, and automated metrics for task success, safety, and hallucinations.
- Build the data infrastructure evaluation depends on: annotation and labeling pipelines, dataset versioning, data quality checks, and tooling that lets researchers and product teams run and interpret experiments without needing platform team help.
- Partner closely with Research, Product, and Platform teams to productize experiments into robust AI solutions
- Mentor and support other engineers through design reviews, feedback, and knowledge sharing.
- Deep, hands-on experience building and operating evaluation systems for modern ML/LLM/agentic systems- not just consuming existing eval tools.
- Design eval methodologies appropriate to different stages of the pipeline:
- Build the data infrastructure evaluation depends on:
- Desire to learn, teach, and collaborate closely with cross-functional peers.
- Experience building and evaluating agentic systems at scale.
Requirements
- Strong architectural skills, with proven experience designing complex, data-intensive software systems and production experience with Python, AWS, Kubernetes, and/or Docker.
- Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking.
- A Bachelor's Degree in CS or other related fields
- Experience with voice/audio quality evaluations.
- Production experience with LLM-centric services (e.g., inference, orchestration, evaluation, monitoring)
- Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks.
- Experience with conversational/customer-support AI domains (e.g., containment rate, conversation quality, goal completion).
- Knowledge of techniques for optimizing model architectures for faster inference.
- Experience with AWS, CI/CD, Kafka, Athena
Compensation
- Benefits include: Competitive compensation with stock options Comprehensive medical, vision, and dental insurance 401k matching Fitness and wellness stipend Mental well-being benefits Professional learning and development stipend Parental l
Benefits
- All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status.
- If you have a disability and need assistance with our employment
- Represent the eval platform to stakeholders outside the immediate team- set expectations on what "good" looks like for a model/agent release, and report on platform health and coverage.
- Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features).
- Benefits include:
This listing is sourced directly from Asapp's careers page and normalized into a canonical job model.