Gleanwork

Machine Learning Engineer, LLM Evals & Observability

San Francisco, CA

Sponsorship not specified$200k-$300kDetected 26 days ago

PythonGoGitMachine LearningData EngineeringNLPLLMsAgentic AIComplianceZendeskMicrosoft Teams

About the role

It's a rare combination of infrastructure engineering, applied ML, and direct product impact.
Evaluate new models and product changes before they ship - providing the quality signal that gates launches and prevents regressions.
Strong backend fundamentals in Go and Python; comfortable with distributed data pipelines.

If you care deeply about quality and want to build the systems that make it measurable, this role is for you.
Build and maintain large-scale evaluation pipelines that measure assistant quality across thousands of real user queries.
Build LLM-powered judges that score metrics like correctness, completeness, and response quality, and align them against human judgment.
Close the loop between quality measurement and improvement using eval results, customer feedback, and techniques like automated prompt iteration to help drive concrete gains in assistant behavior.
Build observability infrastructure for AI agents: trace enrichment, data pipelines, and dashboards that make assistant behavior inspectable.
You care about quality - not just in the systems you build, but in the product you're helping measure and improve.
As part of the interview process, you'll complete a brief AI-focused exercise or discussion so we can understand how you think about, design, and use AI to drive impact in your role.
Depending on your location, the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), or other privacy laws may regulate the way we manage the data of job applicants.

You'll ship agentic capabilities on an open, extensible stack, with the craft and care required for enterprise trust, as we bring Work AI to every employee, in every company.
2+ years of software engineering experience with strong coding skills.
Feel free to reference any tools, platforms, or workflows you use today - prior Glean experience isn't required.
By submitting your application, you are agreeing to our use and processing of your data as required.

Certain roles may be eligible for variable compensation, equity, and benefits.
We offer a comprehensive benefits package including competitive compensation, Medical, Vision, and Dental coverage, generous time-off policy, and the opportunity to contribute to your 401k plan to support your long-term goals.
When you join, you'll receive a home office improvement stipend, as well as an annual education and wellness stipends to support your growth and wellbeing.
The standard base salary range for this position is $200,000 - $300,000 annually.
Compensation offered will be determined by factors such as location, level, job-related knowledge, skills, and experience.

Certain roles may be eligible for variable compensation, equity, and benefits.
When you join, you'll receive a home office improvement stipend, as well as an annual education and wellness stipends to support your growth and wellbeing.
We do not discriminate based on gender, ethnicity, sexual orientation, religion, civil or family status, age, disability, or race.

Thrive in a customer-focused, tight-knit and cross-functional environment - being a team player and willing to take on whatever is most impactful for the company
Collaborate with engineers across the company to make evals a first-class part of how we ship.
We are a diverse bunch of people and we want to continue to attract and retain a diverse range of people into our organization.

This listing is sourced directly from Gleanwork's careers page and normalized into a canonical job model.