Elicit

Evaluation Engineer

Oakland, CA (or remote within US timezones) · Senior · Internship

Sponsorship not specified$140k-$185kDetected 181 days ago

TypeScriptPythonMachine LearningData EngineeringStatisticsResearch

About the role

Elicit radically increases the amount of good reasoning in the world. - For experts, Elicit pushes the frontier forward. - For non-experts, Elicit makes good reasoning more affordable.
Elicit is a scalable ML system based on human-understandable task decompositions, with supervision of process, not outcomes.
This expands our collective understanding of safe AGI architectures.

You'll build a lightning-fast basic evals infrastructure that schedules tasks to introduce practically no latency; and then you'll figure out clever ways to solve the fundamental sources of latency (building a version of Elicit, running it on a query, and evaluating it using LMs)
15% working closely with the evals team to build and improve specific evals (e.g., an eval of our paper search within our systematic review flow)

At least 3 years of experience as a professional software engineer, with demonstrated experience building complex backend systems (e.g., backend for a complex website, data pipelines, etc.)
There's no particular experience you must have, but we'll evaluate your aptitude.
Knowledge of statistics (for e.g. calculating power and credence intervals for evals)
Experience with advanced Python (asyncio/trio and parallel processing strategies)
Knowledge of pharma/biomed

For all roles at Elicit, we use a data-backed compensation framework to keep salaries market-competitive, equitable, and simple to understand.

Fully covered health, dental, vision, and life insurance for you, generous coverage for the rest of your family
Flexible vacation policy, with a minimum recommendation of 20 days/year + company holidays
Every Elician receives a $200 monthly wellbeing stipend to spend on whatever supports your health and wellbeing
A new Mac + $1,000 budget to set up your workstation or home office in your first year, then $500 every year thereafter
$1,000 quarterly AI Experimentation & Learning budget, so you can freely experiment with new AI tools, take courses, purchase educational resources, or attend AI-focused conferences and events
$140-185k + equity
$175-230k + equity
The rest on learning how people interact with the eval system so you can make it work better for them, and understanding what our users want from Elicit so evals measure what matters

In addition to working on important problems as part of a productive and positive team, we also offer great benefits (with some variation based on location):

This listing is sourced directly from Elicit's careers page and normalized into a canonical job model.