Mercor
Research Engineer – Benchmarking
San Francisco
Sponsorship not specifiedDetected 60 days ago
Data StructuresAlgorithmsSQLNoSQLCloud PlatformsMachine LearningLLMsResearchCollaboration
Stay score
odds of building a lasting career here
40Risky
Cap-exempt (no lottery)0
Sponsors this role90
Entry-level history0
PERM / green-card track0
Lottery odds40
Fits your clock70
Thin sponsorship signal and lottery-bound. A low-probability bet with your clock running. Prioritize cap-exempt roles and proven entry-level sponsors first.
Lottery odds assume a STEM candidate.
Personalize to your clock →Community outcomes
No reports yet — be the first to help the next applicant.
About the role
- As a Research Engineer at Mercor, you'll work at the intersection of engineering and applied AI research.
- Your work will define how we measure tool use, agentic behavior, and real-world reasoning.
Responsibilities
- Benchmarking: Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning
- Evaluation systems: Build and operate LLM evaluation systems end-to-end runs, scoring, dashboards, and reporting, so researchers and applied AI teams can track model performance and compare runs at scale.
- categorize failure modes, quantify prevalence, and feed findings into reward design, data curation, and benchmark design.
- Rubrics and evaluators: Create and refine rubrics, automated evaluators, and scoring frameworks that drive training and evaluation decisions
- Benchmarking: Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning; ensure benchmarks scale with training and stay aligned with product and research goals.
Requirements
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Comfort with APIs, SQL/NoSQL, and cloud platforms for running and storing eval results.
- Ability to reason about model behavior, experimental results, and data quality from evals and failure analyses.
Nice to have
- Industry experience on a post-training or evaluation/benchmarking team (highest priority).
- Publications at top-tier venues (NeurIPS, ICML, ACL), especially in evaluation or benchmarking.
- Experience with synthetic data generation, rubric design, or RL-style workflows that use evals for reward shaping.
- Work samples or code (e.g., eval frameworks, benchmark suites, failure-analysis reports or tooling) that demonstrate relevant skills.
Compensation
- Bi-annual performance bonus structure
Benefits
- Bi-annual performance bonus structure
- Generous equity grant vested over 4 years
- Up to $15k Relocation bonus
- $10K housing bonus (if you live within 0.5 miles of our office)
- $1.5K monthly stipend for meals
- $200 monthly laundry reimbursement
- $200 monthly personal wellness reimbursement
- Health, Dental, Vision insurance
Company info
- Mercor's mission is to organize human intelligence to power the AI economy.
- We're a leading AI data company, building the layer between human expertise and frontier models.
- Millions of domain experts on the platform are paid over $4 million per day to train frontier AI models.
- Mercor's APEX benchmark family measures AI's real-world impact on professional work.
- Mercor Enterprise brings this same infrastructure to Fortune 500 companies:
- helping companies capture how their best people actually work, translating that expertise directly back into agents.
- Mercor is creating a new category of work where expertise powers AI advancement.
- Achieving this requires an ambitious, fast-paced and deeply committed team.
- You'll work alongside researchers, operators, and AI companies at the forefront of shaping the systems that are redefining society.
- Mercor is a profitable Series C company valued at $10 billion.
- We work in-person five days a week in our San Francisco, NYC, or London offices.
This listing is sourced directly from Mercor's careers page and normalized into a canonical job model.