Procore Technologies

Software Engineering Manager, AI Platform

US - Texas - Austin · Director

Sponsorship not specified$169k-$232kDetected 7 days ago

PythonGoCI/CDMachine LearningData EngineeringLLMsRAGAgentic AIA/B TestingProcurement

About the role

This position reports into Sr Director of the Procore AI Engineering team and will be 2 days per week hybrid role in our Austin office.

Lead and grow a team of engineers focused on evaluation infrastructure, quality measurement, and developer tooling for AI agents.
Partner with AI/ML, Product, and Agent teams to define quality metrics for agents (relevance, accuracy, latency, safety, user satisfaction, token usage) and build automated pipelines to compute them at scale.
Build frameworks for human-in-the-loop evaluation - annotation workflows, rating interfaces, and inter-rater reliability measurement.
Recruit, mentor, and develop engineers - fostering a culture of ownership, curiosity, and rigorous experimentation.
5+ years managing engineering teams or technical leads, with 7+ years total in software engineering.
Experience building evaluation, quality measurement, or observability platforms for LLM-based or agentic systems (RAG pipelines, multi-step agents, tool-use agents).
In this role, you'll build the infrastructure and tooling that enables users and internal teams to measure, benchmark, and improve the quality of AI agents - including Search Agent, RFI Create Agent, Invoice Agent, and future agentic products.

Strong understanding of evaluation methodologies: precision/recall, LLM-as-judge, human annotation, A/B testing, and statistical significance frameworks.
Proven ability to translate ambiguous problem spaces into clear technical strategies and executable roadmaps.
Familiarity with evaluation frameworks such as RAGAS, DeepEval, LangFuse, or custom eval harnesses.
Experience with construction-tech, procurement, or enterprise B2B SaaS domains.

168,560.00 - 231,770.00 USD Annual
This role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation.
Procore is committed to offering competitive, fair, and commensurate compensation.
Actual compensation will be based on a candidate's job-related skills, experience, education or training, and location.

Define the technical vision and roadmap for the Evaluation Platform - covering offline evaluations (batch benchmarks, regression suites) and online evaluations (live traffic quality monitoring, A/B testing).
Drive engineering excellence: code quality, system reliability, test coverage, on-call health, and technical debt management.

Procore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles' Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.
A criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.
Design and deliver user-facing evaluation tools that allow customers and internal teams to assess agent output quality, compare model versions, and identify regressions.
What we're looking for:
We're looking for a Software Engineering Manager for our Evaluation Platform team to join Procore's Construction Intelligence organization.

This listing is sourced directly from Procore Technologies's careers page and normalized into a canonical job model.