ID.me

ID.me

Staff Software Engineer- AI Agent Evaluations

Mountain View, California, United States · Staff+ · Full-time

Sponsorship not specified$63k-$800kDetected 21 days ago
PythonJavaGoCI/CDDatadogMachine LearningLLMsRAGAgentic AILangGraphComplianceResearchLeadershipCommunicationCollaborationMentoring

About the role

  • This Staff Engineer role sits at the intersection of engineering, applied AI, testing and developer experience.

Responsibilities

  • Define AI Quality Standards: Own the framework for how ID.me evaluates, validates, and monitors AI agents - from prompt-based features to fully autonomous multi-step workflows.
  • Build Eval Infrastructure: Design and maintain evaluation pipelines for LLM outputs, agent behavior, tool use, and multi-turn interactions across development, staging, and production environments.
  • Agentic Test Strategy: Lead the design of test suites that handle non-determinism - red-teaming agents, golden dataset construction, LLM-as-judge pipelines, and property-based testing for AI outputs.
  • Champion Developer Experience: Build the internal tooling, feedback loops, and testing workflows that make it fast and safe for engineers to develop and ship AI features with confidence.
  • Cross-Team Collaboration: Partner with Security, Platform, Product, and AI/ML teams to embed quality gates into agent development workflows.
  • Mentorship: Guide senior and mid-level engineers through evaluation design, observability strategy, and testing approaches specific to AI systems.
  • Design and maintain evaluation pipelines for LLM outputs, agent behavior, tool use, and multi-turn interactions across development, staging, and production environments.
  • Lead the design of test suites that handle non-determinism - red-teaming agents, golden dataset construction, LLM-as-judge pipelines, and property-based testing for AI outputs.
  • Build the internal tooling, feedback loops, and testing workflows that make it fast and safe for engineers to develop and ship AI features with confidence.
  • Partner with Security, Platform, Product, and AI/ML teams to embed quality gates into agent development workflows.

Requirements

  • Bachelor's degree in Computer Science, Engineering, or equivalent experience
  • Demonstrated experience evaluating or testing LLM-powered features or autonomous agents in production
  • Proficiency with AI-assisted development tools (Claude Code, Cursor, or equivalent) - you build with AI every day
  • Experience designing test infrastructure, CI/CD quality gates, or evaluation pipelines at scale
  • Familiarity with agentic frameworks (Claude API / Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar)
  • Production monitoring experience for AI systems: behavioral drift detection, output sampling, shadow scoring
  • Red-teaming or adversarial testing experience for AI models or agents

Nice to have

  • Background in identity verification, fraud detection, or regulated industries
  • Familiarity with Anthropic's model evaluation methodology or similar published eval research
  • Experience with observability tooling (Datadog, OpenTelemetry) applied to AI workloads
  • All ID.me employees share in the responsibility for fulfilling our commitment to equal employment opportunity.
  • Upon request we will provide you with more information about such accommodations.
  • ID.me participates in E-Verify.

Skills

  • Company Overview
  • ID.me is the next-generation digital identity wallet that simplifies how individuals securely prove their identity online.
  • ID.me is committed to "No Identity Left Behind" to enable all people to have a secure digital identity.
  • To learn more, visit https://network.id.me/.
  • ID.me is a full-time, in-office culture.

Compensation

  • The annual base salary listed does not include a company bonus, incentive for sales roles, equity and benefits which will be determined based on experience, skills, education, relevant training, geographic location and role.

Benefits

  • Drive AI-First Engineering Culture: Raise the quality bar across the engineering org by establishing patterns, tooling, and education for how teams write, test, and deploy AI features responsibly.

Company info

  • Raise the quality bar across the engineering org by establishing patterns, tooling, and education for how teams write, test, and deploy AI features responsibly.
Apply directly at ID.me →Create a free account for alerts like thisView ID.me immigration profile

This listing is sourced directly from ID.me's careers page and normalized into a canonical job model.