Angi
Staff AI Product Analyst
Remote - United States · Staff+
Sponsorship not specified$165k-$225kDetected 55 days ago
PythonMachine LearningData EngineeringData ScienceData VisualizationLLMsRAGAgentic AIAI OrchestrationA/B TestingProduct StrategyRecruitingLeadershipCommunicationCollaboration
About the role
- The Staff AI Product Analyst will define AI-native product analytics for Angi, turning visionary technology into prioritized use cases, robust evaluation disciplines, and a trusted data ecosystem.
- As a senior individual contributor, you will have broad influence across the product analytics function and beyond, ensuring that both human teams and AI agents remain aligned on truth, quality, and usefulness.
- The ideal candidate is a systems-thinking data expert who bridges the gap between deep AI capabilities and practical business outcomes.
Responsibilities
- AI Product Strategy & Metrics Design
- Flagship Journey Measurement: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes.
- Causal Framing & Experimentation: Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship.
- Eval Framework Architecture: Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness.
- Performance Monitoring: Monitor live usage of Agentic AI analytics workflows-tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog.
- Strategic Collaboration: Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomes.
- Defining Success: Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metrics.
- Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes.
- Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship.
- Monitor live usage of Agentic AI analytics workflows-tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog.
Requirements
- 10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment.
- 1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsibly.
- Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices.
- Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline.
- Exceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholders.
- Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty.
- Playbooks & Narrative: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty.
Nice to have
- Demonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iteration.
- A systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business value.
- Compensation may vary based on factors such as cost of living.
- work hard and take time when you need it
- The rare opportunity to work with sharp, motivated teammates solving some of the most unique challenges and changing the world
- We value diversity
- We know that the best ideas come from teams where diverse points of view uncover new solutions to hard problems.
- Our hiring process may utilize artificial intelligence (AI) tools to assist in candidate screening and assessment.
Compensation
- The salary band for this position ranges from $165,000 - $225,000 commensurate with experience and performance.
- Compensation may vary based on factors such as cost of living.
Benefits
- This position will be eligible for a competitive year end performance bonus & equity package.
- Full medical, dental, vision package to fit your needs
- Flexible vacation policy; work hard and take time when you need it
- Pet discount plans & retirement plan with company match (401K)
- We welcome and value individuals who bring diverse life experiences, educational backgrounds, cultures, and work experiences.
Company info
- Founded in 1995 as Angie's List and rebranded in 2021
- Global company with 9 brands in 8 countries and employees worldwide
- Homeowners have turned to us for 300 million home projects and counting
- This role is central to building Angi's agentic product analytics capability-defining where AI-native analytics creates real leverage, designing the evaluation frameworks that earn organizational trust, and establishing repeatable system standards.
This listing is sourced directly from Angi's careers page and normalized into a canonical job model.