Posh

Senior Data Scientist, AI

New York City · Senior · Full-time

Sponsorship not specifiedDetected 231 days ago

PythonSQLMachine LearningData EngineeringData ScienceLLMs

About the role

As one of the early data hires at Posh, you'll shape the technical direction of our AI quality strategy and set the standards for how agent performance is defined, measured, and improved over time.
Your work will directly inform how we iterate on our AI agent and how we know when it's ready to ship.
You'll establish best practices for data quality, governance, and documentation, ensuring our evaluation framework remains trustworthy and rigorous as we grow.

Own the data that feeds our evaluation pipeline, from ground truth datasets and labeled examples to behavioral signals and the semantic layer.
Build the testing infrastructure to evaluate agent performance across accuracy, relevance, and user satisfaction.
Run structured experiments and pre/post analyses to assess the impact of model and product changes, and build dashboards that keep the team aligned on performance trends and regressions before they become problems.
Partner with Product to translate business goals into evaluation criteria and measurement requirements.
Implement best practices for data quality, documentation, observability, and lineage across all evaluation-bound datasets.
Define what good looks like, build the rubrics and benchmarks, and own the feedback loops that drive iteration.
Build ETL/ELT pipelines that transform raw behavioral, transactional, and interaction data into clean evaluation inputs.
Instrumenting Agent Tests, Experiments, and Success Metrics: Build the testing infrastructure to evaluate agent performance across accuracy, relevance, and user satisfaction.
Ensuring Strong Data Governance and Documentation: Implement best practices for data quality, documentation, observability, and lineage across all evaluation-bound datasets. Be the person who makes sure the foundation doesn't rot.
Posh enables anyone to build an IRL community based on shared interests, while connecting consumers with the communities of people just like them.

Has at least 5 years of hands-on experience in data science or analytics engineering.
Demonstrates strong proficiency in SQL and Python, with deep experience cleaning data, engineering features, and building efficient, production-ready modeling pipelines.
Possesses 5+ Years of full time Data Experience: Has at least 5 years of hands-on experience in data science or analytics engineering.
Expert in SQL and Python: Demonstrates strong proficiency in SQL and Python, with deep experience cleaning data, engineering features, and building efficient, production-ready modeling pipelines.
Strong Ability to Analyze and Evaluate Models or AI Systems: Skilled in designing experiments, interpreting model performance, and communicating insights clearly to both technical and non-technical stakeholders.
Experience with AI/LLM Evaluation or Agent Quality: Has built or contributed to evaluation frameworks, golden datasets, or quality measurement systems for AI models or agent-based products in production environments.

Experience with tools like LangSmith, Langfuse, or similar LLM observability and evaluation platforms, and/or working on consumer-facing AI products at scale.
This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.
Posh is committed to providing reasonable accommodations for qualified individuals with disabilities in our job application procedures.

This is an in-person position at our New York City office, located in the heart of SoHo.

This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.

We are all social creatures, but the dominant "social" companies today have evolved into digital loneliness machines, driving isolation, anxiety, and mental health challenges around the world.
LLM/Agent Tooling (Bonus): Experience with tools like LangSmith, Langfuse, or similar LLM observability and evaluation platforms, and/or working on consumer-facing AI products at scale.

Exhibits high interest in startups and has experience building the early foundation of a data team at a small tech company.
Has a Background in Early Stage Data Team: Exhibits high interest in startups and has experience building the early foundation of a data team at a small tech company.

This listing is sourced directly from Posh's careers page and normalized into a canonical job model.