Pareto AI
Strategic Projects Lead
US Remote
Sponsorship not specifiedDetected 69 days ago
PythonSQLMachine LearningData EngineeringData ScienceLLMsAgentic AIMLOpsProject ManagementResearchCommunication
About the role
- This is a technical operations role, not a project management role.
Responsibilities
- Pipeline architecture Design end-to-end data collection and evaluation pipelines for RLVR, RLHF, SFT, red-teaming, and model evaluation workflows.
- This includes expert sampling strategy, annotation schema, rubric structure, inter-rater calibration, and QA system design.
- Agentic system deployment Build, test, and iterate on AI agents that automate pipeline tasks - quality gate review, expert matching, output flagging, throughput anomaly detection.
- Design and run audits using inter-rater reliability metrics, calibration sets, and statistical sampling.
- You'll be responsible not just for catching quality issues but for building systems that prevent them - automated checks, structured output validation, and model-assisted review layers where appropriate.
- Demonstrated ownership of a data or ML pipeline from scoping through delivery - including quality design, not just throughput tracking
Requirements
- Proficiency in Python and SQL for data manipulation, pipeline monitoring, and quality analysis - you should be comfortable writing light scripts to parse formats, run statistical checks, and build lightweight tooling
- Working knowledge of LLM internals: RLHF/SFT training loops, how prompt structure affects output distribution, RL environment setup qualities (tool use) for agentic data collection / eval projects.
- Hands-on experience with at least one agentic or LLM workflow framework (LangChain, DSPy, AutoGen, direct tool-use via API, or equivalent)
- Comfort operating with ambiguity in a fast-moving environment where model requirements shift and client priorities evolve
- Direct experience with RL environment data pipelines, evaluation framework design, and red-teaming workflows
- Experience designing or operating agentic systems in a production or near-production context
- Prior client-facing or technical program management experience in an AI/ML-adjacent context
Benefits
- Translate research-driven requirements - evaluation rubrics, domain coverage targets, latency constraints, benchmark specifications - into operational workflows.
Company info
- This is a growing part of the role
Apply directly at Pareto AI →Create a free account for alerts like thisView Pareto AI immigration profile
This listing is sourced directly from Pareto AI's careers page and normalized into a canonical job model.