Overtone
Applied AI/ML Engineer
New York City
Sponsorship not specified$200k-$300kDetected 127 days ago
TypeScriptPythonReactFastAPIPostgreSQLVector DatabasesRESTMachine LearningData ScienceLLMsMLOpsA/B TestingResearchCollaboration
About the role
- As a founding AI Engineer at Overtone, you will build and own the AI systems that power our core product experience-from conversational intelligence to matchmaking insight generation.
- You'll design prompting, evaluation, and model infrastructure that makes our AI reliable, trustworthy, and continuously improving.
- This is a hands-on, early-stage role: you'll run experiments, ship infrastructure, and build the tools that give the team real visibility into model behavior and quality.
Responsibilities
- LLM systems & prompt architecture: Design and maintain the prompts, structured outputs, and orchestration systems that power Overtone's AI-driven features.
- Evaluation frameworks: Build evaluation infrastructure to measure AI quality at scale using tools such as Langfuse, TensorZero, and Promptfoo. You'll work with our team to make sure LLM judges are aligned.
- Model experimentation: Run structured experiments across models, prompts, and configurations to optimize quality, cost, and latency.
- AI observability & tooling: Build internal tools-dashboards, admin interfaces, debugging workflows-that give the team visibility into system behavior and model performance.
- Model serving & production inference: Own the model serving layer - deploying models, managing inference infrastructure (API providers and self-hosted), model versioning, and cost/latency optimization in production.
- Work with the backend lead to define clean service contracts between the AI layer and the rest of the system.
- Memory & personalization systems: Design and maintain vector databases and memory architectures that enable personalization and context-aware experiences.
- Responsible AI systems: Ensure systems are fair, unbiased, and auditable, building evaluation pipelines and human-in-the-loop processes where appropriate.
- 5+ years of engineering experience, with meaningful time spent building LLM-powered products in production
Requirements
- Familiarity with vector databases and retrieval-augmented architectures
- Experience with fine-tuning techniques such as SFT, DPO, or RLHF
- Strong understanding of when ML models outperform prompt-based systems
- Experience working in early-stage environments with high ownership
- Experience with Python, SQLAlchemy, FastAPI, PostgreSQL
Nice to have
- Experience with recommender systems or ranking systems
- Experience with MLOps and model lifecycle infrastructure
- We look for these qualities in everyone we hire at Overtone.
- Curiosity: You seek to understand before you decide.
- You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.
- Craft: You care deeply about quality and coherence.
- You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.
- Collaboration: You engage with others honestly and generatively.
Compensation
- Salary of $200k to $300k.
- We look for these qualities in everyone we hire at Overtone.
- Curiosity: You seek to understand before you decide.
- You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.
- Craft: You care deeply about quality and coherence.
- You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.
Benefits
- This position will receive meaningful stock options with considerable upside potential.
- We provide standard benefits such as health insurance, dental insurance, etc.
Apply directly at Overtone →Create a free account for alerts like thisView Overtone immigration profile
This listing is sourced directly from Overtone's careers page and normalized into a canonical job model.