Centific

Centific

Research Scientist, LLM Evaluation & Post-Training

Remote Work( USA) · Full-time

Sponsorship not specified$150k-$300kDetected 30 days ago
PythonMachine LearningTensorFlowPyTorchLLMsRAGStatisticsA/B TestingResearchExperimental DesignLeadershipCommunicationCollaborationPublic Speaking

About the role

  • This is a high-impact individual contributor and collaborative research role that sits at the intersection of applied ML research, enterprise AI product development, and customer-facing scientific consulting.

Responsibilities

  • Design experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes.
  • Evaluation Framework Development: Develop and validate comprehensive evaluation frameworks for LLM and multimodal systems, covering benchmark and task design, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness/stress testing.
  • Advanced Evaluation Research: Lead research on frontier evaluation domains including long-context, cross-modal, and dynamic multi-turn evaluations.
  • Cross-Functional Collaboration: Partner with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies, and with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines.
  • Expert-level benchmark dataset and test suite design for language and multimodal models
  • Deep understanding of metric design, scoring reliability, and measurement validity

Requirements

  • Research Experience: 5+ years of relevant experience in applied ML research or research science, with substantial work in LLMs or foundation models (graduate research counts).
  • Technical Proficiency: Strong Python coding skills for research experimentation, data processing, evaluation pipelines, statistical analysis, and visualization.
  • Hands-on experience with modern ML frameworks (PyTorch, Hugging Face, JAX/TensorFlow).
  • Evaluation Methodology: Ability to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability.
  • Experience designing reproducible evaluation studies across datasets and model versions.
  • Experience with human evaluation methods and quality assurance (rubric design, inter-rater reliability, adjudication frameworks)
  • Strong understanding of post-training techniques (SFT, RLHF, RLAIF, DPO, PPO, GRPO) and how training objectives interact with evaluation outcomes
  • Ability to reason about model behavior, failure modes, and performance tradeoffs across tasks and domains
  • Familiarity with alignment, safety, and robustness considerations in model evaluation
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.

Nice to have

  • Post-Training Practice: Hands-on experience running fine-tuning or post-training experiments (SFT, preference optimization, RLHF/RLAIF-style workflows).
  • Multimodal & Long-Context: Experience with multimodal evaluation (text-image, audio, video) and long-context benchmarking in real-world settings.
  • Scientific Contribution: Publications and/or open-source benchmark contributions in LLM evaluation, post-training, alignment, or related areas at top venues (NeurIPS, ICML, ICLR, ACL, EMNLP, etc.).
  • Applied Research Consulting: Experience in customer-facing applied research, technical consulting, or cross-functional product/research collaboration.
  • We consider qualified applicants regardless of criminal histories, consistent with legal requirements.

Skills

  • Ability to synthesize complex experimental findings into concise, actionable recommendations for engineering and business stakeholders
  • sampling, uncertainty quantification, significance testing, error analysis, metric interpretation
  • LLM Evaluation Expertise: Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
  • Agentic Evaluation: Experience designing multi-turn, interactive, or agentic evaluation protocols.
  • Safety & Governance: Familiarity with safety, trustworthiness, and governance considerations in GenAI evaluation.
  • Salary: $150K - $300K Annually
  • Centific is an equal-opportunity employer.
  • About Centific
  • Our zero-distance innovation™ solutions for GenAI can reduce GenAI costs by up to 80% and bring solutions to market 50% faster.
  • Research Scientist, LLM Evaluation & Post-Training

Compensation

  • Salary: $150K - $300K Annually

Company info

  • Type: Full-time
  • Role Overview
  • As a Research Scientist, LLM Evaluation & Post-Training, you will be at the frontier of how evaluation design, measurement strategy, and feedback signals drive model improvement across Centific's AI platform products.
  • You will lead research programs that define next-generation evaluation-driven post-training workflows, develop rigorous benchmark frameworks, and partner directly with leading AI organizations to deliver credible, actionable model improvement insights.
  • This role offers the opportunity to shape Centific's internal research agenda, build reusable scientific assets, and publish at top-tier venues.
  • Research Agenda & Experimentation: Define and execute a rigorous research agenda focused on LLM evaluation and post-training, with emphasis on evaluation-driven model improvement. Design experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes.
  • Advanced Evaluation Research: Lead research on frontier evaluation domains including long-context, cross-modal, and dynamic multi-turn evaluations. Study effectiveness and limitations of existing techniques and propose improved methodologies with clear validity and scalability tradeoffs.
  • Model Behavior Analysis: Analyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesign. Translate findings into practical improvements for customer solutions and Centific's internal platforms.
  • Customer Engagement: Engage with customer technical stakeholders at leading AI organizations to understand evaluation goals, review methodologies, and provide expert scientific recommendations. Serve as a credible technical peer to research and engineering leaders.
  • Knowledge & IP Creation: Contribute to internal benchmark datasets, reusable evaluation frameworks, and research assets. Produce high-quality technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations.
  • Thought Leadership: Contribute to Centific's position as a leader in LLM evaluation and post-training through publications, conference presentations, and open-source contributions.
  • Core Technical Competencies
  • You will provide technical depth and leadership across the following domains:
  • Evaluation Science & Benchmarking
  • LLM & Post-Training Methods
  • Quantitative Analysis & Scientific Rigor
  • Strong statistical analysis

Visa & Work Authorization

  • pplicants will receive consideration for employment without regard to race, color, religion, national origin, ancestry, citizenship status, age, mental or physical disability, medical condition, sex (including pregnancy)
Apply directly at Centific →Create a free account for alerts like thisView Centific immigration profile

This listing is sourced directly from Centific's careers page and normalized into a canonical job model.