LawZero
Director, Evaluations
Montreal · Director
Sponsorship not specifiedDetected 57 days ago
Machine LearningLLMsAgentic AIResearchLeadershipCommunication
About the role
- LawZero is a non-profit building safe-by-design AI systems. We're building the Scientist AI, an advanced AI system designed from the ground up to be both highly capable and safe. As we develop both general‑purpose Scientist AI models and safety guardrails for frontier LLMs, we need rigorous, independent evaluation of every capability and safety claim we
- make. We are looking for a Director of Evaluations to build, lead, and grow LawZero's Evaluations Team.
Responsibilities
- Build up the Evaluations Team during your first 3-6 months, scaling to roughly 8-10 people across research, engineering, dataset and benchmark design, and red‑teaming.
- Oversee the design and construction of new datasets, tasks, and virtual or interactive environments to measure performance of the Scientist AI across capabilities, safety (including honesty and goal-directedness), explainability, causal mechanisms and detecting adversarial attacks.
- Lead evaluation of the Scientist AI when deployed as a guardrail around frontier models, including its ability to comply with harm specifications, detect and block harmful responses, explain its decisions, and resist adversarial attacks such as jailbreaks, prompt injection, and data poisoning.
- Establish and lead our automated and manual red‑teaming programmes, both in‑house and in partnership with external providers, to stress test the Scientist AI as a general‑purpose model and as a guardrail.
- Lead the construction of internal tooling and infrastructure needed to run evaluations at scale, automating and standardizing the pipeline wherever possible.
- As needed and where possible, directly support research and product streams with their own internal requirements w.r.t. evaluations and benchmarking to unblock and speed up.
- Own LawZero's public communication of evaluation results, including model and system cards, technical reports, peer‑reviewed publications and blog posts, to build trust with the wider AI safety community.
- A track record of building evaluation datasets, benchmarks, or interactive environments from scratch, including for safety‑relevant such as honesty, sycophancy, refusal behaviour, and adversarial robustness.
- LawZero is a non-profit organization committed to advancing research and creating technical solutions that enable safe-by-design AI systems.
Requirements
- Hands‑on expertise in designing and running large‑scale evaluations of LLMs or other frontier ML systems across capabilities, safety, and adversarial robustness.
- Strong written and verbal communication skills, including the ability to translate technical results for non‑technical audiences such as executives, funders, and policymakers.
Nice to have
- The opportunity to contribute to a unique mission with a major impact
- A team of passionate experts in their field
- A collaborative and inclusive work environment with offices in the heart of Little Italy, in the trendy Mile-Ex district, close to public transportation.
- Its scientific direction is based on new research and methods proposed by Professor Yoshua Bengio, the most cited AI researcher in the world.
- Such AI systems could be used to accelerate scientific discovery, to provide oversight for agentic AI systems, and to advance the understanding of AI risks and how to avoid them.
- LawZero believes that AI should be cultivated as a global public good-developed and used safely towards human flourishing.
- For more information, visit www.lawzero.org
- At LawZero, diversity is important to us.
Skills
- An advanced degree (MSc or higher) in machine learning, computer science, or a closely related field.
Compensation
- A minimum of 20 days vacation per year upon start
Benefits
- Comprehensive health benefits
- A minimum of 20 days vacation per year upon start
- A minimum retirement savings employer contribution of 4%
- Generous flexible benefits designed to contribute to your well-being
Company info
- We welcome applications from highly qualified individuals interested in working towards our mission in a respectful, inclusive and collaborative setting.
Apply directly at LawZero →Create a free account for alerts like thisView LawZero immigration profile
This listing is sourced directly from LawZero's careers page and normalized into a canonical job model.