Jay Analytix
QA Lead — AI Systems & Models Testing
Montreal, QC · Contract
Sponsorship not specifiedDetected 70 days ago
Vector DatabasesAWSGCPAzureCloud PlatformsCI/CDMachine LearningData EngineeringLLMsRAGLangGraphAI OrchestrationLeadershipCommunicationPipeline Integrity
About the role
- QA Lead — AI Systems & Models Testing Quality Assurance • Artificial Intelligence • Contract Position Contract Montreal, QC AI / ML Testing LLM / RAG / LangChain ABOUT THE ROLE We are seeking an experienced QA Lead with deep expertise in AI systems testing to join our team on a contract basis in Montreal, Québec. This role sits at the intersection of
- quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks. You will own the end-to-end test strategy for complex AI products and help define quality standards in a rapidly evolving space. MUST-HAVE SKILLS Proven QA leadership experience designing
Responsibilities
- Lead design and execution of comprehensive test strategies across AI systems, including prompt evaluation, output quality assessment, and bias/safety analysis.
- Develop and maintain test harnesses for LangChain and LangGraph-based applications; review chain and graph construction code to proactively surface integration risks.
- Collaborate with ML engineers, data scientists, and product teams to embed quality practices throughout the AI development lifecycle.
- Document test findings clearly for both technical and non-technical stakeholders.
Requirements
- This role sits at the intersection of quality engineering and artificial intelligence, requiring hands-on proficiency in LLM behavior analysis, RAG pipeline validation, and modern AI orchestration frameworks.
- MUST-HAVE SKILLS Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.
- Hands-on experience with prompt engineering - constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.
- Experience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.
Skills
- Exposure to performance and scalability testing of vector databases under high load.
Benefits
- Validate RAG pipeline integrity - data ingestion, chunking, retrieval accuracy, and embedding consistency - and define edge-case coverage for vector database interactions.
Apply directly at Jay Analytix →Create a free account for alerts like thisView Jay Analytix immigration profile
This listing is sourced directly from Jay Analytix's careers page and normalized into a canonical job model.