Innodata Inc.
QA / Evaluation Lead
Hybrid - Washington D.C
No sponsorship$45k-$50kDetected 12 days ago
TypeScriptPythonDatabricksMachine LearningData EngineeringData ScienceLLMsStatisticsProject Management
About the role
- It's a role for someone who thinks rigorously about evaluation and takes pride in evidence-backed quality.
Responsibilities
- Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus - metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
- Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
- Validate quality scoring and IAA computation within the Innodata data layer
- Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation - define what 'good enough' looks like quantitatively
- QC process design: sampling methodology, escalation workflows, adjudication protocols
Requirements
- 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
- IAA methodology expertise - Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
- Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
- IAA methodology expertise -
Nice to have
- Prior DoD or IC data quality program experience
- CVAT or equivalent annotation platform QC workflow configuration
- Drift detection and model monitoring methodology
- Experience with FMV / video annotation quality standards
- Master's degree preferred.
- Equivalent experience may substitute for degree on a 2-for-1 basis.
- Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required
Skills
- Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA
- Evaluation framework design for AI/ML training data programs
- Please be aware of recruitment scams involving individuals or organizations falsely claiming to represent employers.
- Innodata will never ask for payment, banking details, or sensitive personal information during the application process.
Compensation
- The expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications.
Apply directly at Innodata Inc. →Create a free account for alerts like thisView Innodata Inc. immigration profile
This listing is sourced directly from Innodata Inc.'s careers page and normalized into a canonical job model.