College Board

College Board

Natural Language Measurement Specialist

Remote - USA · Full-time

Sponsorship not specified$88k-$145kDetected 86 days ago
PythonMachine LearningPyTorchPandasNumPyData EngineeringData ScienceNLPLLMsStatisticsResearchExperimental DesignLeadershipCommunicationCollaborationMentoring

Stay score

odds of building a lasting career here

88Lottery-free
Cap-exempt (no lottery)100
Sponsors year-round90
Entry-level history60
PERM / green-card track50
Fits your clock70

Files H-1B any time — no cap, no lottery. The 2026 weighted-selection rule doesn't touch it, which makes this the strongest structural path for entry-level talent. Pay is often lower, but your odds of staying are dramatically higher.

Lottery odds assume a STEM candidate.

Personalize to your clock →

Employer immigration record

from this employer's Department of Labor filings

Green-card filing pattern in this occupation

Context, not a finding about this posting: of this employer's 3 green-card filings in this occupation, 100% were for a worker who already held the job.

Sourced from Department of Labor LCA, PERM and prevailing-wage disclosure data. Employer matching is by name, so figures may be split across an employer's legal entities. Absence of a filing means none appears in our copy of the data, not that none exists.

Community outcomes

No reports yet — be the first to help the next applicant.

About the role

  • We operate at the intersection of educational measurement, data science, and emerging AI technologies.
  • Our work spans the measurement of language-based constructs, the development of large language model (LLM) systems for feedback generation and annotation, and research to ensure the validity, fairness, and reliability of our systems.
  • We are a collaborative, mission-driven team that values both psychometric rigor and technical expertise.

Responsibilities

  • Design and lead validity studies, including human-machine agreement, rater comparison, and fairness analyses across subgroups
  • Develop and apply methods for detecting and mitigating bias in language-based scores
  • Contribute to prompt design, fine-tuning, and evaluation of LLM-based feedback and annotation systems
  • Build evaluation frameworks that connect model behavior to measurement outcomes
  • Collaborate with senior team members to translate measurement findings into production systems
  • Implement high-quality, maintainable code for model development and evaluation
  • Lead and contribute to research studies that evaluate model performance and support assessment validity
  • Document methodologies and findings in a clear and rigorous manner
  • At College Board, we offer more than a paycheck- we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive.

Requirements

  • All CB employees are required to occasionally travel to meet in person for business purposes.
  • familiarity with IRT, generalizability theory, or related frameworks is highly desirable
  • Experience analyzing language or text data, with exposure to NLP or large language models
  • Demonstrated ability to conduct rigorous research (e.g., thesis, publications, or applied research projects)
  • Familiarity with working with structured and unstructured data
  • The ability to travel 5-10 times a year to College Board offices or on behalf of College Board business.

Compensation

  • We're a self-sustaining nonprofit that believes in fair and competitive compensation grounded in your qualifications, experience, impact, and the market.

Benefits

  • You'll have open, transparent conversations about compensation, benefits, and what it's like to work at College Board throughout your hiring process.
  • Develop and refine machine learning models for measuring language-based constructs

Company info

  • The Automated Scoring team provides critical insights and tools to support the design, delivery, and continuous improvement of digital assessments.
  • We combine modern machine learning approaches, including large language models, with strong measurement principles to create scalable, trustworthy solutions that expand opportunity for students.
  • About the Opportunity
  • As a Natural Language Specialist, you will help define and advance how language-based performance is measured in high-stakes educational settings.
  • This role sits at the intersection of natural language processing, large language models, and educational measurement, and is ideal for someone who pairs strong measurement training with a working knowledge of modern AI.
  • You will translate measurement constructs into LLM-based feedback and annotation systems, design and conduct the studies that establish their validity, fairness, and reliability, and ensure that what our models produce holds up to rigorous psychometric standards.
  • You will s erve as the measurement authority for cross-functional partners-psychometricians, engineers, and data scientists-shaping how language-based constructs are defined, evaluated, and applied across the team's portfolio of systems.
  • Y our primary lens will be measurement:
  • defining what feedback and annotations mean, evidencing that they mean it, and improving them over time.
Apply directly at College Board →Create a free account for alerts like thisView College Board immigration profile

This listing is sourced directly from College Board's careers page and normalized into a canonical job model.