Experienced PsychometricianWe're looking for an experienced psychometrician to serve as the measurement science backbone of our certification programs. In this role, you'll ensure that every assessment we build is valid, reliable, fair, and defensible at global scale. You'll work at the intersection of workforce capability, assessment design, and data-driven quality assurance, partnering closely with subject matter experts, instructional designers, and program managers to translate complex technical competencies into rigorous, scalable assessments.

In your first 90 days, you'll complete a psychometric audit of existing assessments, establish quality benchmarks, and begin embedding rigorous measurement practices into our item development and review workflows. Within six months, you'll be the go-to expert on assessment validity and reliability across the program and your fingerprints will be on every certification we launch. This is a high-impact individual contributor role with significant influence over program strategy and quality standards.

Key job

responsibilities: Lead the psychometric design and validation of certification assessments across role-based and specialty-based programs, including written, practical, and performance-based formats

Design and audit scalable job task analyses (JTAs) in partnership with subject matter experts and operations stakeholders to establish defensible content blueprints

Apply classical test theory (CTT) and item response theory (IRT) to evaluate item and test quality; conduct standard-setting studies (e.g., Angoff, Bookmark) to establish and validate cut scores

Develop and govern item development, review, and validation workflows across global content contributors, ensuring consistency and rigor

Analyze assessment data to identify bias, differential item functioning (DIF), and reliability issues; recommend and implement remediation strategies

Maintain independence between training and certification development to protect assessment integrity

Build and maintain psychometric reporting frameworks that surface actionable insights on pass rates, item performance, and certification quality

Partner with third-party assessment vendors on test development, delivery, and quality assurance; hold vendors accountable to psychometric standards

Communicate complex psychometric concepts clearly to non-technical stakeholders, including senior leadership

A day in the life:

Your morning starts with a fresh cup of coffee and a dataset that's already asking questions. You're reviewing item-level statistics from a recent pilot, flagging a handful of underperforming items. Mid-morning, you're deep in a standard-setting session with SMEs, guiding them through an Angoff exercise with patience and precision. After lunch, you're presenting DIF analysis results to the program team translating p-values into plain language without losing the rigor. You close the day refining the content blueprint for an upcoming JTA. It's measurement science at its most applied. You love it.

About the

team: The DC Certifications team is redefining how work readiness is measured across AWS data centers. Rather than tracking training completion, we deliver outcome-based certifications that validate whether a Builder can confidently do the job. Our program assesses competencies tied to specific data center tasks, using practical and simulated environments to maintain high, consistent standards. We automate assessment delivery to keep the process simple, while ensuring independence and rigor through qualified assessors. Behind the scenes, we provide the tooling and data solutions that help sites plan and manage their workforce's capabilities.