AI Evaluation Systems Engineer

About the Role

GoMeasure Platform runs goTalentOS — a specialised marketplace for AI workers, where engineers and domain experts don't only review and improve AI systems but also build and deploy them. We're looking for a AI Evaluation Systems Engineer to join this network on a flexible, contract basis. Build the evaluation machinery — harnesses, ground truth and judge pipelines — rather than performing the reviews by hand. You'll work remotely, pick up engagements matched to your expertise, and be paid hourly for work that ships.

Key Responsibilities

• Design eval harnesses and ground-truth datasets for production AI systems.
• Build LLM-as-judge pipelines and validate them against human agreement.
• Wire evaluation into CI so regressions are caught before release.
• Own production telemetry that tracks quality after deployment.
• Turn reviewer disagreement into sharper rubrics and better ground truth.

Required Qualifications

Domain Expertise

• Built evaluation infrastructure, not only conducted evaluations.
• Understands inter-rater reliability and how judge models fail.
• Comfortable in the codebase — this is an engineering role.

Ways of Working

• Comfortable working independently and asynchronously in a remote, engagement-based setup.
• Strong written communication — decisions and trade-offs are recorded, not just made.
• Able to hand work over cleanly: another engineer can pick it up from your documentation.

How It Works

• Fully remote, contract engagements sized to your availability.
• Paid hourly at a rate set by role, seniority and verified skill level.
• Your skills are verified once through GoMeasure and reused across engagements.