AI Evaluation Rubric Auditor
Flash
$60 – $85/hour · as listed
AI Evaluation Rubric Auditor
Flash is seeking an intermediate-level AI data specialist to audit evaluation rubrics used in AI agent testing. You'll review completed tasks where AI systems are assessed on realistic shopping and delivery scenarios, examining whether grading rubrics fairly capture task performance. Your responsibilities include verifying that rubric assessments are accurate, checking if the information needed was accessible to the AI, and confirming that grading criteria align with the original prompt requirements. This AI training work requires careful attention to detail and sound judgment about evaluation fairness.
This remote role suits professionals with experience in data analysis, quality assurance, or similar verification work who can evaluate complex AI tasks methodically. You'll work approximately 15 hours per week at an intermediate skill level.
Compensation is listed at $60–$85 per hour on an hourly basis through the Mercor marketplace.
From the listing: Mercor marketplace · hourly · 15h/week. The work We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair: \- Was the information available? \- Does the rubric accurately grade what the prompt asked for? \- Would a valid answer have
Applications are completed on the listing’s own site. Pay is shown as posted by the source (“as listed”) or the aggregator’s estimate where marked — offers and availability change, and individual results vary.
Want help getting selected — and finding the best-paid work for your country? See how membership works →