← Work Board

AI Evaluation Rubric Reviewer

Flash

AI Data Intermediate AI training platform Remote

$40 – $75/hour · as listed

AI Evaluation Rubric Reviewer

Flash is seeking intermediate-level AI data specialists to evaluate AI agent performance on real-world scenarios. You'll review AI-generated evaluation rubrics and model responses in a shopping and delivery context, assessing whether failures are legitimate or caused by flawed grading criteria. Your responsibilities include verifying information availability in prompts, checking rubric accuracy against task requirements, identifying cases where valid answers were incorrectly marked, and ensuring rubric structure meets system standards. This work suits those with coding domain knowledge who can think critically about how AI systems are tested and evaluated.

The role is remote and offered at $40–$75 per hour on an hourly basis, with approximately 15 hours per week available. This is platform-based work through Mercor, ideal for those looking to contribute to AI training and evaluation with flexible scheduling.

From the listing: Mercor marketplace · hourly · 15h/week. The work We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair: \- Was the information available? \- Does the rubric accurately grade what the prompt asked for? \- Would a valid answer have been marked wrong? \- Is the rubric structure correct per the gui

Open listing & apply →

Applications are completed on the listing’s own site. Pay is shown as posted by the source (“as listed”) or the aggregator’s estimate where marked — offers and availability change, and individual results vary.

Want help getting selected — and finding the best-paid work for your country? See how membership works →