Repository Benchmark Integrity Reviewer
Job Description
IXO is engaging specialists to evaluate repository-based software benchmarks that measure the intended engineering skill. Your contribution is technical work: make the reasoning inspectable, identify substantive errors and provide evidence that supports a reliable assessment.
Work you will do
• Audit issue specifications, code context and tests to determine whether a task is realistic, solvable and reproducible.
• Check container setup, validation behavior and the risk that leakage or shortcuts undermine the benchmark.
• Document defects in the task and explain the changes needed for a fair assessment of alternative valid solutions.
Required background and routes
• Three or more professional software-engineering years and genuine open-source work demonstrated by merged changes or maintainer responsibilities.
• Read reference patches, test runners and Docker isolation critically; detect leaked answers and exploitable grading logic. Use Python plus at least one of C++, TypeScript, Go or Java.
Preferred background
• SWE-Bench/Verified or similar benchmarks; maintenance of major Python projects such as pytest, sympy, scikit-learn, Flask or Django; previous technical reviewing or grading.
Deliverables
Submit the completed technical artifact or assessment with its supporting evidence, explicit assumptions, reproducible checks where applicable, and concise reasons for each material judgment. Address review findings within the agreed scope.
Location and schedule
Regional eligibility: United States. Remote assignments are scheduled by agreement, with no guaranteed weekly volume. Availability planning can include 40 hours per week depending on the track. IXO confirms the applicable timing before work.
Pay and working terms
$75 • $95/hr USD. The agreed hourly rate, scope, schedule and acceptance criteria are confirmed before work starts. Applying does not guarantee an assignment. Use public, licensed or otherwise authorized material only; do not submit confidential employer information, personal data or restricted research.