AI Evaluation Analyst

Role Overview

Help advance frontier language models by creating rigorous training and evaluation materials that shape how AI systems learn, reason, and respond. This remote contract role centers on clear writing, multi turn conversation design, and detailed analysis of model behavior. AI experience is not required, but strong domain knowledge and careful execution are essential.

Key Responsibilities

• Create detailed, task based multi turn conversations and rubrics that follow project specifications.
• Test and refine conversation drafts with frontier large language models to meet quality and difficulty requirements.
• Deliver complete evaluation assets, including transcripts, target behaviors, binary rubrics, and supporting evidence.
• Maintain strict adherence to evolving specifications while producing work accurately and efficiently.
• Validate and calibrate deliverables with team leads and quality reviewers as guidance changes.
• Work independently and consistently to meet expected output rates and deliverable requirements.

Qualifications

• Native level written English, with exceptional clarity, organization, and attention to detail.
• Working knowledge of frontier LLM behavior and common model failure patterns.
• Ability to interpret and apply highly detailed specifications independently.
• Strong critical thinking and analytical skills, particularly in writing intensive or analysis intensive work.
• Experience with data annotation, RLHF, SFT, AI evaluation, or prompt engineering is preferred.
• Experience creating evaluation items or rubrics, or conducting deep analysis of technology outputs, is preferred.
• A background in research, editorial work, technical writing, or quality assurance is a plus.

Work Terms

• Remote contract engagement.
• Minimum weekly task submission requirements apply.
• Selected candidates should be ready to begin their first tasks within 24 to 48 hours after completing the required onboarding process.

Compensation

• Advertised pay range: $20 to $30 per hour.
• Compensation is paid per task that meets project specifications. Completion time may vary based on experience and workflow.