Conversational AI Analyst Fully Remote

<div><h3>About the job</h3><p>Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include<b>Benchmark</b><b>General Catalyst</b><b>Peter Thiel</b><b>Adam DAngelo</b><b>Larry Summers</b>and<b>Jack Dorsey</b>.</p><h3>Position</h3><p><b>Position:</b>Language Model Evaluator<br/><b>Type:</b><b>Full-time or Part-time Contract Work</b><br/><b>Compensation:</b><b>$23/hour</b><br/><b>Location:</b><b>Geography restricted to Egypt Saudi Arabia UAE USA</b></p><h3>Role Responsibilities</h3><ul><li>Evaluate<b>LLM-generated responses</b>on their ability to effectively answer user queries.</li><li>Conduct fact-checking using trusted public sources and external<b>tools</b>.</li><li>Generate high-quality human evaluation data by annotating response strengths areas for improvement and factual inaccuracies.</li><li>Assess reasoning quality clarity tone and completeness of responses.</li><li>Ensure model responses align with expected conversational behavior and system guidelines.</li><li>Apply consistent annotations by following clear taxonomies benchmarks and detailed evaluation guidelines.</li></ul><h3>Qualifications</h3><h3>Must-Have</h3><ul><li><b>Bachelors degree</b></li><li><b>Native speaker</b>or<b>ILR 5/primary fluency (C2 on the CEFR scale)</b>in<b>Arabic</b></li><li><b>Significant experience using large language models</b>(LLMs)</li><li><b>Excellent writing skills</b></li><li><b>Strong attention to detail</b></li><li><b>Adaptable</b>and comfortable moving across topics domains and customer requirements</li><li>Background or experience in domains requiring<b>structured analytical thinking</b></li><li><b>Excellent college-level mathematics skills</b></li></ul><h3>Preferred</h3><ul><li>Prior experience with<b>RLHF model evaluation or data annotation work</b></li><li>Experience writing or editing<b>high-quality written content</b></li><li>Experience comparing multiple outputs and making<b>fine-grained qualitative judgments</b></li><li><b>Familiarity with evaluation rubrics</b>benchmarks or quality scoring systems</li></ul><h3>Application Process (Takes 2030 mins to complete)</h3><ul><li>Upload resume</li><li>AI interview based on your resume</li><li>Submit form</li></ul><h3>Resources&Support</h3><ul><li>For details about the interview process and platform information please check:</li><li>For any help or support reach out to:</li></ul><p>PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.</p></div>#J-18808-Ljbffr