Conversational AI Analyst Fully Remote

About the jobMercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco our investors include Benchmark General Catalyst Peter Thiel Adam DAngelo Larry Summers and Jack Dorsey.PositionPosition: Language Model EvaluatorType: Full-time or Part-time Contract WorkCompensation: $23/hourLocation: Geography restricted to Egypt Saudi Arabia UAE USARole ResponsibilitiesEvaluate LLM-generated responses on their ability to effectively answer user queries.Conduct fact-checking using trusted public sources and external tools.Generate high-quality human evaluation data by annotating response strengths areas for improvement and factual inaccuracies.Assess reasoning quality clarity tone and completeness of responses.Ensure model responses align with expected conversational behavior and system guidelines.Apply consistent annotations by following clear taxonomies benchmarks and detailed evaluation guidelines.QualificationsMust-HaveBachelors degreeNative speaker or ILR 5/primary fluency (C2 on the CEFR scale) in ArabicSignificant experience using large language models (LLMs)Excellent writing skillsStrong attention to detailAdaptable and comfortable moving across topics domains and customer requirementsBackground or experience in domains requiring structured analytical thinkingExcellent college-level mathematics skillsPreferredPrior experience with RLHF model evaluation or data annotation workExperience writing or editing high-quality written contentExperience comparing multiple outputs and making fine-grained qualitative judgmentsFamiliarity with evaluation rubrics benchmarks or quality scoring systemsApplication Process (Takes 2030 mins to complete)Upload resumeAI interview based on your resumeSubmit formResources & SupportFor details about the interview process and platform information please check:For any help or support reach out to:PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.#J-18808-Ljbffr