Principal AI Engineer

About AlpheyaWe are a B2B WealthTech startup based in Abu Dhabi and backed by BNY Mellon (America’s oldest bank and first company to list on NYSE) and Lunate (a new $100B AUM alternative asset management firm based in Abu Dhabi, UAE). The company has raised $300M to build a state of the art wealth technology platform.Our mission is to power and grow our clients’ Wealth franchises through differentiated experiences, financial solutions, and insights. Our digital wealth management platform will enable banks and other financial institutions in the Middle East to grow and further penetrate affluent, HNW and UHNW investor segments.While still leveraging the capabilities and knowledge of large organizations, our fintech is a startup with truly cross‑functional and agile teams.For more information, please visit www.alpheya.comRole OverviewWe’re hiring a software engineer who builds production systems and who has spent the last few years applying that discipline to AI‑powered products.You will take validated AI prototypes and turn them into production‑grade software systems. You’ll focus on reliability, observability, maintainability, and clear architecture for AI‑powered features in a regulated environment.You will also have responsibility for leading and mentoring the group of data and software engineers to deliver reliably and raise the engineering bar.This is not a DevOps role. You will partner closely with our DevOps/SRE team (who owns core infrastructure, Kubernetes, and Terraform) to ensure AI services are operable and meet agreed SLAs.ResponsibilitiesProductionising AI Features (core focus)Own the AI API surface in production: contracts/schemas, versioning, backward compatibility, and behaviour guarantees for downstream consumersTake RAG/agent prototypes from notebook/PoC to production services: clean interfaces, robust runtime behavior, and safe rollout pathsImplement reliability patterns: timeouts, retries with backoff, idempotency, circuit breakers, rate limiting, graceful degradation, and fallbacksBuild observability end‑to‑end: structured logging, metrics, tracing (OpenTelemetry), and actionable dashboards/alertsOwn release quality: CI/CD for AI services, prompt/config versioning, regression tests, and staged deploymentsDrive operational readiness: runbooks, on‑call‑friendly diagnostics, incident retros, and continuous hardeningArchitecture & System Design (important gap to fill)Design and evolve AI API contracts (endpoints/tool contracts), ensuring safe, stable interfaces and clear ownership boundariesDesign service boundaries and interfaces for AI capabilities (APIs, contracts, and dependencies)Make pragmatic trade‑offs across latency, cost, quality, and compliance; document and communicate decisionsDefine patterns for state, memory, and persistence in agentic workflows (including partial failure handling and recovery)Establish integration patterns with existing platform services and data sources (without duplicating DevOps ownership)Data & Retrieval Systems (as used by product features)Build/operate ingestion and refresh pipelines that support product knowledge bases (freshness, lineage, auditability)Implement retrieval quality monitoring (e.g., drift, relevance), caching strategies, and evaluation harnessesPartner with data/analytics teams on data contracts, validation checks, and SLAsTeam Leadership & Engineering StandardsLead and develop a team of data and software engineers. Set direction, review work, unblock people.Run design reviews and code reviews that raise the bar without slowing deliveryEstablish shared patterns and standards for production AI systems that the team can scale onRaise the engineering bar: code reviews, design reviews, and shared standards for production AI systemsCollaborate across AI Product Engineering, Data Science, DevOps/SRE, Security, and Product to keep ownership boundaries cleanInnovation in AI SDLC & Product DeliveryOwn the evolution of our AI SDLC and AI stack: evaluate, pilot, and productionize tools/practices that measurably improve quality, reliability, delivery speed, latency, or cost (with clear success metrics and rollback paths), and enable innovation by AI product engineers/data scientists through reusable frameworks, templates, and paved pathsBring leading LLM engineering discipline into productionTranslate new capabilities (agents/tooling) into stable, well‑governed product APIs without compromising operability or complianceYou are a software engineer first.7+ years building production backend systems, with strong opinions about API design, error handling, testing, and operabilityProven ability to turn ambiguous prototypes into reliable services with clear operational characteristicsComfortable owning systems across the full lifecycle: design → build → launch → operateTypeScript or Python at a production level: you write services, not scripts. Clean abstractions, proper error handling, tested codeYou can lead engineers. You've mentored, set technical di