Databricks Data Platform Engineer

OverviewJob Role: Databricks Data Platform EngineerJob Location: Abu Dhabi, UAEExperience: 5+ YearsRole Summary: Design and build scalable, high-performance, and governed data pipelines on Azure Databricks to support analytics, reporting, and enterprise data intelligence initiatives.ResponsibilitiesKey Responsibilities – Develop batch pipelines using Delta Lake with proper table design, partitioning, optimization, and lifecycle managementImplement incremental ingestion using Autoloader with schema evolution and checkpointingBuild Structured Streaming pipelines with watermarking, state management, and late data handlingDevelop declarative pipelines using Lakeflow standardsEnsure pipelines support idempotency, replayability, and safe backfillsOptimize Spark workloads (AQE, skew handling, shuffle tuning, join optimization)Build curated, analytics-ready datasets for SQL analytics and downstream consumptionDevelop DBSQL views aligned with enterprise semantic standardsImplement CI/CD deployments using Repos and asset-based packagingAdd monitoring, logging, and operational observability to pipelinesSecondary Responsibilities – Ensure compliance with Unity Catalog access controls and governance policiesEmbed data quality validations as per governance standardsSupport migration of legacy data workloads to Databricks including validation and performance tuningCollaborate with DataOps for deployment readiness and production stabilityQualificationsStrong hands-on experience with Azure Databricks, Spark, and PySparkDeep knowledge of Delta Lake, Autoloader, and Structured StreamingExperience with Lakeflow pipelines and DBSQL developmentExpertise in Spark performance tuning and optimization techniquesUnderstanding of data governance, access control, and data quality frameworksCI/CD and DevOps experience in data platform environmentsExperience supporting production-grade data platforms#J-18808-Ljbffr