Databricks Platform Engineer

Seeking a highly skilled and motivated Data Platform Engineer to join our client's core platform team. This role focuses on building advanced platform capabilities, optimizing and maintaining robust data infrastructure, and enabling data-driven decision-making across the organization.The ideal candidate has deep expertise in Databricks, PySpark, and Python, and demonstrates strong communication skills - articulate, precise, and concise.Key Responsibilities:Collaborate with stakeholders during requirements clarification and sprint planning to ensure alignment with business objectivesDesign and implement technical solutions on the Lakehouse platform (Databricks), including:Prototyping new Databricks capabilitiesExposing these capabilities to support Data Products strategy and Data & AI ecosystemIntegrate data platforms with enterprise tools, including:Incident and monitoring systems (e.g., ServiceNow)Identity management solutionsData observability tools (e.g., Dynatrace)Develop and maintain unit and integration tests to ensure quality and resilienceSupport QA teams during acceptance testingAct as a third-line engineer for production incidents, ensuring system stability and uptimeCollaborate with cloud and infrastructure teams to deliver secure, scalable, and reliable solutionsRequired Skills & Experience:ExperienceMinimum 8 years in IT/Data-related rolesTechnical ExpertiseExpert knowledge of DatabricksProficient in PySpark for distributed computingStrong Python skills for library developmentAdvanced SQL skills for complex query optimization (e.g., Oracle, MS SQL)Experience with Git for version controlFamiliarity with monitoring tools (e.g., ServiceNow, Prometheus, Grafana)Knowledge of scheduling tools (e.g., Stonebranch, Control-M, Airflow)Proficiency in data quality frameworks (e.g., Great Expectations, ideally Monte Carlo)Solid understanding of cloud infrastructure fundamentals (DNS, certificates, identity, load balancing)Note: Primary focus is building platform capabilities rather than writing ETL pipelinesComfortable working in Agile environments (sprint planning, stand‑ups, retrospectives)Skilled in Azure DevOps for project managementProblem-Solving & CommunicationStrong debugging and troubleshooting skills for complex data engineering issuesExceptional written and verbal communication skillsAbility to explain technical concepts to non-technical stakeholdersPreferred ExperienceUnderstanding of the investment data domain#J-18808-Ljbffr