TL;DR
Lead Data Engineer (Databricks): Shaping and delivering an organization's modern Enterprise Data Platform, ensuring consistent engineering standards and guiding the design of scalable data solutions with an accent on technical leadership and hands-on engineering. Focus on high-performance data platforms at scale, mentoring engineers, and influencing the direction of the data platform.
Location: Remote UK or hybrid / onsite at client offices across England
Company
ARCA Resourcing is partnered with an innovative and leading retailer that is investing heavily in its data capabilities to drive smarter decision-making across the business.
What you will do
- Provide technical leadership across multiple delivery squads, ensuring consistent data engineering standards.
- Coordinate technical solutions and align team efforts across initiatives.
- Design and implement high-performance ELT pipeline patterns using Databricks.
- Translate functional designs into scalable and reusable data engineering solutions.
- Champion engineering best practices, including code quality, testing, and documentation.
- Mentor and develop engineers across the team, creating learning and development plans.
Requirements
- Strong experience with Databricks, Apache Spark, and cloud-based data architectures (preferably Azure).
- Excellent programming capability in Python and SQL.
- Experience orchestrating pipelines with Azure Data Factory and Databricks Workflows.
- Solid understanding of Lakehouse architectures and modern data warehousing principles.
- Experience with data modelling, ELT/ETL pipelines, and testing frameworks.
- Proven experience leading engineering teams or coordinating multiple squads.
Nice to have
- Degree in IT, Computer Science, or related discipline.
Culture & Benefits
- Opportunity to work with cutting-edge technologies.
- Mentor engineers and influence the direction of a high-impact data platform.