Data Engineer

2 weeks ago

City of London, United Kingdom Hays Technology Full time

About the RoleWe are seeking a Databricks Data Engineer with strong expertise in designing and optimising large-scale data engineering solutions within the Databricks Data Intelligence Platform. This role is ideal for someone passionate about building high-performance data pipelines and ensuring robust data governance across modern cloud environments.
Key Responsibilities

Design, build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data.
Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies.
Conduct performance testing and cluster tuning to ensure cost-efficient, high-performing workloads.
Implement data quality, lineage tracking, and access control policies aligned with Databricks Unity Catalogue and governance best practices.
Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles.
Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management.
Integrate Databricks solutions with Azure services such as Azure Data Lake Storage, Key Vault, and Azure Functions.What We're Looking For

Proven experience with Databricks, PySpark, and Delta Lake.
Strong understanding of workflow orchestration, performance optimisation, and data governance.
Hands-on experience with Azure cloud services.
Ability to work in a fast-paced environment and deliver high-quality solutions.
SC Cleared candidates

If you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or call us now.
If this job isn't quite right for you, but you are looking for a new position, please contact us for a confidential discussion about your career.

Hays Specialist Recruitment Limited acts as an employment agency for permanent recruitment and employment business for the supply of temporary workers. By applying for this job you accept the T&C's, Privacy Policy and Disclaimers which can be found at (url removed)

  • Data Science and AI Engineering Manager

    2 weeks ago

    City of London, United Kingdom Data Idols Full time

    Data Science and AI Engineering Manager Salary: £100,000 - £125,000 + bonus Location: London / Hybrid Data Idols are working with a major UK brand embarking on an exciting transformation to embed AI at the heart of its technology strategy. This is a brand-new role, a chance to build and lead an AI Engineering function from the ground up within an...

  • Data Science and AI Engineering Manager

    3 weeks ago

    london (city of london), United Kingdom Data Idols Full time

    Data Science and AI Engineering ManagerSalary: £100,000 - £125,000 + bonus Location: London / Hybrid Data Idols are working with a major UK brand embarking on an exciting transformation to embed AI at the heart of its technology strategy. This is a brand-new role, a chance to build and lead an AI Engineering function from the ground up within an...

  • Data Engineer

    6 days ago

    London, United Kingdom Data Idols Full time

    **Data Engineer** Would you like to build enterprise scale data solutions for a global organisation that reaches 400 million users monthly? We are supporting an ambitious data-driven client in the world of Multimedia with their search for a Data Engineer to develop and maintainnew features of their data platform and end-to-end delivery of exciting data...

  • Data Engineer

    2 weeks ago

    London, United Kingdom Data Idols Full time

    **Data Engineer (Azure)** Data Idols are sourcing a Data Engineer for our tech enabled content client in the world of Media. Our client is rapidly scaling, and building a new data team focusing on creating and maintaining a new data environment from scratch - meaning no legacy systemsto deal with! This is a fantastic permanent Data Engineer opportunity,...

  • Data Science and AI Engineering Manager

    3 weeks ago

    City of London, United Kingdom Data Idols Full time

    Data Science and AI Engineering ManagerSalary: £100,000 - £125,000 + bonus Location: London / HybridData Idols are working with a major UK brand embarking on an exciting transformation to embed AI at the heart of its technology strategy. This is a brand-new role, a chance to build and lead an AI Engineering function from the ground up within an established...

  • Data Science and AI Engineering Manager

    2 weeks ago

    City Of London, United Kingdom Data Idols Full time

    Data Science and AI Engineering ManagerSalary: £100,000 - £125,000 + bonus Location: London / HybridData Idols are working with a major UK brand embarking on an exciting transformation to embed AI at the heart of its technology strategy. This is a brand-new role, a chance to build and lead an AI Engineering function from the ground up within an established...

  • Data Engineer

    2 days ago

    London, United Kingdom Data Idols Full time

    **Data Engineer** Data Idols are excited to be working with one of the top 100 companies in the world to recruit a Data Engineer! A company which owns multiple brands and who are investing heavily into 'all things data' to revolutionise their extraordinary team. **Skills and Experience**: - Experience with Azure - Experience with SQL - Experience with...

  • Data Architect

    6 days ago

    City of London, United Kingdom Data Idols Full time

    **Data Architect** Data Idols are working with one of the UK's biggest e-commerce organisations on a newly created data architect position that will drive their digital transformation. Home to some of the best-known fashion and homewares brands they have seen strong growthover the past 5 years and continue to learn more about their customers through...

  • Data Science and AI Engineering Manager

    3 weeks ago

    City of London, Greater London, United Kingdom Data Idols Full time

    Data Science and AI Engineering Manager Salary: £100,000 - £125,000 + bonus Location: London / Hybrid Data Idols are working with a major UK brand embarking on an exciting transformation to embed AI at the heart of its technology strategy. This is a brand-new role, a chance to build and lead an AI Engineering function from the ground up within an...

  • Data Engineer

    6 days ago

    London, United Kingdom Data Idols Full time

    **Data Engineer** Data Idols are proud to be working with a growing veterinary business who are extremely data driven and have a great mission to use data to improve the welfare of animals. They have a revolutionary data team who want to be recognised as the UK's leadingveterinary group and are home to some of the best clinicians world-wide; would you like...