Lead Data Engineer

2 months ago

London, Greater London, United Kingdom Competition and Markets Authority Full time


Are you eager to leverage your technical expertise for real-world impact? If so, this is an exciting opportunity to use and enhance your data engineering skills.

Join us and play a key role in shaping the digital economy and make a positive difference to society.

About the CMA

We help people, businesses, and the UK economy by promoting competitive markets and tackling unfair behaviour. Our work is wide ranging, ambitious and often new and challenging.

The role


The Data, Technology and Analytics (DaTA) unit is an established multidisciplinary team that brings together data engineers, data scientists, technologists, behavioural scientists, and other experts.

You will join a team of about 20 Data Scientists and Engineers within the wider unit.

As a Lead Data Engineer, you will be involved in designing and managing our analytical infrastructure and supporting the delivery of our most technically challenging projects.

On one project, you may engage with tech giants like Google, Amazon, and Facebook and help to navigate vast data landscapes and understand how to regulate these digital powerhouses, and in another, you may develop a custom solution in AWS for data scientists to easily run ad-hoc scheduled jobs, or experiment with LLMs (AWS Bedrock and open-source models) and RAG systems as part of our digital transformation strategy.


In this pivotal role, you'll be at the forefront of developing data-driven tools and data pipelines that enable the CMA to analyse and regulate these influential platform businesses, and others.

You'll build robust data infrastructure to support critical cases and market studies, putting machine learning models in production that detect anti-competitive behaviours, process the news and consumer complaints for our intelligence pipelines, and generally contribute to foster fair competition and protect consumers.


As a Lead Data Engineer, you'll play a vital role in crafting solutions to these issues, ensuring digital markets remain competitive and beneficial for consumers.

About you

We're seeking seasoned professionals with a proven track record in data engineering, DevOps, or infrastructure roles. Your ability to lead technical projects, coupled with expertise in Python and cloud development (preferably AWS), will be essential.

You will collaborate with stakeholders across various disciplines, translating business requirements into complex architectural designs and solutions, and drive the development of our custom, cloud-hosted analytics platform, and related infrastructure.

What we can offer you


This is an opportunity to join a diverse, dynamic, supportive team working across a range of high-profile digital cases and interesting technologies.

We champion a culture of respect, provide training options tailored to maximise your potential, and offers genuine opportunities for career progression, and gaining a unique range of experience.

To find out more about this opportunity and about working for the CMA, please click on the APPLY button.

The closing date for applications is 11.55pm on 7th August 2024.

The CMA, like many other organisations, operate a hybrid working model with a mix of office and home working.

We believe this supports us to deliver our best work while allowing the flexibility to maintain balanced and healthy lives.


The CMA are an accredited Disability Confident Employer, and we are an equal opportunities employer, welcoming suitably qualified applicants from all backgrounds.


We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles.

  • Data Engineering Delivery Lead

    1 week ago

    London, Greater London, United Kingdom NTT DATA Full time

    Lead Data Engineering Projects at a Global IT LeaderNTT DATA, a world-renowned IT service provider, is seeking a skilled and experienced Data Engineering Delivery Manager to join our dynamic team. As the 6th largest IT Service Provider globally, we are dedicated to driving business change and technology transformation across diverse industries.Your Role:In...

  • Lead Data Center Design

    1 week ago

    London, Greater London, United Kingdom Vantage Data Centers Full time

    Vantage Data Centers is seeking a seasoned Senior Design Manager to spearhead design and engineering initiatives within our London market. This role demands a strategic leader with extensive data center design management experience, capable of driving excellence in complex projects. As the heart of our design process, you will oversee the entire lifecycle,...

  • Lead Data Architect for Cloud-Based Solutions

    1 week ago

    London, Greater London, United Kingdom NTT DATA Full time

    NTT DATA is seeking a Lead Data Solutions Architect to join our dynamic, remote-first team. In this role, you will play a pivotal part in designing and delivering cutting-edge cloud-based data solutions for prestigious clients across diverse industries. You'll lead cross-functional teams of skilled professionals, including Data Engineers, Architects,...

  • Data Center Operations Lead

    1 week ago

    London, Greater London, United Kingdom Amazon Data Services UK Limited Full time

    Amazon Data Services UK Limited is seeking a highly motivated and experienced Data Center Operations Manager to lead our team in London Heathrow (LHR). This role offers the unique opportunity to oversee the management of one of the world's largest cloud computing infrastructures, driving operational excellence across all aspects of data center...

  • Data Center Operations Lead

    1 week ago

    London, Greater London, United Kingdom Amazon Data Services UK Limited Full time

    About Amazon Data Services UK Limited Amazon Data Services UK Limited is a leading provider of cloud computing services, dedicated to empowering businesses with innovative and reliable solutions. Job Description: Chief Engineer We are seeking a highly skilled and experienced Chief Engineer to oversee the operation and maintenance of our critical data...

  • Data Engineering Lead

    1 week ago

    London, Greater London, United Kingdom With Intelligence Full time

    About With IntelligenceWith Intelligence has evolved from a traditional financial publisher into a dynamic, product-led fintech company. Our mission is to empower investors and managers worldwide by connecting them to the people and data they need to raise and allocate assets efficiently.We have recently secured a new round of funding from a prominent...

  • Data Center Infrastructure Lead

    1 week ago

    London, Greater London, United Kingdom Amazon Data Services UK Limited Full time

    Amazon Data Services UK Limited is seeking a highly motivated and experienced Data Center Facilities Manager to join our growing team.As a key member of our infrastructure operations, you will play a vital role in ensuring the smooth and efficient operation of our world-class data center facilities. Your responsibilities will encompass a wide range of tasks,...

  • Lead Data Engineer

    1 week ago

    London, Greater London, United Kingdom Group Support Services Full time

    About Group Support Services Group Support Services is dedicated to providing essential infrastructure and expertise to empower our diverse range of business units. We foster a collaborative environment where innovation thrives, and we strive to deliver exceptional support that drives organizational success. About the Team The Data and Platform Engineering...

  • Staff Software Development Engineer

    2 days ago

    London, Greater London, United Kingdom PitchBook Data Full time

    About the RolePitchBook Data is a leading provider of financial data and software solutions. We are seeking a highly skilled and experienced Staff Software Development Engineer to join our Product and Engineering team.Key ResponsibilitiesArchitect, design, develop, and maintain components of PitchBook's services, including backend with aspects of frontend...

  • Lead Data Engineer

    1 week ago

    London, Greater London, United Kingdom Performics Full time

    Senior Data Engineer at PerformicsWe are seeking a highly experienced Senior Data Engineer to play a pivotal role in leading our Product Development team. You will collaborate closely with the Scrum Master, guiding technical builds, approaches, and prioritization strategies.This position encompasses line management responsibilities for UK-based Data...

  • Data Governance Lead

    1 week ago

    London, Greater London, United Kingdom LNRS Data Services Ltd Company Full time

    About the Role This role will lead and manage organization-wide data governance activities, focusing on improving the quality and management of key information assets to enhance the overall data maturity of the business. You will drive adherence to our enterprise data governance framework for policies, standards, and practices, ensuring consistency and...

  • Data Engineering Team Lead

    3 days ago

    London, Greater London, United Kingdom N Consulting Ltd Full time

    Job DescriptionN Consulting Ltd is seeking a highly skilled Engineering Lead (Data Engineer) to join our team.The successful candidate will be responsible for leading our data engineering efforts, leveraging expertise in AWS services, cloud data warehousing solutions, and data pipeline orchestration tools.Key ResponsibilitiesData Engineering Leadership:...

  • Senior Data Engineering Lead

    2 days ago

    London, Greater London, United Kingdom Harnham Full time £90,000 - £110,000

    Senior Data Engineering LeadHarnham is partnering with a leading gaming company to find a seasoned Senior Data Engineering Lead to spearhead the migration from their on-prem servers to their new cloud data platform.This global business has established itself as the premier provider of sports betting data and technology over the past 20 years.As a renowned...

  • Data Engineering Team Lead

    1 week ago

    London, Greater London, United Kingdom NewDay Full time

    Lead a High-Performing Data Engineering Team at NewDay We're seeking a passionate and experienced Data Engineering Manager to join our dynamic team at NewDay. In this role, you'll have the opportunity to lead a talented group of data engineers, shaping the future of our data infrastructure and driving impactful business outcomes. Your Impact Establish...

  • Engineering Team Lead

    2 weeks ago

    London, Greater London, United Kingdom LNRS Data Services Ltd Company Full time

    Position Overview As the Engineering Manager at LNRS Data Services Ltd, you will play a pivotal role in steering the engineering team towards success. Your leadership will guide engineers, offshore contractors, and various specialists, ensuring alignment with the company's vision and objectives. You will also be responsible for reporting progress to...

  • Lead Data Science Engineer

    3 days ago

    London, Greater London, United Kingdom Kingfisher Full time

    Job SummaryWe are seeking a highly skilled Lead Data Science Engineer to join our Data Science team at Kingfisher. As part of this role, you will lead the research and development of data-driven services within the Group Data Science team. Your responsibilities will include:Implementing data science projects to support business objectivesBuilding a...

  • Engineering Team Lead

    2 weeks ago

    London, Greater London, United Kingdom LNRS Data Services Ltd Company Full time

    Position Overview As the Engineering Team Lead at LNRS Data Services Ltd, you will be responsible for steering a talented group of software engineers, offshore contractors, and various technical specialists. This role entails not only providing strategic direction but also reporting on project progress to upper management. You will also play a key role...

  • Engineering Team Lead

    2 weeks ago

    London, Greater London, United Kingdom LNRS Data Services Ltd Company Full time

    Position Overview As the Engineering Team Lead at LNRS Data Services Ltd, you will be responsible for providing strategic leadership, management, and vision to a diverse group of engineers, offshore contractors, and platform suppliers. This role involves reporting progress to upper management and may include oversight of departmental budgets, talent...

  • Lead Data Engineer

    1 week ago

    London, Greater London, United Kingdom Aimpoint Digital Full time

    About Aimpoint Digital Aimpoint Digital is a fully remote data and analytics consultancy that partners with innovative software providers to solve complex business challenges. We blend modern tools and techniques with proven principles to deliver optimal data engineering solutions for our clients. Your Role As a Lead Data Engineer, you will be a trusted...

  • Data Engineering Lead

    28 minutes ago

    London, Greater London, United Kingdom Eames Consulting Full time

    Alteryx Developer PositionEames Consulting is seeking a highly skilled Alteryx Developer to join a leading Banking company on a 3-year fixed-term contract.Extensive experience in Alteryx and Tableau developmentProven track record as a Data Engineer or similar role, delivering end-to-end data solutionsDemonstrable expertise in creating dataflows, extracts,...