Data Engineering Platform Lead

3 weeks ago


London, Greater London, United Kingdom Core Consultants Full time
Job Description:

The role of the Senior Data Engineer will be to design and implement a cutting-edge internal data ingestion platform, leveraging Apache Spark, Kafka, and Airflow. This platform will enable scalable and efficient ingestion of diverse datasets, supporting downstream analytics and visualisation requirements.

The successful candidate will lead a team of data engineers, driving the development of configurable Spark ETL pipelines and leveraging modern data architecture principles. They will act as the product owner for the internal data ingestion platform, ensuring alignment with business needs and long-term scalability.

This is a high-impact role that requires expertise in designing scalable systems to handle diverse datasets efficiently. The ideal candidate will have extensive experience with Apache Spark, Kafka, and Airflow, as well as proficiency in designing and deploying solutions on Azure Kubernetes Service (AKS).

**Key Responsibilities:**

  • Design and implement a robust data ingestion platform using Apache Spark, Kafka, and Airflow.
  • Lead a team of data engineers to develop configurable Spark ETL pipelines and ensure adherence to best practices.
  • Act as the product owner for the internal data ingestion platform, ensuring alignment with business needs and long-term scalability.
  • Collaborate with cross-functional teams to ensure seamless integration with downstream analytics and visualisation tools.

**Required Skills and Qualifications:**

  • Extensive experience with Apache Spark, Kafka, and Airflow for data ingestion and orchestration.
  • Proficiency in designing and deploying solutions on Azure Kubernetes Service (AKS).
  • Deep understanding of Databricks for data preprocessing and transformation.
  • Hands-on experience with Snowflake, SQL warehouses, and Delta Lake for data storage.

**Estimated Salary: $150,000 - $200,000 per annum.



  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Company OverviewData Idols is a prestigious retailer that's evolving into a digital-first, data-led organisation. We're seeking a talented Senior Data Platform Engineer to join our team and contribute directly to data-driven decision-making, helping millions of customers and thousands of colleagues alike.Job DescriptionWe're looking for a skilled Senior Data...


  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Data Idols is a prestigious retailer evolving into a digital-first, data-led organisation. We are seeking a Senior Data Platform Engineer to shape the data landscape and work alongside a talented team to build, optimise, and secure scalable data infrastructure.As a Senior Data Platform Engineer at Data Idols, you will play a vital role in designing,...


  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    About Data IdolsWe are a prestigious retailer that's evolving into a digital-first, data-led organisation. As a Senior Data Platform Engineer, you'll be pivotal to shaping the data landscape and work alongside a talented, collaborative team to build, optimise, and secure scalable data infrastructure.Reporting to the Head of Data Platforms, you'll have the...


  • London, Greater London, United Kingdom CAPGEMINI ENGINEERING Full time

    About the JobThis is an exceptional opportunity to join CAPGEMINI ENGINEERING as a Senior Data Platform Engineer, working on innovative data architecture projects that drive business growth. With a strong background in data engineering, AI, and platform consulting, you will deliver cutting-edge data architecture solutions to clients, leveraging your...


  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Job OverviewData Idols is seeking a highly skilled Senior Data Platform Engineer to join our team. This role will play a pivotal part in shaping the data landscape and working alongside a talented team to build, optimise, and secure scalable data infrastructure.


  • London, Greater London, United Kingdom NewDay Ltd Full time

    We are seeking an experienced Data Engineering Platform Lead to join our team at NewDay Ltd in London. This is a permanent full-time role that offers a competitive salary, bonus opportunities, and a hybrid working approach. In this position, you will lead a team of platform engineers to design, optimize, and maintain our core technology platforms, ensuring...


  • London, Greater London, United Kingdom Monte Carlo Data, Inc. Full time

    **About Monte Carlo Data, Inc.**As businesses increasingly rely on data to drive informed decisions, it's crucial that this data is accurate and reliable. Monte Carlo, the industry-leading data reliability company, has created the first end-to-end Data Observability platform. We've achieved numerous accolades, including Best Workplace for 2024 and a DBTA...


  • London, Greater London, United Kingdom Data Science Festival Full time

    Job Description:We are seeking a highly skilled Data Platform Strategist to lead the development and delivery of our data platform. As a key member of the Data Tech team, you will drive responsible usage of our newly built data platform, ensuring it evolves with cutting-edge technologies and meets the needs of practitioners and business partners.The...


  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    About the RoleWe are currently seeking a Senior Data Platform Engineer to join our team at Data Idols, a renowned retailer in the digital-first era. This is an exciting opportunity to contribute to the design, implementation, and maintenance of a robust data platform essential to our ongoing digital transformation.Your Key Responsibilities:Design and...


  • London, Greater London, United Kingdom Data Science Festival Full time

    About the OpportunityAt the Data Science Festival, we are committed to driving innovation and excellence in data engineering. We are seeking a Cloud Data Engineering Lead to join our team and play a key role in shaping our data strategy.Key ResponsibilitiesDevelop and implement data pipeline architectures using modern cloud technologiesLead cross-functional...

  • Data Platform Lead

    2 weeks ago


    London, Greater London, United Kingdom ZipRecruiter Full time

    Job Summary:A challenging role for a skilled data engineer to lead the development of our cloud-based analytics platform.The ideal candidate will have experience in building scalable data platforms using Python, SQL, and GCP.We are seeking a highly pragmatic engineer to work closely with our analytics team to build platforms that provide business critical...

  • Data Engineer

    2 weeks ago


    London, Greater London, United Kingdom Data Science Festival Full time

    We are currently seeking a Senior Data Platform Engineer to join our prestigious retailer that's evolving into a digital-first, data-led organisation. In this role, you'll be pivotal to shaping the data landscape and will work alongside a talented, collaborative team to build, optimise, and secure scalable data infrastructure.Estimated Salary: 80,000 –...

  • Data Platform Lead

    6 days ago


    London, Greater London, United Kingdom Canada Life Limited Full time

    Are you an expert in data architecture looking for a new challenge? We're seeking a Data Platform Lead to join our Digital and Customer Engineering department at Canada Life UK. In this role, you'll be responsible for designing and implementing our data platform solution architecture using full-stack Azure data platform capabilities.About the RoleThis is an...


  • London, Greater London, United Kingdom Core Consultants Full time

    Job Title: Data Engineering Platform LeadAbout Us:

  • Data Platform Lead

    2 weeks ago


    London, Greater London, United Kingdom Oliver Bernard Full time £100,000 - £110,000

    At Oliver Bernard, we are seeking a talented Data Platform Lead to join our market analytics platform team.Key Responsibilities:Lead the team building our data platformWork closely with the analytics team to build platforms for business-critical insightsRequirements:Proven experience in leading data engineering teamsStrong skills in Python, SQL, and...

  • Cloud Data Engineer

    2 weeks ago


    London, Greater London, United Kingdom Data Science Festival Full time

    About Our Data Science FestivalJob Title: Senior Data Platform EngineerWe are a leading digital-first, data-led organisation. As a Senior Data Platform Engineer, you'll be pivotal to shaping the data landscape and work alongside a talented, collaborative team to build, optimise, and secure scalable data infrastructure.In this role, you'll have the...


  • London, Greater London, United Kingdom Ogury Ltd Full time

    Ogury Ltd, a global adtech company, is seeking an experienced Cloud Data Platform Lead to shape the future of our data platform.About the RoleThis is an exciting opportunity to define the long-term vision and lead execution of Ogury's data platform, making data available to all teams and enabling them to take well-informed data-driven...


  • London, Greater London, United Kingdom Data Idols Full time

    Data Idols is driving its digital transformation, and we need a talented Senior Data Platform Engineer to shape our data infrastructure. You'll design and implement scalable, reliable, and secure data platforms using Azure Services.We're looking for someone with a strong understanding of DevOps practices, solid knowledge of security best practices on Azure,...


  • London, Greater London, United Kingdom LinuxRecruit Full time

    Proprietary Data Platform LeadLinuxRecruit is seeking an experienced Proprietary Data Platform Lead to join our team in London.The successful candidate will be responsible for leading the systems engineering team, focusing on building a proprietary data platform and supporting fibre product development.The ideal candidate will have hands-on experience in...


  • London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Job DescriptionWe are currently seeking a Senior Data Platform Engineer to join our team. This role involves designing and implementing scalable, reliable, and secure data platform infrastructure using Azure Services.Maintaining an optimised platform by monitoring performance across compute and storage solutionsLeading CI/CD deployments for data...