Lead Data Engineer

1 week ago

London Area, United Kingdom Aubay UK Full time

Aubay is seeking a highly motivated Lead Data Engineer to join our dynamic team and contribute to the design, development, and optimisation of our client's Data Lakehouse platform. As a Lead Data Engineer, you will provide technical leadership and work closely with cross-functional teams to translate business requirements into robust technical solutions. The ideal candidate will have extensive experience in data engineering, a strong understanding of data modelling and big data technologies, and the ability to lead and mentor a team of data engineers.

Required Skills and Experience:

  • Extensive experience with Databricks, Apache Spark, and data lake architectures.
  • Proficiency in Python and SQL for data manipulation and analysis.
  • Experience with CI/CD processes using Azure DevOps or Git for streamlined development workflows.
  • Strong knowledge of streaming data technologies such as Kafka and KSQL DB.
  • Familiarity with reporting tools like Power BI and Qlik for data visualisation and analytics.
  • Software engineering experience with .NET, particularly for developing event-driven pipelines and automation testing.
  • Experience with Databricks Unity Catalog and understanding of data governance tools.
  • Strong skills in data modelling, including Snowflake modelling, star schema modelling, and other advanced techniques.
  • Ability to optimise big data workloads in Spark to ensure efficient data processing and analytics.
  • Self-driven and motivated with a strong sense of ownership and accountability.
  • Proven experience in managing and inspiring technical teams to achieve project goals.
  • Excellent communication and collaboration skills, with the ability to effectively interact with both business and IT teams.
  • Strong problem-solving and analytical skills, with a keen ability to provide constructive feedback.
  • Ability to foster a positive team environment and encourage continuous improvement.

Desired Skills and Experience:

  • Experience in delivering projects in the streaming data world, particularly with real-time data processing.
  • Understanding of data governance and data quality best practices, ensuring data integrity and compliance.
  • Familiarity with automation testing frameworks to enhance testing efficiency.
  • Experience working on projects related to energy trading, providing insights into industry-specific challenges and solutions.

Role Responsibilities:

  • Lead the design, development, and implementation of data engineering solutions for the Data Lakehouse platform.
  • Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
  • Provide technical leadership, mentoring, and coaching to the data engineering team to ensure high-quality deliverables.
  • Ensure best practices in data engineering, including data modelling, ETL processes, and data quality.
  • Optimise big data workloads using Apache Spark and other big data technologies to enhance performance and efficiency.
  • Manage CI/CD pipelines using Azure DevOps or Git, ensuring seamless integration and delivery processes.
  • Develop and maintain event-driven pipelines using .NET and other relevant technologies to support real-time data processing.
  • Implement and manage Databricks Unity Catalog and other data governance tools to ensure data security and compliance.
  • Deliver projects in the streaming data environment using Kafka, KSQL DB, and similar technologies to handle high-volume data streams.
  • Utilise reporting tools such as Power BI and Qlik for data visualisation and reporting to provide actionable insights to stakeholders.

About Aubay UK – Ahead of Innovation

Aubay UK is a recognised InSourcing Partner for client-side deployment delivered across London. Our team, based in Canary Wharf, specialises in hiring IT professionals within London’s Energy and FinTech sectors, helping our clients to expand their operations with top-tier talent who are experts in their fields. We work exclusively with clients who are globally recognized as Energy Super Majors/Financial Services and innovative FinTech players.

Aubay UK is the most recently started branch of Aubay Group www.aubay.com. Aubay Group is an international Digital Services Company, listed on a Euronext Stock Exchange, who have been operating for 25 years in the European market and working alongside some of the biggest names in the Banking, Finance, Insurance, Energy, IT/Digital, Manufacturing, Transport and Telecoms sectors. With over 7,800 employees across 7 countries and 16 offices in England, France, Belgium, Luxembourg, Italy, Spain, and Portugal, Aubay Group generated revenues of €534 million in 2023.

What Aubay UK Offers

At Aubay UK, people are at the heart of our business. We offer a competitive remuneration package which includes a range of benefits. You will receive continuous support from our dedicated Talent Acquisition Specialists, who will support your career development and success during your assignment with our client. Other benefits include:

  • 25 Days Annual Leave + Bank Holidays
  • Pension scheme
  • Work-from-home opportunities
  • Access to Udemy training platform and conferences
  • Discount on winter sports holidays at www.aero1607.com
  • Opportunity to work directly for the client at the end of the assignment

Our Client

Our client is one of the Super Major global energy companies with around 84,000 employees across 70+ countries who are working to power progress through cleaner energy solutions.

Specialities: Upstream/Downstream, Biofuels, Integrated Gas, New Energies, Chemicals, Energy and Trading.

You will have the opportunity to work in a challenging but rewarding environment that is fast-paced and changing fundamentally and work towards transforming the business of a Super Major energy company to meet the ambition to be a net-zero emissions energy business by 2050 whilst delivering a world-class business case that has a robust societal license to operate. In your role, you will be expected to enact change and deliver value globally across business lines and geographies.

Apply via our website - www.aubay.co.uk

This role requires you to be based in the UK and have the right to work in the UK.

Aubay UK is proud to be an equal-opportunity employer. Employment decisions will be based on merit, competence, performance, and business needs.

  • Data Engineering Delivery Lead

    2 weeks ago

    London, Greater London, United Kingdom NTT DATA Full time

    Lead Data Engineering Projects at a Global IT LeaderNTT DATA, a world-renowned IT service provider, is seeking a skilled and experienced Data Engineering Delivery Manager to join our dynamic team. As the 6th largest IT Service Provider globally, we are dedicated to driving business change and technology transformation across diverse industries.Your Role:In...

  • Data Engineering Delivery Lead

    3 days ago

    London, Greater London, United Kingdom NTT DATA Full time

    Lead Data Engineering Projects at a Global IT LeaderNTT DATA, a world-renowned IT service provider, is seeking a skilled and experienced Data Engineering Delivery Manager to join our dynamic team. As the 6th largest IT Service Provider globally, we are dedicated to driving business change and technology transformation across diverse industries.Your Role:In...

  • Technical Delivery Lead

    5 days ago

    London, Greater London, United Kingdom NTT DATA Full time

    About the RoleWe are seeking an experienced Senior Data Engineering Project Manager to join our team at NTT DATA. As a key member of our Data Engineering service line, you will be responsible for overseeing Data Engineering projects and programmes across our Data transformation portfolio.Key ResponsibilitiesLead highly-skilled technical teams in a...

  • Technical Delivery Lead

    3 days ago

    London, Greater London, United Kingdom NTT DATA Full time

    About the RoleWe are seeking an experienced Senior Data Engineering Project Manager to join our team at NTT DATA. As a key member of our Data Engineering service line, you will be responsible for overseeing Data Engineering projects and programmes across our Data transformation portfolio.Key ResponsibilitiesLead highly-skilled technical teams in a...

  • Lead Data Engineering Specialist

    3 days ago

    London, Greater London, United Kingdom ON Data Staffing Full time

    About the RoleWe are seeking a highly skilled Senior Data Solutions Architect to join our team at ON Data Staffing. As a key member of our data engineering team, you will play a pivotal role in delivering innovative big data solutions to our clients.Key ResponsibilitiesClient EngagementCollaborate directly with our clients to understand their business...

  • Lead Data Engineering Specialist

    5 days ago

    London, Greater London, United Kingdom ON Data Staffing Full time

    About the RoleWe are seeking a highly skilled Senior Data Solutions Architect to join our team at ON Data Staffing. As a key member of our data engineering team, you will play a pivotal role in delivering innovative big data solutions to our clients.Key ResponsibilitiesClient EngagementCollaborate directly with our clients to understand their business...

  • Lead Data Engineer

    3 weeks ago

    London Area, United Kingdom Dragonfly People Full time

    Lead Data EngineerData | Engineer | Lead | AWS | Python | SQL | Airflow | Snowflake | DBT | Fintech A Lead Data Engineer is required by a rapidly growing consumer Fin-Tech business who are based in London. The Lead Data Engineer will mentor and coach a team of Data Engineers, while driving automation and seeking opportunities to optimise data resources. You...

  • Lead Data Engineer

    3 weeks ago

    London Area, United Kingdom Dragonfly People Full time

    Lead Data Engineer Data | Engineer | Lead | AWS | Python | SQL | Airflow | Snowflake | DBT | Fintech A Lead Data Engineer is required by a rapidly growing consumer Fin-Tech business who are based in London. The Lead Data Engineer will mentor and coach a team of Data Engineers, while driving automation and seeking opportunities to optimise data resources....

  • Lead Data Engineer

    3 weeks ago

    London Area, United Kingdom Dragonfly People Full time

    Lead Data EngineerData | Engineer | Lead | AWS | Python | SQL | Airflow | Snowflake | DBT | Fintech A Lead Data Engineer is required by a rapidly growing consumer Fin-Tech business who are based in London. The Lead Data Engineer will mentor and coach a team of Data Engineers, while driving automation and seeking opportunities to optimise data resources. You...

  • Data Technical Lead

    2 days ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadSalary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the software architecture design as well as evolve and formulate functions. You will collaborate with wider teams to implement technical designs on...

  • Lead Data Engineer

    3 weeks ago

    London Area, United Kingdom Vallum Associates Full time

    Lead Data Engineer - Outside IR35 - Hybrid - London6 months initial contract Hybrid - two days a week in London office near Oxford CircusOutside IR35Competitive daily rates on offer! As a Lead Data Engineer, you’ll play a key role in developing and maintaining robust cloud-based solutions for the benefit of data scientists, analysts, traders, and other...

  • Lead Data Engineer

    6 days ago

    London Area, United Kingdom Vallum Associates Full time

    Lead Data Engineer - Outside IR35 - Hybrid - London 6 months initial contract Hybrid - two days a week in London office near Oxford Circus Outside IR35 Competitive daily rates on offer! As a Lead Data Engineer, you’ll play a key role in developing and maintaining robust cloud-based solutions for the benefit of data scientists, analysts, traders, and...

  • Lead Data Engineer

    3 weeks ago

    London Area, United Kingdom Vallum Associates Full time

    Lead Data Engineer - Outside IR35 - Hybrid - London6 months initial contract Hybrid - two days a week in London office near Oxford CircusOutside IR35Competitive daily rates on offer! As a Lead Data Engineer, you’ll play a key role in developing and maintaining robust cloud-based solutions for the benefit of data scientists, analysts, traders, and other...

  • Data Technical Lead

    2 days ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadSalary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the software architecture design as well as evolve and formulate functions. You will collaborate with wider teams to implement technical designs on...

  • Data Technical Lead

    2 days ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadSalary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the software architecture design as well as evolve and formulate functions. You will collaborate with wider teams to implement technical designs on...

  • Data Technical Lead

    2 days ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadSalary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the software architecture design as well as evolve and formulate functions. You will collaborate with wider teams to implement technical designs on...

  • Data Lead

    3 months ago

    London, United Kingdom Data Idols Full time

    **Data Science Lead**: Data Idols are working with a disrupter within the delivery service space. They are looking for a Data Science Lead to join their growing team and support them as they invest heavily in their data platform. **The Opportunity**: This opportunity comes with huge growth and the chance to really put your mark on this role. The company are...

  • Data Technical Lead

    2 days ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadHit Apply below to send your application for consideration Ensure that your CV is up to date, and that you have read the job specs first.Salary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the...

  • Data Technical Lead

    1 day ago

    London, United Kingdom Data Idols Full time

    Data Technical LeadHit Apply below to send your application for consideration Ensure that your CV is up to date, and that you have read the job specs first.Salary: up to £80k Location: RemoteData Idols is working with one of the world's largest gaming organisations to hire a Data Technical Lead. This is an opportunity to work at scale on leading the...

  • Engineering Lead

    2 weeks ago

    London Area, United Kingdom Provide Full time

    Engineering Lead - Data£120,000 + 30% bonus50/50 hybrid model in Central LondonWe are working exclusively with a global asset management company who have a long, successful history and are looking to take on an Engineering Lead within their Marketing Technology team to lead the data and software engineering function across their data space. You would come...