Technical Lead for Data Pipeline Engineering

2 days ago

London, Greater London, United Kingdom VivaCity Full time
About VivaCity
VivaCity is a pioneer in Smart City technology, dedicated to creating sustainable and safe transportation solutions through data insights. Our culture is built on collaboration, openness, and a passion for innovation and learning.

Job Overview
We are seeking an experienced Principal Data Platform Engineer to join our Cloud and AI team. As a key member of our team, you will design, improve, and maintain critical systems that transport vast amounts of data and messages every year. Your work will have a significant impact on our mission to empower safe and sustainable transport worldwide through data insights.

Key Responsibilities
- Design and implement large-scale cloud-based data pipelines using Apache Kafka, Go microservices, PostgreSQL database, Kubernetes, Vault, and other cloud services
- Optimise system performance, maintainability, and operational costs while ensuring they meet business requirements
- Work closely with the Cloud and AI team to establish technical standards and lead architectural decision-making
- Share knowledge proactively with colleagues and provide mentorship, technical guidance, and training to engineers across the department

Requirements
- Deep expertise in Apache Kafka, including broker management, scaling, upgrades, and integration with high-throughput data pipelines
- Proficiency in Unix systems, ideally Linux (we use Ubuntu)
- Experience in operationally managing production software components, including observability, logging, metrics, error reporting, debugging, and live incident management
- Proficiency with PostgreSQL, including query optimisation and SQL proficiency
- Familiarity with common DevOps tools and CI/CD processes
- Knowledge of information security principles, including identity and access management and authentication/authorisation processes

What We Offer
We offer a competitive salary: £80,000 - £110,000 per annum, flexible working arrangements, and a collaborative culture that encourages innovation and learning. If you are a motivated and talented engineer looking for a new challenge, please apply.

  • Lead Data Pipeline Developer

    2 days ago

    London, Greater London, United Kingdom Data Idols Full time £85,000 - £95,000

    About the Role: Lead Data Pipeline DeveloperWe are seeking an experienced lead developer to join our data team as a Lead Data Pipeline Developer. In this role, you will be responsible for designing, developing, and maintaining complex data pipelines that deliver business-critical insights. The ideal candidate will have experience in data warehousing, data...

  • Data Pipeline Architect

    2 days ago

    London, Greater London, United Kingdom Northern Data Group Full time

    About the TeamNorthern Data Group is a leading provider of High-Performance Computing (HPC) solutions. Our team is dedicated to delivering innovative data-driven solutions to our clients.Main Responsibilities:Design, build, and maintain data pipelines and infrastructure to support data analysis and reporting.Collaborate with business stakeholders to...

  • Data Driven Pipeline Engineer

    4 weeks ago

    London, Greater London, United Kingdom Genesis Full time

    Job Title: Data Driven Pipeline EngineerSalary Range: £70,000 - £100,000 per annumLocation: St Pauls, LondonAbout GenesisGenesis is a leading provider of consulting services to the energy industry, renowned for its innovative approach and world-class expertise.We are seeking an experienced Data Driven Pipeline Engineer to join our team in St Pauls, London....

  • Global Data Pipeline Lead

    3 weeks ago

    London, Greater London, United Kingdom Vanguard Full time

    We are looking for a Global Data Pipeline Lead to join our team at Vanguard, a leading investment company. The successful candidate will be responsible for leading the development of common FAS data pipelines for International FAS, working closely with stakeholders across all three regions: Americas, Europe, and APAC.The Global Data Pipeline Lead will...

  • Data Pipeline Lead

    2 days ago

    London, Greater London, United Kingdom Levy Search Full time

    We are looking for a Data Pipeline Lead to join our team at Levy Search. As a highly skilled professional with hands-on technical leadership experience and Microsoft SQL Server expertise, you will be responsible for ensuring optimal database performance and data pipeline optimisation.About the CompanyLevy Search is a company that values innovation and...

  • Chief Data Engineering Lead

    4 weeks ago

    London, Greater London, United Kingdom Data Idols Full time

    Data Idols is looking for a Chief Data Engineering Lead to support the delivery of data transformation.As a key member of the team, you will collaborate with the wider data teams and drive high standards across the data function. The role involves building pipelines to deliver insights enabling the company to be more data driven in their decision making.Key...

  • Data Engineering Lead

    2 days ago

    London, Greater London, United Kingdom Data Science Festival Full time

    About the RoleAs a Staff Data Engineer, you'll be responsible for building and maintaining complex data pipelines, collaborating with our wider data teams to deliver insights that drive business growth.The ideal candidate will have extensive experience with data pipelines, data warehousing, and cloud-based technologies such as Azure.

  • Cloud Data Engineering Lead

    2 days ago

    London, Greater London, United Kingdom Data Science Festival Full time

    About the RoleAs a Staff Data Engineer at the Data Science Festival, you will play a key role in driving high standards across the data function. You will design, develop, and maintain data pipelines using Python and Azure, and collaborate with the wider data teams to achieve business objectives. The ideal candidate will have extensive experience with data...

  • Senior Technical Lead

    2 days ago

    London, Greater London, United Kingdom Xcede Full time

    Job Overview: Xcede is a leading provider of Data Engineering services. We are seeking a Senior Technical Lead - Data Engineering to join our team in London. The successful candidate will be responsible for leading technical projects, collaborating with cross-functional teams, and delivering high-quality results.Key Responsibilities: Lead technical projects,...

  • Technical Lead

    3 weeks ago

    London, Greater London, United Kingdom ITV Full time

    About Our TeamWe are a dynamic and innovative team, responsible for helping ITV realise the value of data. We put our curiosity to work in influencing the business, and as a mix of number crunchers and behavioural experts, we blend science into a creative business. As a Technical Lead - Data Engineering, you will be part of this team, contributing to...

  • Pipelines Engineering Lead

    2 days ago

    London, Greater London, United Kingdom Synapsco Full time

    Pipeline Engineering PositionWe are currently seeking an experienced Pipeline Engineer to join our team at Synapsco. In this role, you will be responsible for designing and developing pipeline systems for offshore projects.Main ResponsibilitiesTo design, develop, and implement pipeline systems for offshore projects, ensuring compliance with industry...

  • Data Pipelines Manager, FAS Common Pipelines

    4 weeks ago

    London, Greater London, United Kingdom Vanguard Full time

    About the Technical Product Owner RoleThis role will drive the roadmap and strategy to deliver common FAS data pipelines for International FAS teams. You will work closely with stakeholders in all three regions: Americas, Europe, and APAC, to identify common requirements and work toward a solution that meets the needs of all regions in a timely manner....

  • Fintech Startup Data Pipeline Specialist

    2 days ago

    London, Greater London, United Kingdom Harrington Starr Full time £70,000 - £80,000

    Fintech Startup Seeks Data Engineer LeaderWe are a fast-paced fintech startup looking for a skilled Data Engineer Leader to drive the design and implementation of our data pipeline architecture. The successful candidate will lead cross-functional teams and collaborate with stakeholders to deliver high-quality solutions.Job SummaryThe Data Engineer Leader...

  • Expert Data Pipeline Developer

    2 days ago

    London, Greater London, United Kingdom MBN Solutions Full time £55,000 - £65,000

    Lead Data Engineer Position at MBN SolutionsMBN Solutions is partnering with a leading Financial Services Consulting firm to find an experienced Data Engineer for their expanding IT team. This opportunity is based in the United Kingdom and offers occasional travel to London (2-3 times per month).This Lead Data Engineer position plays a key role in driving...

  • Technical Lead

    4 weeks ago

    London, Greater London, United Kingdom carwow Full time

    About the JobWe are seeking a skilled Technical Lead to join our Data Engineering team at Carwow. As a Technical Lead, you will be responsible for shaping the technical strategy for our data products, developing and testing product improvements, and leading the team in scoping new features. We prioritize data-driven outcomes, technical innovation, and...

  • Enterprise Data Pipeline Engineer

    5 days ago

    London, Greater London, United Kingdom Palantir Technologies Full time

    Palantir Technologies is looking for an Enterprise Data Pipeline Engineer to join our team. In this role, you will be responsible for designing and implementing large-scale data pipelines that meet the needs of our customers.The estimated salary for this position is $140,000 per year, based on the national average for a senior software engineer with similar...

  • Data Pipeline Architect

    1 day ago

    London, Greater London, United Kingdom DataCareers Full time £65,000 - £75,000

    Data Engineer VacancyWe are seeking an experienced Data Engineer to join our team at DataCareers. The successful candidate will be responsible for designing and building scalable data pipelines to process and transform raw data into structured data products.This is a key role in the development of our organisation's analytics platform, and we require a...

  • Senior Technical Lead

    4 weeks ago

    London, Greater London, United Kingdom Carwow Group Full time

    As a Technical Lead at Carwow Group, you will play a crucial role in shaping the technical direction of our Data Engineering team and driving the development of innovative data solutions.We offer a dynamic and supportive work environment, with opportunities for professional growth and development. Our team is passionate about empowering drivers and the...

  • Pipeline Design Lead

    4 weeks ago

    London, Greater London, United Kingdom oilandgas Full time

    Job Title: Pipeline Design LeadWe are seeking an experienced Pipeline Design Lead to join our team in St Pauls, London. The successful candidate will be responsible for leading the detailed design aspects of the offshore section of the pipeline that will be utilised for CO2 transportation and disposal in North Sea oil reservoirs.About the Role:Proactively...

  • Data Engineer Team Lead

    4 weeks ago

    London, Greater London, United Kingdom Data Science Festival Full time

    We are looking for a highly skilled Senior Data Engineer to join our team at the Data Science Festival. This role involves supporting the delivery of data transformation and collaborating with the wider data teams.Job DescriptionThe ideal candidate will have extensive experience with data pipelines, data warehousing, and driving high standards across the...