Senior Data Pipeline Specialist

23 hours ago


London, Greater London, United Kingdom Vallum Associates Full time
Senior Data Pipeline Specialist

We are seeking an experienced Senior Data Pipeline Specialist to join our team at Vallum Associates. This role is critical in designing, implementing, and maintaining efficient data pipelines using Apache Kafka. As a key member of our engineering team, you will work closely with stakeholders to identify business needs and develop solutions that drive innovation and growth.

### Key Responsibilities:
  • Data Pipeline Design:
    • Design and implement data pipelines using Apache Kafka, ensuring scalability and reliability.
    • Collaborate with cross-functional teams to integrate Kafka with core banking systems and other third-party applications.
    • Develop and maintain CI/CD pipelines integrating Python scripts for testing and deployment.
  • Python Automation:
    • Develop Python-based automation scripts for system monitoring, error handling, and data pipeline management.
    • Implement tools to monitor and alert potential system failures.
  • Performance Optimization:
    • Monitor Kafka performance, resolve bottlenecks, and optimize throughput.
    • Analyze system logs and errors, resolving issues related to Kafka and Python scripts.
  • Team Collaboration:
    • Work closely with infrastructure and security teams to meet compliance standards.
    • Participate in code reviews and contribute to the development of best practices.

### Requirements:
  • Technical Expertise:
    • Strong proficiency in Apache Kafka (setup, configuration, and administration).
    • Advanced knowledge of Python for automation and scripting.
    • Familiarity with Kafka Streams, Connectors, and Schema Registry.
  • Automation and Tools:
    • Experience with tools like Jenkins, Ansible, Terraform, or similar for automation.
    • Knowledge of CI/CD pipelines and DevOps practices.
  • Soft Skills:
    • Strong analytical and problem-solving skills.
    • Excellent communication and collaboration abilities.

### Estimated Salary: £70,000 - £90,000 per annum

  • London, Greater London, United Kingdom Data Science Festival Full time

    We are recruiting a Highly Skilled Data Pipeline Specialist to join our team at the Data Science Festival. This role involves designing and developing data pipelines using Azure cloud platform, maintaining and improving existing data solutions, and collaborating with wider data teams to ensure delivery of high-quality data insights.About YouThe ideal...


  • London, Greater London, United Kingdom Northern Data Group Full time

    OverviewNorthern Data Group is a cutting-edge company that's changing the face of High-Performance Computing. With a strong focus on innovation and sustainability, we're shaping the future of data centers worldwide.Job SummaryWe're seeking a highly skilled Senior Data Engineer to join our team. As a key member of our Data Team, you will play a crucial role...


  • London, Greater London, United Kingdom Data Idols Full time £85,000 - £95,000

    About the Role: Lead Data Pipeline DeveloperWe are seeking an experienced lead developer to join our data team as a Lead Data Pipeline Developer. In this role, you will be responsible for designing, developing, and maintaining complex data pipelines that deliver business-critical insights. The ideal candidate will have experience in data warehousing, data...

  • Data Engineer

    1 day ago


    London, Greater London, United Kingdom Data Science Festival Full time

    Company OverviewThe Data Science Festival is one of the best-known retail brands in the UK, looking for a skilled Staff Data Engineer to support the delivery of data transformation during an exciting time within the company. As a Staff Data Engineer, you will collaborate with the wider data teams to build and maintain data pipelines.


  • London, Greater London, United Kingdom Tampa Gardens Senior Living Full time

    Job DescriptionThe Senior Machine Learning Engineer role at Bento involves designing and building a data pipeline, deploying and tuning AI models for Bento merchants focusing on customer churn prediction and on-screen recovery.This expert will provide machine learning expertise to Bento engineers and product managers, optimally shaping products to meet...


  • London, Greater London, United Kingdom Northern Data Group Full time

    About the TeamNorthern Data Group is a leading provider of High-Performance Computing (HPC) solutions. Our team is dedicated to delivering innovative data-driven solutions to our clients.Main Responsibilities:Design, build, and maintain data pipelines and infrastructure to support data analysis and reporting.Collaborate with business stakeholders to...


  • London, Greater London, United Kingdom Data Idols Full time £85,000 - £95,000

    Data Idols is seeking a highly skilled Data Transformation Specialist to support the delivery of data transformation during an exciting time. As a key member of the team, you will collaborate with the wider data teams to build and maintain data pipelines. This role requires extensive experience with data pipelines, data warehousing, and driving high...


  • London, Greater London, United Kingdom Data Idols Full time £75,000 - £95,000

    **Overview of Role**We are seeking a highly skilled Senior Data Engineer to join our team at Data Idols. As a key member of our data function, you will be responsible for designing, building, and maintaining robust data pipelines that drive business growth.The successful candidate will have a strong background in data engineering, with expertise in data...


  • London, Greater London, United Kingdom Harrington Starr Full time

    We are looking for a highly skilled Senior Data Pipeline Developer to join our team at Harrington Starr. As a Senior Data Pipeline Developer, you will be responsible for designing and implementing automated data pipelines using Python and AWS.The ideal candidate will have experience in building and maintaining software infrastructure, as well as monitoring...


  • London, Greater London, United Kingdom Cedar Cares, Inc Full time

    Career Opportunity at Cboe EuropeWe are seeking a highly skilled Python developer to join our team as a Senior Data Pipeline Specialist. As a member of our global software development team, you will be responsible for designing and building highly reliable Python batch systems to support Cboe Europe's trading operations.The ideal candidate has solid Python...


  • London, Greater London, United Kingdom Proactive Appointments Full time

    About the RoleWe are seeking a highly skilled Data Pipeline Specialist to join our team. As a key member of our technology team, you will be responsible for designing and implementing scalable data pipelines.Salary: £50,000 - £70,000 per annumAbout the Job:Develop robust ETL data pipelines for large scale data into SQL and NoSQL systemsMaintain and...


  • London, Greater London, United Kingdom Charlotte Tilbury Full time

    Join our dynamic team at Charlotte Tilbury Beauty as a Senior DataOps Engineer!As a key member of our data team, you will be responsible for streamlining the development of data products, building solutions to automate repetitive tasks, and designing solutions to protect sensitive data. You will work closely with our data engineering, analytics, and science...


  • London, Greater London, United Kingdom Opus Recruitment Solutions Ltd Full time

    Role Overview: We are seeking a Senior Data Pipeline Developer to join our digital consultancy team at Opus Recruitment Solutions Ltd. As a senior member of the team, you will be responsible for developing and maintaining complex data pipelines that meet the needs of our clients.Key Responsibilities:Develop and maintain data pipelines using various...


  • London, Greater London, United Kingdom Charlotte Tilbury Full time

    Job SummaryWe are seeking an experienced Senior DataOps Engineer to join our team in London. The successful candidate will have experience working in agile development teams on ELT pipelines where data is the product and understand the common pitfalls of what often makes development slow or inefficient.The estimated salary for this role is £90,000 -...


  • London, Greater London, United Kingdom Hexegic Full time

    Data Pipelines Specialist: We're looking for an experienced data engineer to join our team as a data pipelines specialist. As a data pipelines specialist, you'll be responsible for designing, building, and maintaining data pipelines and applications using Python or similar languages, API awareness for data retrieval and integration, front-end/user-facing...


  • London, Greater London, United Kingdom Fifty-Five Full time

    Job DescriptionWe are seeking a talented Cloud Data Pipeline Specialist to join our team at Fifty-Five. As a key member of our data engineering team, you will be responsible for designing and developing data pipelines using Google Cloud Platform (GCP) services.This is an exciting opportunity for a highly skilled individual with expertise in data engineering...

  • Senior Data Architect

    3 weeks ago


    London, Greater London, United Kingdom Data Idols Full time

    Senior Data EngineerThe OpportunityWe are seeking a Senior Data Engineer to join our team at Data Idols, a leading retail brand in the UK. As a Senior Data Engineer, you will play a key role in driving high standards across the data function and building reliable and high-quality data solutions.Key ResponsibilitiesDesign and develop data pipelines using...


  • London, Greater London, United Kingdom Data Science Festival Full time

    Senior Data EngineerAs a Senior Data Engineer at Data Science Festival, you will play a key role in shaping the company's data transformation strategy. Your expertise in building and maintaining data pipelines will be invaluable in driving high standards across the data function. You will build reliable and high-quality data solutions, supporting the team in...


  • London, Greater London, United Kingdom Data Science Festival Full time

    About the RoleWe are excited to announce a new opportunity for a Senior Data Engineer to join our team at Data Science Festival. As a key member of our data team, you will play a critical role in delivering high-quality data solutions and driving business growth.The ideal candidate will have a strong background in data engineering, with expertise in building...


  • London, Greater London, United Kingdom Builder - What would you Build? Full time

    We're on a mission to revolutionize the way apps are built. As a Data Science Pipeline Specialist at Builder, you'll play a key role in developing and maintaining our data pipelines, working closely with data scientists and engineers to productionize machine learning models and AI services.About the TeamOur Intelligent Systems team is driven by a passion for...