Big Data Pipeline Architect

2 days ago


Manchester, United Kingdom Smart DCC Full time

At Smart DCC, we are looking for a skilled Big Data Pipeline Architect to lead the development of our data pipeline infrastructure. This role comes with a competitive salary of £90,000 plus benefits.

The ideal candidate will have expertise in designing and implementing scalable data pipelines that support Smart DCC's data science and analytics ambitions. They should also be proficient in developing real-time data processing pipelines, optimizing batch processing workflows, and implementing infrastructure automation using tools like Terraform or Ansible.

A strong understanding of data warehousing concepts, data modelling techniques, and schema design for building and maintaining data warehouses or data lakes is essential. Additionally, the candidate should have advanced proficiency in database technologies such as SQL Server, Oracle, MySQL, or PostgreSQL, as well as expertise in implementing and managing data pipelines.

  • Detailed ETL Expertise: Design and implement complex ETL processes to efficiently extract, transform, and load data from disparate sources into target data stores, ensuring data quality and integrity.
  • Resource Management: Monitor and optimize resource allocation, container resource limits, and cluster configurations to maximize resource utilization, minimize costs, and ensure optimal performance under varying workloads.
  • Cross-functional Collaboration:


  • Manchester, United Kingdom Qodea Full time

    Senior Big Data EngineerWe're looking for an experienced Senior Big Data Engineer to join our team, offering a $130,000 salary and the opportunity to work on cutting-edge projects using Google Cloud Platform. As a Senior Engineer, you'll lead the design and implementation of large-scale data processing systems.Main Responsibilities:Design, build, and...

  • Big Data Engineer

    4 weeks ago


    Manchester, United Kingdom Octopus Energy Full time

    We're Octopus Energy, a technology company dedicated to creating a smart, sustainable energy system. As a Big Data Engineer, you'll be responsible for designing and implementing scalable data infrastructure to support our growing business needs.You'll work on a wide range of projects, from building data pipelines to integrating new data sources, ensuring...


  • Manchester, United Kingdom 83zero Limited Full time

    About 83zero LimitedWe are a global leader in digital services, driving innovation in customer experience through CRM, marketing, business intelligence, and cloud solutions.We are seeking a skilled Data Engineer to join our team. As a key member of our data engineering team, you will play a crucial role in designing and implementing robust data pipelines...


  • Manchester, United Kingdom Qodea Group. Full time

    Job DescriptionWe are looking for a seasoned Data Engineer with a strong background in cloud computing and big data technologies. As a member of our team, you will be responsible for:Designing, building, and maintaining data pipelines to support data ingestion, ETL, and storageBuilding and maintaining automated data pipelines to monitor data quality and...


  • Manchester, United Kingdom Qodea Full time

    **Responsibilities**Design, build, and maintain data pipelines to support data ingestion, ETL, and storage.Build and maintain automated data pipelines to monitor data quality and troubleshoot issues.Implement and maintain databases and data storage solutions.Ensure data quality, accuracy, and completeness.Collaborate with data scientists and analysts to...


  • Manchester, United Kingdom Qodea Full time

    Data Engineer - Cloud and Big DataThis role involves working as part of a dynamic team to design and implement data engineering solutions using Google Cloud Platform. With a $110,000 salary and flexible working arrangements, this is an excellent opportunity to grow your career in a fast-paced environment.Key Tasks:Design and implement data pipelines and...


  • Manchester, United Kingdom ZipRecruiter Full time

    Role SummaryThis is a contract opportunity for an experienced Data Architect with enhanced DV clearance. You will be working on a project that involves big data development, utilising your skills in Hadoop, Kafka, and Elasticsearch.


  • Manchester, United Kingdom Moody's Investors Service Full time

    About the RoleWe are looking for a highly motivated Senior Big Data Cloud Program Specialist to lead the design and implementation of enterprise-grade big data cloud programs. As a key member of our team, you will be responsible for developing and maintaining complex data pipelines and ensuring seamless integration with our existing infrastructure.Key...

  • Data Architect

    2 weeks ago


    Manchester, United Kingdom Np Group Full time

    Job Title: Data & AI Platform Architect (Databricks Specialist) Location: Manchester or London Employment Type: Full-time Experience Level: Senior About the Role As a Data & AI Platform Architect , you will work closely with clients to design, implement, and optimize Databricks-based platforms tailored to their unique needs.You will play a critical role in...

  • Cloud Data Architect

    4 weeks ago


    Manchester, United Kingdom PSD Technology Contracts Ltd. Full time

    About the Role:We are looking for a highly skilled Cloud Data Architect to join our team in Manchester. As a Cloud Data Architect, you will be responsible for designing, implementing, and maintaining our cloud-based data infrastructure.Key Responsibilities:Design and develop scalable cloud-based data pipelines using Python and Azure Data Factory.Collaborate...

  • Big Data Scientist

    1 week ago


    Manchester, United Kingdom Nexus Jobs Limited Full time

    Job Description Big Data Scientist - Expert - Manchester Our Client an international organisation is looking to recruit a Big Data Expert with 7 to 10 years expertise as a Big Data Scientist. You will have Natural Language Processing (NLP) knowledge and experience and understand and analyse large data sets to discover new insights. Working in a...

  • Data Architect

    3 months ago


    Manchester, United Kingdom ShortList Recruitment Limited Full time

    Data ArchitectGreater Manchester£85,000Are you a skilled Data Architect ready to take the lead in a cutting-edge digital transformation? We're looking for someone to play a crucial role in rolling out a forward-thinking data strategy that will define the future of our organisation.Key Responsibilities:Data Strategy and Architecture: Develop a comprehensive...


  • Manchester, United Kingdom iO Associates Full time

    Senior Lead Data Engineer salary: £80,000. A seasoned data engineering professional to lead, innovate, and make an impact in our forward-thinking organisation. This hands-on leadership role involves guiding a talented team of engineers, driving technical excellence, and shaping the future of data solutions.We're looking for an experienced Senior Lead Data...


  • Manchester, United Kingdom Data Science Festival Full time

    About the RoleThe Data Warehouse Architect will play a key role in shaping the future of our data management capabilities. With a strong background in data warehousing and business intelligence, you will design and implement scalable data architectures that support our business growth strategies.

  • Data Engineer

    2 weeks ago


    Manchester, United Kingdom Athsai Full time

    Location: UK (Remote) Experience: 8-10 Years Salary: £70,000 Start Date: January Job Description: We are looking for a dedicated professional with substantial hands-on experience in big data technologies to join our team.In this role, you will have the opportunity to design, implement, and debug effective ETL pipelines, contributing to our overall data...

  • Data Architect

    3 months ago


    Greater Manchester, United Kingdom ShortList Recruitment Limited Full time €85,000

    Data Architect Greater Manchester £Are you a skilled Data Architect ready to take the lead in a cutting-edge digital transformation? We're looking for someone to play a crucial role in rolling out a forward-thinking data strategy that will define the future of our organisation. Data Strategy and Architecture: Develop a comprehensive organisational data...


  • Manchester, United Kingdom Escode, Part of NCC Group Full time

    Job OverviewWe are seeking a skilled Data Engineer to join our Data and Reporting Team at Escode, Part of NCC Group. As a Data Engineer, you will play a critical role in driving the creation of a brand-new data platform in the Microsoft Azure and Fabric space.This is an exciting opportunity to work on a cutting-edge project that will provide services and...

  • Cloud Engineer

    4 weeks ago


    Manchester, United Kingdom CV-Library Full time

    At CV-Library, we're seeking a highly skilled Cloud Engineer to join our analytics department as a Data Architect.The successful candidate will be responsible for designing and implementing modern, cloud-based data pipelines that enable the delivery of data used by internal and external stakeholders. This includes leading data quality initiatives, ensuring...

  • Data Architect

    3 months ago


    Greater Manchester, United Kingdom ShortList Recruitment Limited Full time

    Data ArchitectGreater Manchester£85,000Are you a skilled Data Architect ready to take the lead in a cutting-edge digital transformation? We're looking for someone to play a crucial role in rolling out a forward-thinking data strategy that will define the future of our organisation.Key Responsibilities:Data Strategy and Architecture: Develop a comprehensive...

  • Data Architect

    3 months ago


    Greater Manchester, United Kingdom ShortList Recruitment Limited Full time €85,000

    Data Architect Greater Manchester £85,000 Are you a skilled Data Architect ready to take the lead in a cutting-edge digital transformation? We're looking for someone to play a crucial role in rolling out a forward-thinking data strategy that will define the future of our organisation. Key Responsibilities: Data Strategy and Architecture: Develop a...