AWS Data Pipeline Manager

2 days ago

London, Greater London, United Kingdom TalentHawk Full time £500
About This Job Opportunity

TalentHawk is hiring an experienced Data Engineer to join our team.

This role involves leading our data engineering efforts, managing and optimizing AWS-based data processes, orchestrating data flows, and supporting business intelligence capabilities.

Key Responsibilities:

  • Data Pipeline Management: Design, implement, and maintain scalable data pipelines using AWS services such as Apache Kafka, AWS Glue, and PySpark.
  • Database Integration: Integrate MySQL databases on AWS (approximately 25TB) as primary data sources feeding the data pipeline.
  • Real-Time Data Ingestion: Utilize Apache Kafka to manage data flows from MySQL into AWS S3 buckets, enabling real-time streaming.
  • Data Processing: Implement ETL processes using AWS Glue and PySpark to build data lakes within S3 for further analysis.
  • Business Intelligence (BI):
  • Facilitate data availability for BI purposes, supporting daily updates and aiming to move from overnight batch jobs to near-real-time 15-minute refreshes.
  • Collaborate on BI dashboard and reporting creation to provide actionable insights.
  • Complex Data Reconciliation:
  • Reconcile transaction records across multiple merchants, currencies, and network operators, addressing both straightforward and complex data relationships.

Requirements:

  • Data Engineering Expertise:
  • Proficiency with MySQL, Apache Kafka, AWS Glue, PySpark, and AWS S3 for managing data pipelines and ETL processes.
  • Business Intelligence Skills:
  • Experienced in designing and maintaining BI reports and dashboards with frequent data refresh capabilities.
  • Data Reconciliation Experience:
  • Skill in reconciling transactional data across diverse systems, operators, and currencies to ensure data accuracy.

Estimated Daily Rate: £500.

  • AWS Data Pipeline Specialist

    4 days ago

    London, Greater London, United Kingdom MRJ Recruitment Full time £70,000 - £80,000

    AWS Data Engineer - UK OperationsMJR Recruitment is searching for a highly skilled data engineer to join our team in the UK. With a salary of £72k and a focus on supporting large-scale analytics, this role requires expertise in designing and developing robust data pipelines on AWS.**Job Responsibilities:*** Design and maintain scalable data environments...

  • AWS Data Engineer Position

    3 weeks ago

    London, Greater London, United Kingdom Data Science Festival Full time

    Job SummaryWe are seeking a skilled AWS Data Engineer to join our team at Data Science Festival. This is a fully remote opportunity with an estimated salary range of £80,000 - £110,000 per annum.The successful candidate will be responsible for building and maintaining reliable data pipelines, collaborating with the wider data teams, and driving high...

  • Data Pipeline Development Manager

    7 hours ago

    London, Greater London, United Kingdom Paragon Alpha - Hedge Fund Talent Business Full time

    Job TitleA Data Pipeline Development Manager is needed to lead the development and maintenance of pipelines and platforms for quantitative research data sets at a systematic hedge fund. This role requires expertise in Python programming and SQL.Key Responsibilities:Develop and implement ETL processes using Python and SQL.Manage the maintenance of AWS-based...

  • Data Pipeline Specialist

    1 day ago

    London, Greater London, United Kingdom Proactive Appointments Full time

    About the RoleWe are seeking a highly skilled Data Pipeline Specialist to join our team. As a key member of our technology team, you will be responsible for designing and implementing scalable data pipelines.Salary: £50,000 - £70,000 per annumAbout the Job:Develop robust ETL data pipelines for large scale data into SQL and NoSQL systemsMaintain and...

  • Data Pipeline Architect

    2 days ago

    London, Greater London, United Kingdom DataCareers Full time £65,000 - £75,000

    Data Engineer VacancyWe are seeking an experienced Data Engineer to join our team at DataCareers. The successful candidate will be responsible for designing and building scalable data pipelines to process and transform raw data into structured data products.This is a key role in the development of our organisation's analytics platform, and we require a...

  • Lead Data Pipeline Developer

    3 days ago

    London, Greater London, United Kingdom Data Idols Full time £85,000 - £95,000

    About the Role: Lead Data Pipeline DeveloperWe are seeking an experienced lead developer to join our data team as a Lead Data Pipeline Developer. In this role, you will be responsible for designing, developing, and maintaining complex data pipelines that deliver business-critical insights. The ideal candidate will have experience in data warehousing, data...

  • Data Pipeline Architect

    3 days ago

    London, Greater London, United Kingdom Northern Data Group Full time

    About the TeamNorthern Data Group is a leading provider of High-Performance Computing (HPC) solutions. Our team is dedicated to delivering innovative data-driven solutions to our clients.Main Responsibilities:Design, build, and maintain data pipelines and infrastructure to support data analysis and reporting.Collaborate with business stakeholders to...

  • Data Pipeline Optimization Expert

    8 hours ago

    London, Greater London, United Kingdom Northern Data Group Full time

    OverviewNorthern Data Group is a cutting-edge company that's changing the face of High-Performance Computing. With a strong focus on innovation and sustainability, we're shaping the future of data centers worldwide.Job SummaryWe're seeking a highly skilled Senior Data Engineer to join our team. As a key member of our Data Team, you will play a crucial role...

  • Data Pipeline Architect

    3 weeks ago

    London, Greater London, United Kingdom Harnham Full time £70,000 - £80,000

    We're excited to announce an opportunity for a highly skilled Data Engineer to join our team at Harnham. As a Senior Python Software Engineer, you will play a key role in designing and developing our AWS platform and data pipelines, leveraging your expertise in Python, ML, and CI/CD.About the RoleThis challenging position demands a technical expert who can...

  • Senior Data Pipeline Developer

    3 days ago

    London, Greater London, United Kingdom Accelero Full time £65,000

    About the Job:We are seeking an experienced AWS Data Engineer to join our team at Accelero. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining scalable data pipelines on AWS to process large-scale datasets.Your Key Responsibilities Will Include:Designing and implementing data pipelines using AWS...

  • Cloud Engineer

    4 days ago

    London, Greater London, United Kingdom Intec Select Full time £140,000

    Job Description:We are seeking an experienced Cloud Engineer - Data Pipelines and Automation to join our client's team at Intec Select. As a key member of our digital marketing and technology agency, you will play a critical role in designing, implementing, and managing AWS infrastructure, data pipelines, and real-time data streams for high-volume data...

  • Data Pipeline Architect

    3 days ago

    London, Greater London, United Kingdom Xcede Full time

    OverviewWe are seeking an experienced Data Engineer to join our team at Xcede. As a Data Engineer, you will play a key role in optimising our data architecture and building reliable and scalable data pipelines for the application of advanced analytics and machine learning.About the Role:The successful candidate will be responsible for designing, building and...

  • Data Pipeline Architect

    4 days ago

    London, Greater London, United Kingdom Harnham Full time £75,000 - £95,000

    Role OverviewWe are seeking a highly skilled Data Engineer to join our client's expanding Data and Analytics team in London. As a key member of the team, you will be responsible for building and maintaining a new AWS serverless data platform, streamlining real-time data using Kafka, and improving best coding practices within the business.About the TeamThe...

  • Data Engineering Expert

    4 weeks ago

    London, Greater London, United Kingdom Phiture Full time

    Job OverviewWe are seeking an experienced Data Engineer to join our team at Phiture. As a key member of our data infrastructure team, you will be responsible for designing, building, and maintaining robust data pipelines that support our trading and analytics platforms.About the RoleThe successful candidate will have a strong background in data engineering,...

  • AWS Data Science Professional

    10 hours ago

    London, Greater London, United Kingdom TESTQ Technologies Limited Full time

    Key ResponsibilitiesDesign, implement, and optimize data pipelines for various data integration and transformation tasks, using AWS services such as Glue and Athena.Build and maintain data lakes using AWS Glue, ensuring efficient data extraction, transformation, and loading (ETL) processes.Develop and manage Terraform scripts to automate the deployment of...

  • Machine Learning Developer for Data Pipelines

    3 weeks ago

    London, Greater London, United Kingdom Career Renew Full time

    Platform DevelopmentCareer Renew is developing a cutting-edge cryptocurrency platform, and we are seeking a Machine Learning Developer to work on the data pipeline. The successful candidate will have experience with machine learning, data pipelines, and ETL processes. The average salary for this role is Average Salary: £90,000 - £120,000 per year.Key...

  • Senior Data Pipeline Developer

    4 days ago

    London, Greater London, United Kingdom Harrington Starr Full time

    We are looking for a highly skilled Senior Data Pipeline Developer to join our team at Harrington Starr. As a Senior Data Pipeline Developer, you will be responsible for designing and implementing automated data pipelines using Python and AWS.The ideal candidate will have experience in building and maintaining software infrastructure, as well as monitoring...

  • Data Pipeline Architect

    4 weeks ago

    London, Greater London, United Kingdom Abound Full time

    Your New RoleWe are seeking an experienced Data Engineer to join our team at Abound, a leading player in consumer lending. The successful candidate will be responsible for designing, building, and maintaining ETL pipelines to support our data-driven business decisions.The role involves working closely with data scientists, analysts, and other stakeholders to...

  • Data Engineer

    1 day ago

    London, Greater London, United Kingdom Data Science Festival Full time

    Company OverviewThe Data Science Festival is one of the best-known retail brands in the UK, looking for a skilled Staff Data Engineer to support the delivery of data transformation during an exciting time within the company. As a Staff Data Engineer, you will collaborate with the wider data teams to build and maintain data pipelines.

  • Expert Data Pipeline Developer

    23 hours ago

    London, Greater London, United Kingdom Alphasights Full time

    Job OverviewAlphaSights is seeking an Expert Data Pipeline Developer to join our team. As a key member of our multidisciplinary innovation pods, you will design, develop, deploy and support data infrastructure, pipelines and architectures.Your primary responsibilities will include:Data pipeline development: You will be responsible for designing and...