Airflow Senior Data Engineer

1 month ago

London, Greater London, United Kingdom Acrotrend Full time

Location:
Central London (hybrid - the candidate should be able to travel to the client location 1-2 times per week)

Experience: 8+ Years, with 5-6 years of Airflow experienceRemuneration: Commensurate with your experience Employment type: Full time/
Contractor Duration: 6-month contractAcrotrend is a part of the NowVertical Group Inc. ("NOW") family of companies. NOW is a publicly listed company on the TSXV. We are a growing team, with operations in multiple countries.

NOW's technology and services serve as the foundation of our client's digital transformations, working with our deep vertical expertise to create next-generation Vertical Intelligence (VI) solutions in the industry.

Acrotrend is a hands-on customer insight and analytics consultancy trusted by leading brands such as RX, GSK, The Economist, Informa, Nuffield Health, Cancer Research UK, Sky, Disney and others to help acquire more customers, retain them and increase Customer Lifetime Value (CLTV).Acrotrend works across the whole end-to-end customer journey to diagnose why customers are churning and devise a bespoke roadmap to deliver predictive visibility and actionable insights using best-in-class technologies for Data Engineering and AI to retain them.

In 2021 Acrotrend was awarded 'Silver' rating in the Financial Times UK's Leading Management Consultancies List.

In 2022, Acrotrend bagged 'Silver' ratings in three categories - 'Data Analytics & Big Data,' 'Internet, Media & Entertainment' and 'Sales & CRM'', and a 'Bronze' rating in 'Healthcare & Life Sciences' to be positioned above some of the industry's leading names.

Check out our website to understand what we do and how we work.
We are looking for a Airflow Senior Data Engineer.

RequirementsYour Role:

Provide technical leadership on data engineering (acquisition, transformation, distribution), across multiple projects and squads Guide, nurture, and coach a team of data engineers, architects and database administrators working in an agile, fast-paced environment Drive adoption and evolution of frameworks and products based on best-in-class data management technologies, such as ETL/ELT tools, open-source frameworks, workflow management tools, etc.

Provide suggestions and recommendation on how to improve the overall data engineering architecture towards a scalable, cost effective ecosystem Your Deliverables:
Design, implement, and manage data pipelines using Apache Airflow. Develop, test, and maintain custom Airflow operators and DAGs. Monitor and troubleshoot Airflow workflows to ensure optimal performance and reliability. Collaborate with data engineers and data scientists to understand data requirements and translate them into scalable workflows. Optimize and tune Airflow configurations for performance and scalability. Ensure data quality and integrity across all workflows. Document processes, workflows, and configurations.

Develop, test, and maintain DBT infrastructure that is executed by Airflow Your Expertise:

Excellent understanding of enterprise data warehousing principles of ETL/ELT and the tools/technologies used to implement them Proven experience with Apache Airflow, including designing and maintaining complex workflows.

Strong knowledge of Python programming. Experience with SQL and relational databases. Familiarity with cloud platforms (e.g., AWS) and their data services. Experience with version control systems such as Git. Strong problem-solving skills and attention to detail. Excellent communication and collaboration skills. Be an expert in handling ECS, Fargate, EC2, EKS infrastructure hosting Airflow. Be an expert in DBT including handling Infrastructure related upgrades and optimising DBT code. BenefitsYou'll love it because...

  • You will have the opportunity to work in a successful, fast-moving mission-driven company
  • You will have autonomy and growth opportunities
  • You will collaborate with other brilliant minds
  • You will always be surrounded by new challenges from clients, consultants and technology

  • Senior Data Engineer

    1 week ago

    London, Greater London, United Kingdom Acrotrend Full time

    About Acrotrend Acrotrend is a leading customer insight and analytics consultancy, trusted by renowned brands to enhance customer acquisition, retention, and lifetime value (CLTV). We leverage cutting-edge technologies in data engineering and AI to deliver actionable insights and drive impactful results for our clients. The Role: Senior Data Engineer -...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom SoundCloud Ltd Full time

    SoundCloud Ltd empowers artists and fans to connect and share through music. Founded in 2007, SoundCloud is an artist-first platform empowering artists to build and grow their careers by providing them with the most progressive tools, services, and resources. With over 375 million tracks from 40 million artists, the future of music is SoundCloud. We are...

  • Senior Data Engineer

    4 days ago

    London, Greater London, United Kingdom Simple Machines Full time

    Exciting Opportunity at a Leading Technology FirmWe are seeking a highly skilled Senior Data Engineer to join our team at Simple Machines, a dynamic technology company that specializes in data engineering and software solutions. As a Senior Data Engineer, you will play a crucial role in crafting innovative data products and developing efficient data...

  • Senior Data Engineer

    4 days ago

    London, Greater London, United Kingdom Dext Full time

    About DextDext is a leading fintech SaaS product company that empowers accountants and bookkeepers with innovative technology solutions to make businesses better.Our suite of tools makes accountants more productive, profitable, and powerful, giving them back the most precious commodity - time - which they can then use to add greater value to their...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom HEAD OFFICE Full time

    What's the role about?As the UK's most effective omni-channel retailer, E-Commerce is a key part of Our Business - we run two of the busiest websites in the UK, driving a significant proportion of our sales. Our Engineering practice includes 60+ teams, working on everything from Cloud Platforms to Mobile Apps, from E-Commerce functionality to Machine...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Partnerize Full time

    Who We Are:Partnerize is a performance marketing technology company that empowers marketers to discover, engage, and convert audiences at scale. Our platform leverages automation and data-driven insights to deliver exceptional results for our clients.Why Partnerize?At Partnerize, we believe in fostering a culture of collaboration, innovation, and growth. We...

  • Python Data Engineer PySpark Spark Finance Trading London

    1 month ago

    London, Greater London, United Kingdom Joseph Harry Full time

    Python Data Engineer (Software Engineer Programmer Developer Data Engineer Python PySpark Spark Glue Athena Iceberg Airflow Dagster DBT Java Agile AWS GCP Buy Side Asset Manager Investment Management Finance Front Office Trading Financial Services Pandas Numpy Scipy Banking) required by our asset management client in London.Python Data Engineer (Software...

  • Data Engineering Specialist

    2 weeks ago

    London, Greater London, United Kingdom Expedia Group Full time

    Are you passionate about data science and building efficient data pipelines to uncover valuable insights from massive datasets?At Expedia Group, our Data Science Engineering team thrives on this challenge. We collaborate closely with Data Scientists to leverage AI/ML technologies, transforming Expedia's vast data resources into intelligent, real-time...

  • Senior Data Architect

    1 week ago

    London, Greater London, United Kingdom CAIS Full time

    CAIS is a leading provider of access to alternative investments and structured notes for independent advisors, asset managers, and bank issuers. We are seeking a highly skilled Senior Data Engineer to join our dynamic technology team.In this role, you will play a pivotal part in developing and maintaining the robust data infrastructure that powers our...

  • Senior Data Architect

    3 days ago

    London, Greater London, United Kingdom Adecco Full time

    Job Title: Senior Data EngineerJob Summary:Adecco is seeking an experienced Senior Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing and developing data pipelines, working with upstream digital teams to accelerate bringing additional microservice data onto our Analytical Platform.Key Responsibilities:Design and...

  • Senior Data Architect

    2 days ago

    London, Greater London, United Kingdom Talent Full time £100,000

    About the RoleWe are seeking a highly skilled Senior Data Architect to play a key role in developing and maintaining data pipelines, ETL processes, and architectures as well as building and managing data warehouses for our rapidly growing FinTech company.Key ResponsibilitiesDesign and construct cloud-based data engineering solutions using AWS.Develop and...

  • Senior Data Engineer

    4 days ago

    London, Greater London, United Kingdom Solirius Consulting Full time

    About Us:Solirius Consulting is a leading provider of technical consultancy and application delivery services. Our mission is to help our clients solve complex business problems and stay ahead in the ever-changing technological landscape.We pride ourselves on our collaborative approach, embedding our consultants into our clients' businesses to provide a...

  • Senior Data Engineer

    1 month ago

    London, Greater London, United Kingdom Amicus Full time

    Job DescriptionData Engineer – Up to £100,000 – London Based (Hybrid 2 Days a week)I have a new opportunity for a ML Data Engineer to join a dynamic tech business based in Central LondonCompanyYou will be joining the UK division of US SAAS platform operating within in the hospitality sector. Up until the start of this year they were an early stage...

  • Data Engineer

    1 week ago

    London, Greater London, United Kingdom Pepperjam Full time

    As a Data Engineer at Pepperjam, you will play a pivotal role in shaping our data infrastructure and empowering data-driven decision making across the organization. Your responsibilities will encompass: Designing, building, and maintaining robust data pipelines leveraging Google Cloud Platform, Google BigQuery, Apache Airflow, Airbyte, HDFS, Kafka, and...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom E1 EDF Trading Ltd Full time

    About E1 EDF Trading Ltd E1 EDF Trading Ltd is a leading player in the energy trading sector, leveraging data-driven insights to make informed decisions and navigate the complexities of the global energy market. Our dedicated Data Insights team plays a crucial role in transforming raw data into actionable intelligence, empowering our organization to optimize...

  • Senior Data Platform Engineer

    3 days ago

    London, Greater London, United Kingdom Schroders Full time

    About the RoleWe are seeking a highly skilled Senior Data Platform Engineer to join our Enterprise Data team at Schroders. As a key member of our team, you will be responsible for establishing and integrating innovative technologies into our data platform services, serving our data practitioners, including data engineering and data science.Key...

  • Senior Data Infrastructure Engineer

    2 days ago

    London, Greater London, United Kingdom TRM-International Full time

    TRM-International: Data Infrastructure EngineerWe are seeking a skilled Data Infrastructure Engineer to join our team at a leading Wholesale financial Institute. As a key member of our organization, you will play a crucial role in developing and maintaining our enterprise Data Platform.Key Responsibilities:Develop and Maintain Data Infrastructure: Design,...

  • Senior Software Engineer

    4 days ago

    London, Greater London, United Kingdom Dext Full time

    About DextDext is a leading fintech SaaS product company that empowers accountants and bookkeepers with innovative technology solutions to make businesses better. For over a decade, we've been a trusted partner, providing tools that make our partners more productive, profitable, and powerful.Our MissionWe're on a mission to give accountants and bookkeepers...

  • Senior Data Platform Engineer

    3 days ago

    London, Greater London, United Kingdom Schroders Full time

    About the RoleWe are seeking a highly skilled Senior Data Platform Engineer to join our Enterprise Data team at Schroders. As a key member of our team, you will be responsible for establishing and integrating innovative technologies into our data platform services, serving our data practitioners, including data engineering and data science.Key...

  • Senior Data Architect

    4 days ago

    London, Greater London, United Kingdom Time Out Full time

    Job Title: Senior Data ArchitectLocation: Remote (with occasional visits to our London office)Department: Product & EngineeringReporting to: Scrum MasterRole OverviewWe are seeking a highly skilled Data Architect to join our team and play a key role in designing and maintaining our data infrastructure on Google Cloud Platform (GCP) with a focus on BigQuery....