Data Engineer for Large-Scale Data Platforms

7 hours ago


Belfast, United Kingdom Divvy Cloud Corp. Full time

About the Role

">

At Divvy Cloud Corp., we are seeking a highly skilled Data and Performance Engineer to join our team. As a technical leader, you will collaborate with product teams to optimize their data pipelines and retrieval processes for performance and efficiency.

">

You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimizations. Your expertise in distributed SQL query optimization, database system design, and schema optimization will be invaluable in building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.

">

The technologies you will work with:

">
  • Trino
  • Iceberg
  • Parquet
  • Spark
  • Airflow
  • Kafka
  • AWS services such as Glue, S3, EKS
">

Your responsibilities:

">
  • Analyse and optimize distributed SQL queries to improve performance
  • Suggest optimisations to our data pipelines
  • Provide recommendations for efficient partitioning strategies and schema designs
  • Conduct performance tuning for the data pipelines and queries
  • Develop performance monitoring strategies and tools
">

What you'll need:

">
  • 5+ years of hands-on software engineering experience, with a specific focus on database query optimization
  • Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design
  • Experience in continuously monitoring and optimizing data pipelines for performance and cost-effectiveness
  • Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams
  • Skills to analyze and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements
  • Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation
  • Excellent verbal and written communication skills
  • Strong, creative problem-solving ability
">

Estimated salary: $140,000 - $180,000 per year

">

We believe the best ideas and solutions come from diverse teams.

">

About Rapid7

">

Rapid7 is creating a more secure digital future for all by helping organizations strengthen their security programs in the face of accelerating digital transformation. Our portfolio of best-in-class solutions empowers security professionals to manage risk and eliminate threats across the entire threat landscape from apps to the cloud to traditional infrastructure to the dark web. We foster open source communities and cutting-edge research–using these insights to optimize our products and arm the global security community with the latest in attacker methods. Trusted by more than 10,000 customers worldwide, our industry-leading solutions and services help businesses stay ahead of attackers, ahead of the competition, and future-ready for what's next.

">

  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the RoleWe are seeking an innovative and self-motivated Data and Performance Engineer to collaborate with our product teams in Divvy Cloud Corp. to optimise data pipelines and retrieval processes for performance and efficiency.This role is crucial to the platform's success as it focuses on building a highly scalable and reliable data mesh that powers...


  • Belfast, United Kingdom Kainos Full time

    Discover a challenging opportunity as a Senior Data Engineer at Kainos, where you will design and develop data processing software components for solutions handling large datasets.We are seeking an experienced professional to join our agile team, providing strong development leadership and taking responsibility for significant technical components of data...

  • Data Engineer

    4 weeks ago


    Belfast, United Kingdom Data Intellect Full time

    Job OverviewWe are seeking a highly skilled Data Engineer to join our team at Data Intellect. As a Data Engineer, you will be responsible for designing, developing, and implementing data-driven solutions that transform our clients' businesses.Key ResponsibilitiesDesign and Develop Data-Driven SolutionsCollaborate with Cross-Functional TeamsImplement and...


  • Belfast, United Kingdom Rapid7 Full time

    About the RoleAs a Principal Engineer on our Data Platform Engineering team, you will be responsible for architecting and scaling streaming and batch data pipelines. You will design the CI/CD infrastructure that ensures efficient development and deployment of data services. You will play a key role in shaping the architecture of our data platform,...


  • Belfast, United Kingdom Version 1 Full time

    Job DescriptionWe are seeking a highly skilled Data Engineer to join our team at Version 1. The successful candidate will be responsible for designing and implementing data ingestion and transformation pipelines for large-scale organisations. This is an exciting opportunity to work with cutting-edge technologies and contribute to the development of...

  • Data Engineer

    2 months ago


    Belfast, United Kingdom Data Intellect Full time

    Job DescriptionWhat you’ll be doing:Build and enhance bespoke Data Science and Engineering solutions for our clientsEnhance and grow your data science and engineering knowledge among subject matter experts in our learning and development ecosystem Deliver high quality, best practice software solutions Use a range of architectures, tools and...


  • Belfast, United Kingdom Rapid7 Full time

    As a Principal Engineer, you’ll get the opportunity to be a hands-on engineer, learning best practice engineering processes and approaches whilst receiving ongoing development through coaching, mentoring and pairing with other engineers on your team. From problem-solving to challenging old ways of thinking, you will have the opportunity to unleash your...


  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    Data is the backbone of any successful business, and at Divvy Cloud Corp., we understand the importance of having a robust data infrastructure.We are seeking a skilled Data Platform Solutions Architect to help us design and implement large-scale data lakes using Apache Iceberg, manage high-throughput data architectures, and ensure efficient data partitioning...


  • Belfast, United Kingdom Data Intellect Full time

    Job DescriptionWe are seeking a highly skilled Data Scientist and Engineer to join our team at Data Intellect. As a key member of our data science and engineering team, you will be responsible for designing, developing, and implementing data-driven solutions for our clients.Key Responsibilities:Design and develop bespoke data science and engineering...


  • Belfast, United Kingdom Data Intellect Full time

    Role OverviewData Engineering ProfessionalAbout Data IntellectWe are a leading provider of data-driven solutions, utilizing cutting-edge technology to deliver results-driven outcomes.Culture and ValuesOur team is built on core values of innovation, collaboration, and customer satisfaction, driving our pursuit of excellence.Job ResponsibilitiesIn this role,...

  • Lead Data Engineer

    4 weeks ago


    Belfast, United Kingdom Divvy Cloud Corp. Full time

    Job SummaryWe are seeking a Lead Data Engineer to join our Data Platform Engineering team at Divvy Cloud Corp. As a key member of the team, you will be responsible for leading the design and implementation of scalable data pipelines using Apache Kafka, Apache Iceberg, and Apache Spark.You will work closely with cross-functional teams to deliver highly...


  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the TeamThe Rapid7 Data Platform is a unified, integrated platform that provides customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities.Job Description:We are looking for an experienced Data and Performance Engineer to collaborate with our product teams to optimize their data...


  • Belfast, United Kingdom Ocho Full time

    Staff Data Engineer Location: Belfast (Hybrid)About the RoleOcho is a leading company in AI development and we are seeking a skilled Data Engineer to join our team in Belfast. The successful candidate will design, build and optimise data pipelines and architectures to support AI applications.Key Responsibilities include collaborating with data scientists, ML...

  • Data Engineer

    4 months ago


    Belfast, United Kingdom Rapid7 Full time

    Data Engineer II Rapid7 seeks a highly motivated and inquisitive aspiring Data Engineer II to join our quickly scaling data engineering function. Come and join our efforts in unlocking the value of data through industry-leading innovation, cutting edge modern tooling, democratization at scale and building exceptional and trusted data products for the...


  • Belfast, United Kingdom ESO Full time

    In this role as a Data Platform Engineer at ESO, you'll play a vital part in building and maintaining our robust data platform. You'll collaborate closely with cross-functional teams to deliver scalable, efficient, and secure solutions that cater to the diverse needs of our customers.To succeed in this position, you'll need a deep understanding of data...


  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Divvy Cloud Corp. in Staff Big Data Software Engineer. The successful candidate will be responsible for designing, developing, and implementing high-performance data platforms using big data technologies such as Trino, Iceberg, Parquet, Spark, Airflow, Kafka, and AWS...

  • Senior Data Architect

    4 weeks ago


    Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the RoleThe Principal Engineer position is a key member of our Data Platform Engineering team at Divvy Cloud Corp. In this role, you will be responsible for designing and implementing scalable data pipelines using Apache Kafka, Apache Iceberg, and Apache Spark.You will work closely with cross-functional teams to deliver highly available, performant,...


  • Belfast, United Kingdom ESO Full time

    Are you a seasoned software developer looking for a new challenge? Do you have experience with front-end development using Angular and Typescript, as well as back-end development skills in C# .NET? If so, we may have the perfect opportunity for you!Our Data Platform team at ESO is seeking a skilled Data Platform Engineer to join our team, working closely...


  • Belfast, United Kingdom Search 5.0 Full time £75,000 - £90,000

    Data Leadership OpportunityWe are looking for an experienced Data Engineering Manager to lead our data function in Belfast. As a key member of our team, you will be responsible for designing and building large-scale data solutions.The successful candidate will have a proven track record in leading teams to deliver complex data projects. They should have...


  • Belfast, United Kingdom aPriori Full time

    About the RoleWe are seeking a highly skilled Cloud Platform Data Engineer to join our team at aPriori, a leading provider of digital transformation solutions for manufacturers. This is an exciting opportunity to work on groundbreaking projects, driving innovation and efficiency in the industry.Job SummaryThe Cloud Platform Data Engineer will be responsible...