Lead Big Data Software Engineer

3 months ago


Belfast, United Kingdom Rapid7 Full time

About the Team

The Rapid7 Data Platform is a unified, integrated platform powered by Rapid7’s product suite providing our customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities. 

Our teams are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers. This role is crucial to the platform’s success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.

About the Role

We are seeking an innovative, self-motivated Data and Performance Engineer who will act as a technical leader to collaborate with our product teams to optimise their data pipelines and retrieval processes for performance and efficiency. You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimisations.

Technologies you will work with:

Trino

Iceberg

Parquet

Spark

Airflow

Kafka

AWS services such as Glue, S3, EKS

In this role, you will:

Analyse and optimise distributed SQL queries to improve performance

Suggest optimisations to our data pipelines

Provide recommendations for efficient partitioning strategies and schema designs

Conduct performance tuning for the data pipelines and queries

Develop performance monitoring strategies and tools

The skills you'll bring include:

5+ years of hands-on software engineering experience, with a specific focus on database query optimization

Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design

Experience in continuously monitoring and optimising data pipelines for performance and cost-effectiveness

Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams

Skills to analyse and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements

Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation

Excellent verbal and written communication skills.

Strong, creative problem solving ability.

Nice to haves:

Trino/Presto data-mesh

AWS, Terraform, Kubernetes

Java

Kafka

We believe the best ideas and solutions come from diverse teams. If you're excited about this role and feel your experience can make an impact, don't hesitate – apply today



  • Belfast, United Kingdom Corvus Full time

    Job Description: Lead Big Data Solutions ArchitectWe are looking for an experienced Lead Big Data Solutions Architect to join our team at Corvus in Belfast. As a key member of our team, you will be responsible for designing, building, and delivering large-scale data solutions using cutting-edge technologies.About the Role:You will lead the development of...

  • Data Engineer

    2 weeks ago


    Belfast, United Kingdom Search 5.0 Full time £45,000 - £60,000

    At Search 5.0, we're seeking a talented Data Engineer to join our Belfast office. As a key member of our team, you'll be responsible for designing and deploying production data pipelines in big data architectures.We're looking for someone with expertise in technologies such as Python, SQL, AWS / Azure, and knowledge of data cleaning, wrangling,...


  • Belfast, United Kingdom Citigroup, Inc. Full time

    About the JobCitigroup, Inc. is seeking a highly skilled Principle Data Engineer to join its Technology team. This role involves designing and developing data engineering solutions using Big Data technologies. You will be responsible for providing technical walk-throughs to various stakeholders, working independently as well as within a team, and...


  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the TeamOur teams at Divvy Cloud Corp. are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers.This role is critical to the team's success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use...


  • Belfast, United Kingdom PA Consulting Services Ltd Full time

    Company Overview:P.A Consulting Services Ltd is a leading global consulting firm that helps businesses drive innovation and transformation. With a diverse team of experts, we deliver cutting-edge digital solutions to complex business challenges. Our commitment to diversity, inclusion, and employee growth enables us to bring ingenuity to life and deliver...


  • Belfast, United Kingdom Data Intellect Full time

    Role OverviewData Engineering ProfessionalAbout Data IntellectWe are a leading provider of data-driven solutions, utilizing cutting-edge technology to deliver results-driven outcomes.Culture and ValuesOur team is built on core values of innovation, collaboration, and customer satisfaction, driving our pursuit of excellence.Job ResponsibilitiesIn this role,...


  • Belfast, United Kingdom MCS Group Full time

    MCS Group is proud to partner with industry-leading consultancies that drive innovation and change. In Belfast, one such consultancy is seeking a talented Lead Data Engineer to join their growing data team.This is an exceptional opportunity to contribute to the development of seamless user experiences by leading teams to deliver functioning client...


  • Belfast, United Kingdom Divvy Cloud Corp. Full time

    About the PositionWe are seeking a highly skilled Data and Performance Engineer to join our team at Divvy Cloud Corp. in Staff Big Data Software Engineer. The successful candidate will be responsible for collaborating with our product teams to optimize their data pipelines and retrieval processes for performance and efficiency.Responsibilities:Analyse and...


  • Belfast, United Kingdom Citigroup, Inc. Full time

    Job DescriptionCitigroup, Inc. is a global leader in financial services, offering a broad range of products and services to consumers, corporations, governments, and institutions worldwide. We are currently seeking an experienced Senior Data Engineering Specialist to join our Applications Development team.This senior-level position involves designing and...

  • Big Data Engineer

    4 weeks ago


    Belfast, United Kingdom Laraveldaily Full time

    Laraveldaily is seeking a skilled Big Data Engineer to join our team. As a permanent, hybrid employee, you will play a critical role in maintaining and developing our existing data platform, as well as building new ETL pipelines and contributing to data quality initiatives.About the RoleThis exciting opportunity offers the chance to work on challenging data...

  • Senior Data Engineer

    7 hours ago


    Belfast, United Kingdom Data Intellect Full time

    About the RoleWe are seeking a skilled Senior Data Engineer to join our team at Data Intellect. As a key member of our engineering department, you will be responsible for designing and implementing scalable data solutions that meet the needs of our clients.ResponsibilitiesDesign and Develop Data Solutions: Collaborate with cross-functional teams to identify...


  • Belfast, United Kingdom Search 5.0 Full time

    Data Pipeline ExpertiseBelfast-based Search 5.0 is seeking multiple senior data engineers to join their team. This innovative company tackles complex challenges and achieves results across various industries, including healthcare, energy, government, financial services, and consumer products. With a truly collaborative culture, employees are encouraged to...

  • Python Engineer

    4 weeks ago


    Belfast, United Kingdom Data Intellect Full time

    Job Description What you'll be doing: · Build and enhance bespoke Python & Data Engineering solutions for our clients · Enhance and grow your data engineering knowledge among subject matter experts in our learning and development ecosystem · Deliver high quality, best practice software solutions · Use a range of architectures, tools and frameworks ·...


  • Belfast, United Kingdom Citi Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team in this exciting new role. As a key member of our engineering team, you will be responsible for designing and implementing new data engineering solutions using Big Data Technologies.Key ResponsibilitiesDesign & develop Data engineering solutions using Big Data...


  • Belfast, United Kingdom Data Intellect Full time

    About UsData Intellect is a leading data and technology consultancy specializing in creating cutting-edge financial and capital markets technology solutions. Our expertise extends to diverse industries, including smart energy and healthcare.We turn big data problems into smart data solutions by utilizing our skills in Q programming language and kdb+ time...


  • Belfast, United Kingdom Tenth Revolution Group Full time

    We are seeking a Big Data Integration Specialist to join our team at the Tenth Revolution Group.Job Summary:You will be responsible for integrating big data solutions using Snowflake, ensuring seamless data flow and optimal performance.You will work closely with cross-functional teams to design and implement data architectures that meet our clients' business...


  • Belfast, United Kingdom Kainos Full time

    Discover a challenging opportunity as a Senior Data Engineer at Kainos, where you will design and develop data processing software components for solutions handling large datasets.We are seeking an experienced professional to join our agile team, providing strong development leadership and taking responsibility for significant technical components of data...


  • Belfast, United Kingdom aPriori Full time

    Job DescriptionThe ideal candidate for this Data Engineer role will have a solid background in data engineering, software development, and data management. They will be responsible for designing, implementing, and maintaining data products and data platform capabilities, ensuring data integrity, quality, and efficiency.Key Requirements- Experience with cloud...

  • Software Engineer

    4 weeks ago


    Belfast, United Kingdom Bright Software Group Full time

    Job description Who are we??At Bright, weve engineered cutting-edge software for accounting, payroll, tax, and practice management.We've assembled a team of top talent and stand ready to lead the industry with our superior software solutions and unparalleled customer support.?We're brilliant people creating brilliant software!


  • Belfast, United Kingdom Ocho Full time

    **About the Role**We are seeking an experienced Software Engineering Team Lead to join our dynamic team at Ocho, a leading local tech success story with over 400,000 clients. As a key member of our engineering team, you will be responsible for guiding a small team of engineers, overseeing technical direction, mentoring team members, and delivering...