Lead Big Data Software Engineer
3 months ago
About the Team
The Rapid7 Data Platform is a unified, integrated platform powered by Rapid7’s product suite providing our customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities.
Our teams are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers. This role is crucial to the platform’s success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.
About the Role
We are seeking an innovative, self-motivated Data and Performance Engineer who will act as a technical leader to collaborate with our product teams to optimise their data pipelines and retrieval processes for performance and efficiency. You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimisations.
Technologies you will work with:
Trino
Iceberg
Parquet
Spark
Airflow
Kafka
AWS services such as Glue, S3, EKS
In this role, you will:
Analyse and optimise distributed SQL queries to improve performance
Suggest optimisations to our data pipelines
Provide recommendations for efficient partitioning strategies and schema designs
Conduct performance tuning for the data pipelines and queries
Develop performance monitoring strategies and tools
The skills you'll bring include:
5+ years of hands-on software engineering experience, with a specific focus on database query optimization
Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design
Experience in continuously monitoring and optimising data pipelines for performance and cost-effectiveness
Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams
Skills to analyse and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements
Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation
Excellent verbal and written communication skills.
Strong, creative problem solving ability.
Nice to haves:
Trino/Presto data-mesh
AWS, Terraform, Kubernetes
Java
Kafka
We believe the best ideas and solutions come from diverse teams. If you're excited about this role and feel your experience can make an impact, don't hesitate – apply today
-
Lead Big Data Solutions Architect
1 week ago
Belfast, United Kingdom Corvus Full timeJob Description: Lead Big Data Solutions ArchitectWe are looking for an experienced Lead Big Data Solutions Architect to join our team at Corvus in Belfast. As a key member of our team, you will be responsible for designing, building, and delivering large-scale data solutions using cutting-edge technologies.About the Role:You will lead the development of...
-
Data Engineer
2 weeks ago
Belfast, United Kingdom Search 5.0 Full time £45,000 - £60,000At Search 5.0, we're seeking a talented Data Engineer to join our Belfast office. As a key member of our team, you'll be responsible for designing and deploying production data pipelines in big data architectures.We're looking for someone with expertise in technologies such as Python, SQL, AWS / Azure, and knowledge of data cleaning, wrangling,...
-
Big Data Technical Lead
2 weeks ago
Belfast, United Kingdom Citigroup, Inc. Full timeAbout the JobCitigroup, Inc. is seeking a highly skilled Principle Data Engineer to join its Technology team. This role involves designing and developing data engineering solutions using Big Data technologies. You will be responsible for providing technical walk-throughs to various stakeholders, working independently as well as within a team, and...
-
Big Data Software Engineering Specialist
4 weeks ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the TeamOur teams at Divvy Cloud Corp. are responsible for consolidating data from all Rapid7 products, transforming it for optimised retrieval, and ensuring high-performance and seamless access to our customers.This role is critical to the team's success as it focuses on building a highly scalable and reliable data mesh that powers cross-product use...
-
Big Data Pipeline Developer
2 weeks ago
Belfast, United Kingdom PA Consulting Services Ltd Full timeCompany Overview:P.A Consulting Services Ltd is a leading global consulting firm that helps businesses drive innovation and transformation. With a diverse team of experts, we deliver cutting-edge digital solutions to complex business challenges. Our commitment to diversity, inclusion, and employee growth enables us to bring ingenuity to life and deliver...
-
Data Engineering Professional
2 weeks ago
Belfast, United Kingdom Data Intellect Full timeRole OverviewData Engineering ProfessionalAbout Data IntellectWe are a leading provider of data-driven solutions, utilizing cutting-edge technology to deliver results-driven outcomes.Culture and ValuesOur team is built on core values of innovation, collaboration, and customer satisfaction, driving our pursuit of excellence.Job ResponsibilitiesIn this role,...
-
Cloud Data Engineer Lead
4 hours ago
Belfast, United Kingdom MCS Group Full timeMCS Group is proud to partner with industry-leading consultancies that drive innovation and change. In Belfast, one such consultancy is seeking a talented Lead Data Engineer to join their growing data team.This is an exceptional opportunity to contribute to the development of seamless user experiences by leading teams to deliver functioning client...
-
Big Data and Performance Optimization Expert
2 weeks ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the PositionWe are seeking a highly skilled Data and Performance Engineer to join our team at Divvy Cloud Corp. in Staff Big Data Software Engineer. The successful candidate will be responsible for collaborating with our product teams to optimize their data pipelines and retrieval processes for performance and efficiency.Responsibilities:Analyse and...
-
Lead Big Data Solutions Architect
2 weeks ago
Belfast, United Kingdom Citigroup, Inc. Full timeJob DescriptionCitigroup, Inc. is a global leader in financial services, offering a broad range of products and services to consumers, corporations, governments, and institutions worldwide. We are currently seeking an experienced Senior Data Engineering Specialist to join our Applications Development team.This senior-level position involves designing and...
-
Big Data Engineer
4 weeks ago
Belfast, United Kingdom Laraveldaily Full timeLaraveldaily is seeking a skilled Big Data Engineer to join our team. As a permanent, hybrid employee, you will play a critical role in maintaining and developing our existing data platform, as well as building new ETL pipelines and contributing to data quality initiatives.About the RoleThis exciting opportunity offers the chance to work on challenging data...
-
Senior Data Engineer
7 hours ago
Belfast, United Kingdom Data Intellect Full timeAbout the RoleWe are seeking a skilled Senior Data Engineer to join our team at Data Intellect. As a key member of our engineering department, you will be responsible for designing and implementing scalable data solutions that meet the needs of our clients.ResponsibilitiesDesign and Develop Data Solutions: Collaborate with cross-functional teams to identify...
-
Big Data Transformation Expert
2 weeks ago
Belfast, United Kingdom Search 5.0 Full timeData Pipeline ExpertiseBelfast-based Search 5.0 is seeking multiple senior data engineers to join their team. This innovative company tackles complex challenges and achieves results across various industries, including healthcare, energy, government, financial services, and consumer products. With a truly collaborative culture, employees are encouraged to...
-
Python Engineer
4 weeks ago
Belfast, United Kingdom Data Intellect Full timeJob Description What you'll be doing: · Build and enhance bespoke Python & Data Engineering solutions for our clients · Enhance and grow your data engineering knowledge among subject matter experts in our learning and development ecosystem · Deliver high quality, best practice software solutions · Use a range of architectures, tools and frameworks ·...
-
Senior Data Engineer Lead
1 week ago
Belfast, United Kingdom Citi Full timeAbout the RoleWe are seeking a highly skilled Senior Data Engineer to join our team in this exciting new role. As a key member of our engineering team, you will be responsible for designing and implementing new data engineering solutions using Big Data Technologies.Key ResponsibilitiesDesign & develop Data engineering solutions using Big Data...
-
Financial Software Expert
3 weeks ago
Belfast, United Kingdom Data Intellect Full timeAbout UsData Intellect is a leading data and technology consultancy specializing in creating cutting-edge financial and capital markets technology solutions. Our expertise extends to diverse industries, including smart energy and healthcare.We turn big data problems into smart data solutions by utilizing our skills in Q programming language and kdb+ time...
-
Big Data Integration Specialist
1 week ago
Belfast, United Kingdom Tenth Revolution Group Full timeWe are seeking a Big Data Integration Specialist to join our team at the Tenth Revolution Group.Job Summary:You will be responsible for integrating big data solutions using Snowflake, ensuring seamless data flow and optimal performance.You will work closely with cross-functional teams to design and implement data architectures that meet our clients' business...
-
Belfast, United Kingdom Kainos Full timeDiscover a challenging opportunity as a Senior Data Engineer at Kainos, where you will design and develop data processing software components for solutions handling large datasets.We are seeking an experienced professional to join our agile team, providing strong development leadership and taking responsibility for significant technical components of data...
-
Cloud Data Engineering Lead
2 weeks ago
Belfast, United Kingdom aPriori Full timeJob DescriptionThe ideal candidate for this Data Engineer role will have a solid background in data engineering, software development, and data management. They will be responsible for designing, implementing, and maintaining data products and data platform capabilities, ensuring data integrity, quality, and efficiency.Key Requirements- Experience with cloud...
-
Software Engineer
4 weeks ago
Belfast, United Kingdom Bright Software Group Full timeJob description Who are we??At Bright, weve engineered cutting-edge software for accounting, payroll, tax, and practice management.We've assembled a team of top talent and stand ready to lead the industry with our superior software solutions and unparalleled customer support.?We're brilliant people creating brilliant software!
-
Software Engineering Lead
7 days ago
Belfast, United Kingdom Ocho Full time**About the Role**We are seeking an experienced Software Engineering Team Lead to join our dynamic team at Ocho, a leading local tech success story with over 400,000 clients. As a key member of our engineering team, you will be responsible for guiding a small team of engineers, overseeing technical direction, mentoring team members, and delivering...