Data Engineer for Large-Scale Data Platforms
7 hours ago
About the Role
">At Divvy Cloud Corp., we are seeking a highly skilled Data and Performance Engineer to join our team. As a technical leader, you will collaborate with product teams to optimize their data pipelines and retrieval processes for performance and efficiency.
">You will work with the Data Platform teams to implement monitoring and testing strategies to ensure the performance of the data and their queries as well as identify optimizations. Your expertise in distributed SQL query optimization, database system design, and schema optimization will be invaluable in building a highly scalable and reliable data mesh that powers cross-product use cases through a distributed query engine for big data analytics.
">The technologies you will work with:
">- Trino
- Iceberg
- Parquet
- Spark
- Airflow
- Kafka
- AWS services such as Glue, S3, EKS
Your responsibilities:
">- Analyse and optimize distributed SQL queries to improve performance
- Suggest optimisations to our data pipelines
- Provide recommendations for efficient partitioning strategies and schema designs
- Conduct performance tuning for the data pipelines and queries
- Develop performance monitoring strategies and tools
What you'll need:
">- 5+ years of hands-on software engineering experience, with a specific focus on database query optimization
- Strong database system expertise in query execution planning, query optimization, performance tuning, parallel computing, and schema design
- Experience in continuously monitoring and optimizing data pipelines for performance and cost-effectiveness
- Ability to design, develop, implement, and operate highly reliable large-scale data lake systems in cooperation with product teams
- Skills to analyze and performance test the data mesh performance and scalability, identify bottlenecks, recommend and develop improvements
- Mentorship and guidance of junior engineers, providing technical leadership and fostering a culture of continuous improvement and innovation
- Excellent verbal and written communication skills
- Strong, creative problem-solving ability
Estimated salary: $140,000 - $180,000 per year
">We believe the best ideas and solutions come from diverse teams.
">About Rapid7
">Rapid7 is creating a more secure digital future for all by helping organizations strengthen their security programs in the face of accelerating digital transformation. Our portfolio of best-in-class solutions empowers security professionals to manage risk and eliminate threats across the entire threat landscape from apps to the cloud to traditional infrastructure to the dark web. We foster open source communities and cutting-edge research–using these insights to optimize our products and arm the global security community with the latest in attacker methods. Trusted by more than 10,000 customers worldwide, our industry-leading solutions and services help businesses stay ahead of attackers, ahead of the competition, and future-ready for what's next.
">-
Data Engineer for Large Scale Data Lakes
2 weeks ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the RoleWe are seeking an innovative and self-motivated Data and Performance Engineer to collaborate with our product teams in Divvy Cloud Corp. to optimise data pipelines and retrieval processes for performance and efficiency.This role is crucial to the platform's success as it focuses on building a highly scalable and reliable data mesh that powers...
-
Belfast, United Kingdom Kainos Full timeDiscover a challenging opportunity as a Senior Data Engineer at Kainos, where you will design and develop data processing software components for solutions handling large datasets.We are seeking an experienced professional to join our agile team, providing strong development leadership and taking responsibility for significant technical components of data...
-
Data Engineer
4 weeks ago
Belfast, United Kingdom Data Intellect Full timeJob OverviewWe are seeking a highly skilled Data Engineer to join our team at Data Intellect. As a Data Engineer, you will be responsible for designing, developing, and implementing data-driven solutions that transform our clients' businesses.Key ResponsibilitiesDesign and Develop Data-Driven SolutionsCollaborate with Cross-Functional TeamsImplement and...
-
Data Platform Architect
4 weeks ago
Belfast, United Kingdom Rapid7 Full timeAbout the RoleAs a Principal Engineer on our Data Platform Engineering team, you will be responsible for architecting and scaling streaming and batch data pipelines. You will design the CI/CD infrastructure that ensures efficient development and deployment of data services. You will play a key role in shaping the architecture of our data platform,...
-
Belfast, United Kingdom Version 1 Full timeJob DescriptionWe are seeking a highly skilled Data Engineer to join our team at Version 1. The successful candidate will be responsible for designing and implementing data ingestion and transformation pipelines for large-scale organisations. This is an exciting opportunity to work with cutting-edge technologies and contribute to the development of...
-
Data Engineer
2 months ago
Belfast, United Kingdom Data Intellect Full timeJob DescriptionWhat you’ll be doing:Build and enhance bespoke Data Science and Engineering solutions for our clientsEnhance and grow your data science and engineering knowledge among subject matter experts in our learning and development ecosystem Deliver high quality, best practice software solutions Use a range of architectures, tools and...
-
Principal Software Engineer
2 months ago
Belfast, United Kingdom Rapid7 Full timeAs a Principal Engineer, you’ll get the opportunity to be a hands-on engineer, learning best practice engineering processes and approaches whilst receiving ongoing development through coaching, mentoring and pairing with other engineers on your team. From problem-solving to challenging old ways of thinking, you will have the opportunity to unleash your...
-
Data Platform Solutions Architect
6 hours ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeData is the backbone of any successful business, and at Divvy Cloud Corp., we understand the importance of having a robust data infrastructure.We are seeking a skilled Data Platform Solutions Architect to help us design and implement large-scale data lakes using Apache Iceberg, manage high-throughput data architectures, and ensure efficient data partitioning...
-
Data Scientist and Engineer
2 months ago
Belfast, United Kingdom Data Intellect Full timeJob DescriptionWe are seeking a highly skilled Data Scientist and Engineer to join our team at Data Intellect. As a key member of our data science and engineering team, you will be responsible for designing, developing, and implementing data-driven solutions for our clients.Key Responsibilities:Design and develop bespoke data science and engineering...
-
Data Engineering Professional
2 days ago
Belfast, United Kingdom Data Intellect Full timeRole OverviewData Engineering ProfessionalAbout Data IntellectWe are a leading provider of data-driven solutions, utilizing cutting-edge technology to deliver results-driven outcomes.Culture and ValuesOur team is built on core values of innovation, collaboration, and customer satisfaction, driving our pursuit of excellence.Job ResponsibilitiesIn this role,...
-
Lead Data Engineer
4 weeks ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeJob SummaryWe are seeking a Lead Data Engineer to join our Data Platform Engineering team at Divvy Cloud Corp. As a key member of the team, you will be responsible for leading the design and implementation of scalable data pipelines using Apache Kafka, Apache Iceberg, and Apache Spark.You will work closely with cross-functional teams to deliver highly...
-
High-Performance Data Platform Specialist
50 minutes ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the TeamThe Rapid7 Data Platform is a unified, integrated platform that provides customers enhanced visibility into their attack surface, operational efficiency, risk management, and decision-making capabilities.Job Description:We are looking for an experienced Data and Performance Engineer to collaborate with our product teams to optimize their data...
-
Senior Data Engineering Lead
3 weeks ago
Belfast, United Kingdom Ocho Full timeStaff Data Engineer Location: Belfast (Hybrid)About the RoleOcho is a leading company in AI development and we are seeking a skilled Data Engineer to join our team in Belfast. The successful candidate will design, build and optimise data pipelines and architectures to support AI applications.Key Responsibilities include collaborating with data scientists, ML...
-
Data Engineer
4 months ago
Belfast, United Kingdom Rapid7 Full timeData Engineer II Rapid7 seeks a highly motivated and inquisitive aspiring Data Engineer II to join our quickly scaling data engineering function. Come and join our efforts in unlocking the value of data through industry-leading innovation, cutting edge modern tooling, democratization at scale and building exceptional and trusted data products for the...
-
Data Platform Engineer
2 days ago
Belfast, United Kingdom ESO Full timeIn this role as a Data Platform Engineer at ESO, you'll play a vital part in building and maintaining our robust data platform. You'll collaborate closely with cross-functional teams to deliver scalable, efficient, and secure solutions that cater to the diverse needs of our customers.To succeed in this position, you'll need a deep understanding of data...
-
Data Engineer for Scalable Data Solutions
6 hours ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the RoleWe are seeking a highly skilled Data Engineer to join our team at Divvy Cloud Corp. in Staff Big Data Software Engineer. The successful candidate will be responsible for designing, developing, and implementing high-performance data platforms using big data technologies such as Trino, Iceberg, Parquet, Spark, Airflow, Kafka, and AWS...
-
Senior Data Architect
4 weeks ago
Belfast, United Kingdom Divvy Cloud Corp. Full timeAbout the RoleThe Principal Engineer position is a key member of our Data Platform Engineering team at Divvy Cloud Corp. In this role, you will be responsible for designing and implementing scalable data pipelines using Apache Kafka, Apache Iceberg, and Apache Spark.You will work closely with cross-functional teams to deliver highly available, performant,...
-
Data Platform Engineer
7 hours ago
Belfast, United Kingdom ESO Full timeAre you a seasoned software developer looking for a new challenge? Do you have experience with front-end development using Angular and Typescript, as well as back-end development skills in C# .NET? If so, we may have the perfect opportunity for you!Our Data Platform team at ESO is seeking a skilled Data Platform Engineer to join our team, working closely...
-
Data Engineering Manager
2 days ago
Belfast, United Kingdom Search 5.0 Full time £75,000 - £90,000Data Leadership OpportunityWe are looking for an experienced Data Engineering Manager to lead our data function in Belfast. As a key member of our team, you will be responsible for designing and building large-scale data solutions.The successful candidate will have a proven track record in leading teams to deliver complex data projects. They should have...
-
Cloud Platform Data Engineer
4 weeks ago
Belfast, United Kingdom aPriori Full timeAbout the RoleWe are seeking a highly skilled Cloud Platform Data Engineer to join our team at aPriori, a leading provider of digital transformation solutions for manufacturers. This is an exciting opportunity to work on groundbreaking projects, driving innovation and efficiency in the industry.Job SummaryThe Cloud Platform Data Engineer will be responsible...