Data Engineer

4 weeks ago

London, UK, United Kingdom Vallum Associates Full time

Requirements

  • At least two years of professional experience as a data engineer with a proven ability to provide effective data solutions in a production setting.
  • Knowledge of developing real-time data stream systems (ideally Kafka).
  • Proven track record in developing data systems using PySpark and Apache Spark for batch processing.
  • Capable of managing data intake from various sources, including data streams, unstructured data, relational databases, and NoSQL databases.
  • Extensive knowledge in distributed systems, cloud architecture, and data pipelines.
  • Proficiency in Python programming (knowledge of Scala or Rust is a plus).
  • Familiarity with ETL principles in contemporary data applications (Dagster, Airflow, Perfect).
  • Familiarity with AWS services such as Glue, Redshift, Athena, and S3.
  • Proficiency with Terraform, Kubernetes, and ArgoCD (expertise not required; cloud team support available).
  • Thorough understanding of software engineering techniques, including version control, continuous integration and delivery (CI/CD), testing, and documentation.
  • Well-versed in data lifecycle management, including development, storage, and access.
  • Acquainted with Jira, Confluence, and other agile work and collaboration tools.
  • A cooperative team member with strong communication skills to interact with various internal and external teams and stakeholders.
  • Capable of operating in a fast-paced setting.
  • Advocate of continuous development and open-mindedness.
  • Proficiency in English, with the ability to work in the EU being a bonus.


What the Role Entails

As a growing company with expanding data demands, scalable, dependable, and efficient data architecture is required. We are now seeking a Data Engineer to join our Data team to help build, upgrade, and enhance our current data architecture.

  • Develop and automate streaming data architecture, supporting the expansion of the data team in an interesting full-time position with plenty of challenges.
  • Collaborate closely with other data engineers, scientists, analytics teams, and cloud engineering teams to develop and manage data products across multiple domains, providing scalable and dependable data for internal decision-making and client visibility.
  • Play a major role in the development and optimisation of data systems to support our Data Lake/Data Mesh architecture and data processing needs, ensuring the provision of scalable, high-performance data products for the entire organisation and proper data governance.
  • Utilise and improve our current AWS-based data platform.
  • Work with our tech stack, which includes dbt/DuckDB for transformation, Kafka/RabbitMQ as a streaming platform, Deltalake as a data format, Dagster for managing data assets, and Terraform, Kubernetes, and ArgoCD as the underlying infrastructure.


Additional Responsibilities

  • Build tools to produce data products and automate the self-serve platform.
  • Create dependable and highly scalable data assets using Python, Scala, Spark/PySpark.
  • Manage both fresh and current data sources into our pipelines for ELT and reverse ETL.
  • Create a data strategy that uses appropriate resources to manage, classify, and locate important data while guaranteeing safe access.
  • Use CI/CD, testing, and standard monitoring procedures to guarantee data quality and dependability.
  • Continuously seek out and implement new technologies and industry best practices to enhance our data infrastructure.
  • Work within the tech team using an agile methodology.
  • Understand the business to comprehend the data demands of various internal and external stakeholders and design the best possible solutions.


This position offers the opportunity to work with a bright and varied team to develop and improve our data platform for data processing, access, gathering, and monitoring, building the future of our data infrastructure.

  • Senior Data Engineer

    2 weeks ago

    London, UK, UK, United Kingdom Data Idols Full time

    Senior Data EngineerSalary: up to £85,000Location: London (Hybrid)Data Idols are working with an exciting scale-up revolutionising delivery logistics and moving items across the UK with efficiency and ease all whilst cutting emissions. Trusted by millions of customers already they are expanding their data team to manage increased volumes of data and demand....

  • Head of Data Architecture

    3 weeks ago

    London, UK, United Kingdom Paritas Recruitment - Data & Tech Full time

    We are seeking a Data Architect to take the lead in a newly formed department within a trading house. Leading a multi disciplinary team, we are seeking individuals who can come in and make the decisions on future architecture designs, contribute to a wider data strategy and face off to a range of senior stakeholders across the business. Experience in other...

  • Data Engineer

    7 days ago

    London,, UK, United Kingdom The People Network Full time

    Data EngineerA Data Engineer is required to join a global consultancy providing top quality Data solutions to the Government sector.In this exciting Data Engineer role, you will work on designing, constructing, and maintaining scalable data pipelines and transforming raw data.This is a fantastic opportunity for someone who comes from a Data Engineering...

  • Data Engineer

    2 days ago

    London,, UK, United Kingdom W Talent Full time

    Are you a Data Engineer with a demonstrable background of building and maintaining data pipelines in Azure? There is a high growth technology consulting business we are working with at the moment to help find people like you to add to their highly skilled team. If you are an Azure Data Engineer looking to apply your skills on interesting projects and learn...

  • Data Engineer

    2 weeks ago

    London,, UK, United Kingdom Mastek Full time

    Job Description: Data EngineerLocation: [UK]SC Cleared: RequiredJob Type: Full-TimeExperience: 12-14 yearsJob description:We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support data-driven...

  • Data Engineer

    2 days ago

    London,, UK, United Kingdom Oliver Bernard Full time

    Data Engineer - up to £55k We have partnered with a growing SaaS company who operate within the hospitality sector. Their purpose is to boost performance by pulling together data from all different technologies within the business in real time to improve performance. As a Data Engineer you will be expected to build and maintain their data pipeline and ETL...

  • Data Engineer

    2 days ago

    London,, UK, United Kingdom Oliver Bernard Full time

    Data Engineer - up to £70kWe have partnered with a growing SaaS company who operate within the hospitality sector. Their purpose is to boost performance by pulling together data from all different technologies within the business in real time to improve performance.As a Data Engineer you will be expected to build and maintain their data pipeline and ETL...

  • Data Engineer

    2 weeks ago

    London,, UK, United Kingdom Prism Digital Full time

    Lead Azure Data Engineer | PySpark (Python) & Synapse | Tech for Good / CharityEssential skills required:Azure – solid experience required of the Azure Data ecosystemPython - ESSENTIAL as PySpark is used heavily. You will be tested on PySparkAzure Synapse - ESSENTIAL as is used heavilySparkAzure Data Lake / Data Bricks / Data FactoryBe happy to act as a...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Response Informatics Full time

    We are looking for a Lead Data Engineer to join our Data Engineering Team. In this senior role, you will be responsible for managing our Enterprise Data Warehouse and ensuring the efficient flow and quality of data.Your tasks will include implementing data processes, managing data architecture, designing ETL processes, and analyzing data for business...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Prism Digital Full time

    Lead Azure Data Engineer | PySpark (Python) & Synapse | Tech for Good / CharityEssential skills required:Azure – solid experience required of the Azure Data ecosystemPython - ESSENTIAL as PySpark is used heavily. You will be tested on PySparkAzure Synapse - ESSENTIAL as is used heavilySparkAzure Data Lake / Data Bricks / Data FactoryBe happy to act as a...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Prism Digital Full time

    Lead Azure Data Engineer | PySpark (Python) & Synapse | Tech for Good / CharityEssential skills required:Azure – solid experience required of the Azure Data ecosystemPython - ESSENTIAL as PySpark is used heavily. You will be tested on PySparkAzure Synapse - ESSENTIAL as is used heavilySparkAzure Data Lake / Data Bricks / Data FactoryBe happy to act as a...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Travelex Full time

    Job Title: Data EngineerJob Type: Full Time, PermanentWorking location: London, HybridRole Purpose At Travelex we are developing modern data services, which will be at the heart of our relationship with our customers. Our data architecture is becoming more service-oriented and more real-time – which creates a range of engineering problems for you to solve....

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Travelex Full time

    Job Title: Data EngineerJob Type: Full Time, PermanentWorking location: London, HybridRole Purpose At Travelex we are developing modern data services, which will be at the heart of our relationship with our customers. Our data architecture is becoming more service-oriented and more real-time – which creates a range of engineering problems for you to solve....

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Understanding Recruitment Full time

    Data EngineerDo you want to join a leading investment firm specializing in quantitative trading, driven by innovation and excellence? Keep reading! Location: London, UKAs a Data Engineer, you'll build and maintain infrastructure for trading strategies, collaborating with data scientists and analysts.Responsibilities: Data Pipelines Data Integration️...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Understanding Recruitment Full time

    Data EngineerDo you want to join a leading investment firm specializing in quantitative trading, driven by innovation and excellence? Keep reading! Location: London, UKAs a Data Engineer, you'll build and maintain infrastructure for trading strategies, collaborating with data scientists and analysts.Responsibilities: Data Pipelines Data Integration️...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Bolton Associates Full time

    We are working on behalf of an International (Re)Insurance client based in London to recruit a Lead/Senior Data Engineer, joining an established team of Analytics Engineers helping optimise the Actuarial teams' efficiency through data automation and the use of emerging technologies.The successful candidate will work on the following:- Actuarial Data...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Hexegic Full time

    Data Engineers are responsible for aiding UK Health in developing a cleaned, validated, and discoverable data foundation to enable new analytical workflows and applications.They will need to work closely with UK Health subject matter experts to scope new data integrations, design a data model, and perform data cleansing and transformation within the data...

  • Data Engineer

    1 week ago

    London,, UK, United Kingdom Omnis Partners Full time

    Data Engineer Up to £70,000Healthcare Hybrid in Central London Join a global healthcare company in Central London as a Data Engineer, providing data solutions with a focus on ETL development, cloud deployment, and building data & analytics capabilities; offering up to £70,000, hybrid work, and a discretionary bonusCompanyA global Healthcare company host to...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Understanding Recruitment Full time

    Fantastic new opportunity for a Data Engineer to join a high performing team at a leading hedge fund!You'll be vital in building and maintaining infrastructure for the efficient execution of their quantitative and discretionary trading strategies.What we'd love to see: MSc in Comp Sci, Engineering etc. from a Top University Demonstrable Data...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Understanding Recruitment Full time

    Fantastic new opportunity for a Data Engineer to join a high performing team at a leading hedge fund!You'll be vital in building and maintaining infrastructure for the efficient execution of their quantitative and discretionary trading strategies.What we'd love to see: MSc in Comp Sci, Engineering etc. from a Top University Demonstrable Data...