Data Engineer

4 weeks ago


Leeds, United Kingdom Damia Group Full time

**Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)**


Hit Apply below to send your application for consideration Ensure that your CV is up to date, and that you have read the job specs first.

We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world.

The successful candidate will be responsible for designing, building, and maintaining data pipelines using Apache Spark and Scala. This includes tasks like:

  • Extracting data from various sources (databases, APIs, files)
  • Transforming and cleaning the data
  • Loading the data into data warehouses or data lakes (e.g., BigQuery, Amazon Redshift)
  • Automating the data pipeline execution using scheduling tools (e.g., Airflow)
  • Work with Big Data technologies: You'll likely work with various Big Data technologies alongside Spark, including:

Hadoop Distributed File System (HDFS) for storing large datasets

Apache Kafka for real-time data streaming

Apache Hive for data warehousing on top of HDFS

Cloud platforms like AWS, Azure, or GCP for deploying and managing your data pipelines

Key skills:

  • At least 8+ Years of IT Experience with designing, building, and maintaining data pipelines.
  • At least 4+ Years of experience with designing, building, and maintaining data pipelines using Apache Spark and Scala
  • Programming languages: Proficiency in Scala and Spark is essential. Familiarity with Python and SQL is often a plus.
  • Big Data technologies: Understanding of HDFS, Kafka, Hive, and cloud platforms is valuable.
  • Data engineering concepts: Knowledge of data warehousing, data pipelines, data modelling, and data cleansing techniques is crucial.
  • Problem-solving and analytical skills: You should be able to analyse complex data problems, design efficient solutions, and troubleshoot issues.
  • Communication and collaboration: The ability to communicate effectively with data scientists, analysts, and business stakeholders is essential.

Beneficial skills:

  • Machine learning libraries: Familiarity with Spark ML or other machine learning libraries in Scala can be advantageous.
  • Cloud computing experience: Experience with cloud platforms like AWS, Azure, or GCP for data pipelines deployment is a plus.
  • DevOps tools: Knowledge of DevOps tools like Git, CI/CD pipelines, and containerization tools (Docker, Kubernetes) can be beneficial.

  • Data Centre Engineer

    4 weeks ago


    Leeds, Leeds, United Kingdom Abs Data Full time

    Job title: Data Centre Engineer – PART TIME Company: Salute Job description : Are you Data Center Facility Engineer looking for a great opportunity but only available part time? Salute has several great opportunities in our key locations – supporting critical facilities – working on a 1 in 3 week call out rota.Salute is a global leader in...

  • Lead Data Engineer

    4 weeks ago


    Leeds, Leeds, United Kingdom Harnham - Data & Analytics Recruitment Full time

    Lead Data EngineerUp to £100,000 per annumLocation - Remote - attend office in London once every 3 weeksURGENT HIRE Lead Data Engineer sought for a well-established insurance companyThe CompanyThis company is a fast-growing data-focused general insurance provider. They offer a range of products and services, which include car, bike, van, and home insurance...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom SPG Resourcing Full time

    Salary: 70k Location: Home-based (UK only) with some travel within the UK Start Date: Immediate An innovative Professional Services company is seeking a highly skilled and experienced Senior Azure Data Engineer to join their team and play a pivotal role in their data modernisation journey. Amongst other initiatives, you will be a part of a wider team of...

  • Data Engineer

    2 weeks ago


    Leeds, United Kingdom SPG Resourcing Full time

    Day Rate: £400-£500 Location: Home-based (UK only) with some travel within the UK Start Date: Immediate An innovative Professional Services company is seeking a highly skilled and experienced Senior Azure Data Engineer to join their team and play a pivotal role in their data modernisation journey. Amongst other initiatives, you will be a part of a wider...

  • Data Engineer

    4 weeks ago


    Leeds, Leeds, United Kingdom Interface Recruitment UK Full time

    Job Title : Senior Data EngineerThe Role:Chance to work with Cloud technology Big data environment Pioneering technology Truly creative / studio environment Flexbility Leeds City location but with free parking Friendly team with a one for all attitude Summary of position The senior data engineer will be working across our client and SaaS product...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom SPG Resourcing Full time

    An innovative Professional Services company is seeking a highly skilled and experienced Senior Azure Data Engineer to join their team and play a pivotal role in their data modernisation journey.Amongst other initiatives, you will be a part of a wider team of data engineers who can design, build, and maintain robust data pipelines and solutions on the...

  • Data Engineer

    3 weeks ago


    Leeds, United Kingdom SPG Resourcing Full time

    An innovative Professional Services company is seeking a highly skilled and experienced Senior Azure Data Engineer to join their team and play a pivotal role in their data modernisation journey.To be considered for an interview, please make sure your application is full in line with the job specs as found below.Amongst other initiatives, you will be a part...

  • Data Engineer

    4 weeks ago


    Leeds, Leeds, United Kingdom Fruition IT Full time

    Job DescriptionData EngineerLeeds - on site 2x per weekup to £55,000Looking to join a dynamic team in the financial sector? Fruition IT are looking for an experienced Data Engineer to join a leading company specializing in financial solutions.Working closely with the Senior Data Engineer and Architects, you will play a crucial role in their journey of...

  • Data Engineer

    1 month ago


    Leeds, United Kingdom BJSS Full time

    About UsWe’re an award-winning innovative tech consultancy - a team of creative problem solvers. Since 1993 we’ve been finding better, more sustainable ways to solve complex technology problems for some of the world’s leading organisations and delivered solutions that millions of people use every day.In the last 30 years we won several awards,...

  • Data Engineer

    1 month ago


    Leeds, United Kingdom BJSS Full time

    About UsWe’re an award-winning innovative tech consultancy - a team of creative problem solvers. Since 1993 we’ve been finding better, more sustainable ways to solve complex technology problems for some of the world’s leading organisations and delivered solutions that millions of people use every day.In the last 30 years we won several awards,...

  • Data Engineer

    1 month ago


    Leeds, United Kingdom BJSS Full time

    About Us We’re an award-winning innovative tech consultancy - a team of creative problem solvers. Since 1993 we’ve been finding better, more sustainable ways to solve complex technology problems for some of the world’s leading organisations and delivered solutions that millions of people use every day. In the last 30 years we won several awards,...

  • Data Engineer

    1 month ago


    Leeds, United Kingdom BJSS Full time

    About Us We’re an award-winning innovative tech consultancy - a team of creative problem solvers. Since 1993 we’ve been finding better, more sustainable ways to solve complex technology problems for some of the world’s leading organisations and delivered solutions that millions of people use every day. In the last 30 years we won several awards,...

  • Data Engineer

    2 months ago


    Leeds, Leeds, United Kingdom Sky Betting and Gaming Full time

    What you'll doWe're looking for a dynamic Data Engineer to join one of the most exciting Tribes within Sky Betting and Gaming – the Data TribeData are the tribe that takes raw information, and puts intelligence in the hands of decision makers. Our teams design and build SBG's data capabilities, from data warehousing platforms and business reporting, to...

  • Data Engineer

    4 weeks ago


    Leeds, Leeds, United Kingdom Fruition IT Full time

    Data EngineernLeeds - on site 2x per weeknup to £55,000nLooking to join a dynamic team in the financial sector? Fruition IT are looking for an experienced Data Engineer to join a leading company specializing in financial solutions.nWorking closely with the Senior Data Engineer and Architects, you will play a crucial role in their journey of migrating from...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Damia Group Full time

    **Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)**We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world. The successful candidate will be responsible for designing,...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Damia Group Full time

    **Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)**We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world. The successful candidate will be responsible for designing,...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Damia Group Full time

    **Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)**We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world. The successful candidate will be responsible for designing,...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Damia Group Full time

    **Spark Scala Engineer – 12 month initial contract – Leeds (3 days in the office) - £550 per day (Inside IR35)** We are looking for a Spark Scala Engineer to join a long-term project for a management consulting firm working with one of the biggest financial services organisations in the world. The successful candidate will be responsible for...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Change Digital – Digital & Tech Recruitment Full time

    Financial Services firm looking for a heavy hitting Data engineer where some of your Key Responsibilities will include: Develop new pipelines based on business needs or to improve the current system. Maintain existing data pipelines, anticipating and addressing issues promptly and professionally. Monitor critical processes effectively. Collaborate with...

  • Data Engineer

    4 weeks ago


    Leeds, United Kingdom Change Digital – Digital & Tech Recruitment Full time

    Financial Services firm looking for a heavy hitting Data engineer where some of your Key Responsibilities will include: Develop new pipelines based on business needs or to improve the current system. Maintain existing data pipelines, anticipating and addressing issues promptly and professionally. Monitor critical processes effectively. Collaborate with...