Senior Databricks Engineer

4 weeks ago

Edinburgh, United Kingdom Version 1 Full time

Job Description

This is an exciting opportunity for an experienced developer of large-scale data solutions. You will join a team delivering a transformative cloud hosted data platform for a key Version 1 customer.

The ideal candidate will have a proven track record as a senior/self-starting data engineer in implementing data ingestion and transformation pipelines for large scale organisations. We are seeking someone with deep technical skills in a variety of technologies, specifically SPARK performance\tuning\optimisation and Databricks, to play an important role in developing and delivering early proofs of concept and production implementation.

You will ideally have experience in building solutions using a variety of open source tools & Microsoft Azure services, and a proven track record in delivering high quality work to tight deadlines.

Your main responsibilities will be:

Designing and implementing highly performant data ingestion & transformation pipelines from multiple sources using Databricks and Spark Streaming and Batch processes in Databricks SPARK performance\tuning\optimisation Providing technical guidance for complex geospatial problems and spark dataframes Developing scalable and re-usable frameworks for ingestion and transformation of large data sets Data quality system and process design and implementation. Integrating the end to end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is maintained at all times Working with other members of the project team to support delivery of additional project components (Reporting tools, API interfaces, Search) Evaluating the performance and applicability of multiple tools against customer requirements Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.

Qualifications

Direct experience of building data piplines using Azure Data Factory and Databricks Spark. Building data integration with Python Hands on experience designing and delivering solutions using the Azure Data Analytics platform. Experience building data warehouse solutions using ETL / ELT tools like Informatica, Talend. Comprehensive understanding of data management best practices including demonstrated experience with data profiling, sourcing, and cleansing routines utilizing typical data quality functions involving standardization, transformation, rationalization, linking and matching.

  • Senior Data Engineer

    3 days ago

    Edinburgh, United Kingdom Change Digital Full time

    Are you an experienced Senior Data Engineer ? Would you like to work for a fast growing business that specialises in Business Intelligence & Data Analytics ? Its a Hybrid working role and you will be based from either their London, Leeds, Manchester or Edinburgh office, typically 3 days a week from home and 2 day in office / on client site. Preferred...

  • Senior Data Engineer

    3 days ago

    Edinburgh, Edinburgh, United Kingdom Change Digital Full time

    Are you an experienced Senior Data Engineer ?Would you like to work for a fast growing business that specialises in Business Intelligence & Data Analytics ?Its a Hybrid working role and you will be based from either their London, Leeds, Manchester or Edinburgh office, typically 3 days a week from home and 2 day in office / on client site.Preferred skills:...

  • Senior Data Engineer

    1 month ago

    Edinburgh, United Kingdom Best Buy Full time

    As a Senior Data Engineer you will be revamping our patient admissions/enrollment capabilities - we’d like to introduce support for net new fields captured within that process, as well as define the most appropriate locations for that data to be stored so that it can be used within different workflows of the future. As a specialist in data engineering,...

  • Senior Azure Data Engineer

    4 weeks ago

    Edinburgh, United Kingdom Storm ID Ltd Full time

    As a Data Engineer at Storm you will work alongside other software and cloud infrastructure engineers to design, build and maintain data pipelines, data management solutions and analytics components for our client’s projects across the healthcare, justice and local government sectors. We know that many Data Engineer's will come from a variety of...

  • Senior Azure Data Engineer

    4 weeks ago

    Edinburgh, United Kingdom Storm ID Full time

    The role As a Data Engineer at Storm you will work alongside other software and cloud infrastructure engineers to design, build and maintain data pipelines, data management solutions and analytics components for our client’s projects across the healthcare, justice and local government sectors. We know that many Data Engineer's will come from a variety...

  • Senior Azure Data Engineer

    4 days ago

    Edinburgh, United Kingdom Storm ID Full time

    As a Data Engineer at Storm you will work alongside other software and cloud infrastructure engineers to design, build and maintain data pipelines, data management solutions and analytics components for our client’s projects across the healthcare, justice and local government sectors. We know that many Data Engineer's will come from a variety of...

  • Senior Azure Data Engineer

    4 weeks ago

    Edinburgh, United Kingdom Storm ID Full time

    The role As a Data Engineer at Storm you will work alongside other software and cloud infrastructure engineers to design, build and maintain data pipelines, data management solutions and analytics components for our client’s projects across the healthcare, justice and local government sectors. We know that many Data Engineer's will come from a variety...

  • Senior Electrical Engineer

    2 days ago

    Edinburgh, United Kingdom Elliot Marsh Full time

    Our client has an exciting opportunity for a Senior Electrical Engineer to join the team. Location: Edinburgh, Scotland - Hybrid Salary: £60K per annum About Us: Join a dynamic and rapidly growing manufacturing business as our client expands their operations with a new site on the outskirts of Glasgow. They are committed to innovation, sustainability, and...

  • Senior Validation Engineer

    1 month ago

    Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Validation Engineer to join an innovative company building the first and only space-based cellular broadband network based in Edinburgh. As a Senior Validation Engineer you will have a number of responsibilities that include;IC Validation and board bringupValidation bring up using FPGA prototypesSupport ATE...

  • Senior Validation Engineer

    1 month ago

    Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Validation Engineer to join an innovative company building the first and only space-based cellular broadband network based in Edinburgh. As a Senior Validation Engineer you will have a number of responsibilities that include; IC Validation and board bringup Validation bring up using FPGA prototypes ...

  • Senior Electrical Engineer

    2 days ago

    Edinburgh, United Kingdom Elliot Marsh Full time

    Our client has an exciting opportunity for a Senior Electrical Engineer to join the team. Location: Edinburgh, Scotland Hybrid Salary: £60K per annum About Us: Join a dynamic and rapidly growing manufacturing business as our client expands their operations with a new site on the outskirts of Glasgow. They are committed to innovation, sustainability,...

  • Senior Electrical Engineer

    2 days ago

    Edinburgh, United Kingdom Elliot Marsh Full time

    Our client has an exciting opportunity for a Senior Electrical Engineer to join the team. Location: Edinburgh, Scotland - Hybrid Salary: £60K per annum About Us: Join a dynamic and rapidly growing manufacturing business as our client expands their operations with a new site on the outskirts of Glasgow. They are committed to innovation,...

  • Senior Application Engineer

    1 month ago

    Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Applications Engineer to join a growing semiconductor company focusingon battery management systems for the automotive industry, based in Edinburgh. This role is split 50/50 between travelling and lab work. As a Senior Applications Engineer you will have a number of responsibilities that include;International...

  • Senior Application Engineer

    1 month ago

    Edinburgh, Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Applications Engineer to join a growing semiconductor company focusingon battery management systems for the automotive industry, based in Edinburgh. This role is split 50/50 between travelling and lab work.As a Senior Applications Engineer you will have a number of responsibilities that include;International...

  • Senior Validation Engineer

    4 weeks ago

    Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Validation Engineer to join an innovative company building the first and only space-based cellular broadband network based in Edinburgh. As a Senior Validation Engineer you will have a number of responsibilities that include;IC Validation and board bringupValidation bring up using FPGA prototypesSupport ATE...

  • Senior Application Engineer

    1 month ago

    Edinburgh, Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Applications Engineer to join a growing semiconductor company focusingon battery management systems for the automotive industry, based in Edinburgh. This role is split 50/50 between travelling and lab work.As a Senior Applications Engineer you will have a number of responsibilities that include;International...

  • Senior Electrical Engineer

    3 weeks ago

    Edinburgh, United Kingdom Elliot Marsh Full time

    Senior Electrical Engineer Edinburgh, Scotland - Hybrid Join a dynamic and rapidly growing manufacturing business as our client expands their operations with a new site on the outskirts of Glasgow. They are committed to innovation, sustainability, and excellence in all aspects of their work. As they embark on this exciting journey, they are seeking a...

  • Senior Electrical Engineer

    3 weeks ago

    Edinburgh, United Kingdom Elliot Marsh Full time

    Our client has an exciting opportunity for a Senior Electrical Engineer to join the team. Edinburgh, Scotland - Hybrid Join a dynamic and rapidly growing manufacturing business as our client expands their operations with a new site on the outskirts of Glasgow. They are committed to innovation, sustainability, and excellence in all aspects of their...

  • Senior Software Engineer

    7 days ago

    Edinburgh, United Kingdom Sanderson Recruitment Full time

    -Senior Software Engineer -£75,000 Depending on experience. -Once per month in Edinburgh, Bath or London Are you a seasoned software engineer looking for a thrilling opportunity to innovate and drive impactful solutions? Our client, a pioneering force in the fintech industry, is seeking a talented individual like you to join their dynamic team! As a Senior...

  • Senior Application Engineer

    4 weeks ago

    Edinburgh, United Kingdom IC Resources Full time

    An opportunity has become available for a Senior Applications Engineer to join a growing semiconductor company focusing on battery management systems for the automotive industry, based in Edinburgh. This role is split 50/50 between travelling and lab work. As a Senior Applications Engineer you will have a number of responsibilities that...