Lead Data Engineer

1 month ago

South East London, United Kingdom Canada Life Full time
Location: London, Potters Bar, Bristol or Isle of Man (hybrid working options available) The Lead Data Engineer is a hands-on technical role responsible for designing, developing, and maintaining data pipelines within the IT department. The pipelines will be realised in a modern lake environment and the engineer will collaborate in cross-functional

teams to gather requirements and develop the conceptual data models. This role plays a crucial part in driving data-driven decision-making across the organisation, ensuring data availability, quality, and accessibility for various business needs. This is not a line management role, but you will play a key part in guiding and upskilling more junior data engineers and setting data standards and guidelines. Duties/Responsibilities Data Pipeline Development Design, model, develop and maintain data pipelines to ingest, store, process, and present data. Ensure data quality, accuracy, and consistency. Collaborate with data architects to ensure data pipelines align with the overall data architecture strategy. Data Transformation and Integration Perform data transformation tasks, including data cleansing, enrichment, and aggregation, to prepare data for analytics and reporting. Integrate data from structured and unstructured sources, ensuring compatibility and alignment with data models and business requirements. Automate data transformation processes to improve efficiency. Data Quality Assurance Implement and maintain data quality checks and validation processes to identify and resolve data anomalies and errors. Monitor data pipelines for data quality issues and implement data quality improvements. Collaborate with business stakeholders to define data quality requirements. Data Modelling and Schema Design Collaborate with data architects and data scientists to design and implement data models, schemas, and structures. Ensure that data models support business reporting and analytics needs while optimizing query performance. Maintain data dictionaries and metadata to document data structures and relationships. Performance Optimization Optimize data storage, retrieval, and query performance by implementing indexing, partitioning, and caching strategies. Monitor data processing performance and address bottlenecks as they arise. Stay updated with best practices in data processing performance tuning. Documentation and Knowledge Sharing Create and maintain documentation for data pipelines, data transformation processes, and data integration procedures. Foster a culture of knowledge sharing within the data engineering team and across the organization. Collaboration and Stakeholder Engagement Collaborate effectively with cross-functional teams, data stakeholders, and business units to understand data requirements and deliver data solutions that meet business needs. Communicate technical concepts and data solutions to non-technical stakeholders in a clear and understandable manner. Knowledge/Skills/Experience Essential Extensive experience in data engineering, including designing and developing data pipelines for retrieval / ingestion / presentation / semantics in an Azure environment. Strong DataBricks, SQL, Python, Power BI Data acquisition from various data sources including Salesforce, API, XML, json, parquet, flat file systems and relational data. Effective communication and collaboration skills to work with cross-functional teams and gather data requirements. Skills in data modelling (both structured and unstructured data) working directly with the business & data scientists. Ability to optimise data solutions for performance, scalability, and efficiency. Desirable: Experience in a financial corporation Lake House / Delta Lake and Snowflake Experience with Spark clusters, both elastic permanent and transitory clusters Familiarity with data governance, data security, and compliance requirements. Power Automate.

  • Lead Data Engineer

    1 month ago

    City of London, United Kingdom Data Idols Full time

    **Lead Data Engineer** Data Idols are proud to be working with an iconic UK brand to assist them in hiring a Lead Data Engineer as they undertake a digital and data-driven journey that will transform the business. Data is at the heart of the organisation and forms a key part oftheir future plans. **Skills and Experience**: - Awareness and understanding of...

  • Lead Engineer/Data Engineer

    1 month ago

    London, United Kingdom Abs Data Full time

    Lead Data Engineer Do you enjoy creating valuable insights with data? Do you want to play a key role in transforming our firm into an agile organization? Being agile will make us more responsive, more adaptable, and ultimately more innovative. We’re looking for a Data Engineer to: ~ transform data into valuable insights that inform business...

  • Data Engineering Lead Engineer

    2 weeks ago

    London, United Kingdom ON Data Staffing Full time

    We are seeking a dynamic and experienced Databricks professional to join our client, a growing startup consultancy, as a Lead Data Engineer. As Lead Data Engineer, you will play a pivotal role in driving the success of our client's Databricks practice through your expertise in presales, strong technical acumen, consulting skills, and effective team...

  • Lead Data Engineer

    1 month ago

    London, United Kingdom Harnham - Data & Analytics Recruitment Full time

    **LEAD DATA ENGINEER £70,000 - £75,000 + BENEFITS HYBRID - LONDON OR MAIDENHEAD** Are you a Senior Data Engineer looking for an exciting new role working within a team of likeminded individuals? Do you want to become an integral part of a team working within a betting company? If this is the role you have been searching for read on! **THE...

  • Lead Data Engineer

    1 month ago

    London, United Kingdom Harnham - Data & Analytics Recruitment Full time

    **LEAD DATA ENGINEER** **£100,000 - £110,000 + EQUITY** **LONDON - HYBRID** This role is with a rapidly growing FinTech Scale up who are looking to build out their data engineering team. They are looking to hire a Lead Data Engineer to work closely with the Engineering Manager and Architects to take their data platform & capabilityto the next...

  • Data Lead

    1 month ago

    London, United Kingdom Data Idols Full time

    **Data Science Lead**: Data Idols are working with a disrupter within the delivery service space. They are looking for a Data Science Lead to join their growing team and support them as they invest heavily in their data platform. **The Opportunity**: This opportunity comes with huge growth and the chance to really put your mark on this role. The company are...

  • Data Engineering Lead

    2 weeks ago

    London, United Kingdom ON Data Staffing Full time

    We are seeking a dynamic and experienced Databricks professional to join our client, a growing startup consultancy, as a Lead Data Engineer. As Lead Data Engineer, you will play a pivotal role in driving the success of our client's Databricks practice through your expertise in presales, strong technical acumen, consulting skills, and effective team...

  • Lead Data Engineer

    4 weeks ago

    London, United Kingdom ON Data Staffing Full time

    We are seeking a dynamic and experienced Databricks professional to join our client, a growing startup consultancy, as a Lead Data Engineer.As Lead Data Engineer, you will play a pivotal role in driving the success of our client's Databricks practice through your expertise in presales, strong technical acumen, consulting skills, and effective team...

  • Data Engineer

    1 month ago

    City of London, United Kingdom Data Idols Full time

    **Senior Data Engineer** Data Idols are proud to be working with an iconic UK brand to assist them in hiring a Senior Data Engineer as they undertake a digital and data-driven journey that will transform the business. Data is at the heart of the organisation and forms a key partof their future plans. **The Opportunity** As a Senior Data Engineer you will be...

  • Senior Engineer, Data Engineering

    3 weeks ago

    London, United Kingdom Data Idols Full time

    Senior Data Engineer Location: London (Hybrid) Data Idols are working with one of the fast-growing companies in the logistics sector that are making the world a greener place. They are looking for a Senior Data Engineer to join their team within the Data Space. As a Senior Data Engineer, you will work alongside different data teams to design,...

  • Senior Engineer, Data Engineering

    3 weeks ago

    London, United Kingdom Data Idols Full time

    Senior Data Engineer Location: London (Hybrid) Data Idols are working with one of the fast-growing companies in the logistics sector that are making the world a greener place. They are looking for a Senior Data Engineer to join their team within the Data Space. As a Senior Data Engineer, you will work alongside different data teams to design,...

  • Senior Engineer, Data Engineering

    2 days ago

    London, United Kingdom Data Idols Full time

    Job Description Senior Data Engineer Location: London (Hybrid) Data Idols are working with one of the fast-growing companies in the logistics sector that are making the world a greener place. They are looking for a Senior Data Engineer to join their team within the Data Space. As a Senior Data Engineer, you will work alongside different data teams to...

  • Gcp Data Engineer

    1 month ago

    London, United Kingdom Data Idols Full time

    **GCP Data Engineer** Would you like to build enterprise scale data solutions for a global organisation that reaches 400 million users monthly? As a GCP Data Engineer, you will be involved in delivering solutions from initial concept to production. **The Opportunity** Working for a well known brand you will be responsible for all aspects of the data...

  • Lead Data Scientist

    1 month ago

    London, United Kingdom Harnham - Data & Analytics Recruitment Full time

    Lead Data Scientist London £90,000 - £100,000 This is a great next step for a Data Science expert to join an established company targeting online safety! THE COMPANY This well-known company have set up a new area in online safety and they are investing significantly in their Data presence; their Data team has grown from 4 people in 2021 to 50 people and...

  • Lead Data Scientist

    4 days ago

    City of London, Greater London, United Kingdom Data Idols Full time

    Job Description Lead Data Scientist Data Idols are working with a great company who are on the mission to save households £1000's every year! They are looking for a Lead Data Scientist to lead their existing Data Science team and promote best practices. You will be part of a customer focused, data driven environment where you are encouraged to explore...

  • Lead Data Scientist

    3 weeks ago

    London, United Kingdom Data Science Festival Full time

    Lead Data Scientist Data Idols are working with a great company who are on the mission to save households £1000’s every year! They are looking for a Lead Data Scientist to lead their existing Data Science team and promote best practices. You will be part of a customer focused, data driven environment where you are encouraged to explore new ideas and get...

  • Data Software Engineer | Python

    1 month ago

    London, United Kingdom Abs Data Full time

    Regulatory Reporting, Big Data Software Engineer We’re looking for software engineers to: provide technology solutions that will solve business problems and strengthen our position as digital leaders in financial services work with our Regulatory Reporting team to translate business and external regulatory needs using Behavior Driven Development...

  • Data Lead for Performance

    4 days ago

    London, United Kingdom ON Data Staffing Full time

    Job Description We are seeking a dynamic and experienced Databricks professional to join our client, a Boutique Data & AI consultancy, as a Databricks Practice Lead in London. As Databricks Practice Lead, you will play a pivotal role in driving the success of our client's Databricks practice through your expertise in presales, strong technical acumen,...

  • Data Lead for Performance

    3 weeks ago

    London, United Kingdom ON Data Staffing Full time

    We are seeking a dynamic and experienced Databricks professional to join our client, a Boutique Data & AI consultancy, as a Databricks Practice Lead in London. As Databricks Practice Lead, you will play a pivotal role in driving the success of our client's Databricks practice through your expertise in presales, strong technical acumen, consulting skills,...

  • Senior Data Engineer

    1 month ago

    London, United Kingdom Harnham - Data & Analytics Recruitment Full time

    **SENIOR DATA ENGINEER** **LONDON BASED** **£90,000-100,000 PER ANNUM** This UK-based Fintech are searching for a new Senior Data Engineer to help architect and build a new data platform in AWS. You will also be responsible for liaising with stakeholders and influencing future business decisions. **THE COMPANY** This growing UK-based Fintech company...