Python Big Data Engineer

2 weeks ago


London, Greater London, United Kingdom N Consulting Ltd Full time

About N Consulting Ltd

N Consulting Ltd is a leading provider of data-driven solutions, helping businesses unlock the power of their information to make informed decisions and drive growth.

Role Overview

We are seeking a skilled Python Big Data Engineer to join our team. In this role, you will be responsible for designing, developing, and implementing data processing pipelines using Databricks and related technologies. You will work closely with stakeholders to understand their needs and deliver innovative solutions that meet their business objectives.

Key Responsibilities

  • Develop and optimize ETL processes using Python, Databricks, and Apache Spark.
  • Containerize Databricks workloads using Docker to ensure portability and reproducibility.
  • Implement CI/CD pipelines and automation for Databricks workflows.
  • Utilize DevOps tools such as Bitbucket or GitLab to manage code versioning and collaboration.
  • Set up, configure, and manage Databricks clusters to ensure optimal runtime performance.

Required Skills and Experience

  • 5+ years of experience in data processing and Python development.
  • Strong understanding of ETL principles and best practices.
  • Hands-on experience with Databricks, Apache Spark, and Docker.
  • Proficiency in DevOps tools such as Bitbucket or GitLab.
  • Experience managing and optimizing cloud-based data processing environments.


  • London, Greater London, United Kingdom N Consulting Ltd Full time

    Job Summary:N Consulting Ltd is seeking a highly skilled Python Big Data Developer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Python and related technologies.Key Responsibilities:Data Processing: Design and implement efficient...


  • London, Greater London, United Kingdom N Consulting Ltd Full time

    Job Summary:N Consulting Ltd is seeking a highly skilled Python Big Data Developer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Python and related technologies.Key Responsibilities:Data Processing: Design and implement efficient...

  • Big Data Developer

    4 days ago


    London, Greater London, United Kingdom N Consulting Ltd Full time

    Job Summary:N Consulting Ltd is seeking a highly skilled Python Big Data Developer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Python and related technologies.Key Responsibilities:Data Processing: Design and implement efficient...

  • Big Data Developer

    5 days ago


    London, Greater London, United Kingdom N Consulting Ltd Full time

    Job Summary:N Consulting Ltd is seeking a highly skilled Python Big Data Developer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Python and related technologies.Key Responsibilities:Data Processing: Design and implement efficient...

  • Big Data Specialist

    5 days ago


    London, Greater London, United Kingdom Saragossa Full time £150,000

    About the RoleWe are seeking a highly skilled Pensions Developer to join our team at Saragossa, a global Multistrat Hedge Fund. As a key member of our Risk Engineering Team, you will play a critical role in designing, developing, and implementing Big Data and real-time applications.Key ResponsibilitiesDesign and develop risk platforms to support our business...


  • London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Salt, a leading company in the finance industry. As a Data Engineer, you will play a critical role in designing and implementing data engineering solutions on AWS, leveraging cloud DevOps and big data administration expertise.Key ResponsibilitiesCloud DevOps & Big Data...


  • London, Greater London, United Kingdom Salt Full time

    Job DescriptionJob SummarySalt is seeking a highly skilled Cloud Data Engineer to join our team. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems on AWS.Key ResponsibilitiesCloud and Big Data Administration:Manage and optimize big data environments on AWS, ensuring efficient administration...


  • London, Greater London, United Kingdom Salt Full time

    Job Title: AWS Cloud Data EngineerJob Type: ContractLocation: RemoteJob Description:Data Engineer (Cloud, DevOps, Big Data) - Finance - RemoteJob Summary:Salt is seeking a highly skilled Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation,...

  • Data Engineer

    4 days ago


    London, Greater London, United Kingdom Information Tech Consultants Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Information Tech Consultants. As a Data Engineer, you will play a critical role in designing, building, and maintaining large-scale data processing systems.Key ResponsibilitiesDesign and develop scalable data processing systems using cloud and on-premises technologiesWork with...

  • Data Engineer

    5 days ago


    London, Greater London, United Kingdom Information Tech Consultants Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Information Tech Consultants. As a Data Engineer, you will play a critical role in designing, building, and maintaining large-scale data processing systems.Key ResponsibilitiesDesign and develop scalable data processing systems using cloud and on-premises technologiesWork with...


  • London, Greater London, United Kingdom Freshminds Full time £200

    Job Opportunity at FreshmindsAbout the RoleFreshminds is undertaking a comprehensive transformation programme, and we are seeking an experienced Data Engineer to join our team. This is an exciting opportunity to manage the delivery of high-impact projects and contribute to one of the UK's largest retail transformation programmes.Key...


  • London, Greater London, United Kingdom Freshminds Full time £200

    Job Opportunity at FreshmindsAbout the RoleFreshminds is undertaking a comprehensive transformation programme, and we are seeking an experienced Data Engineer to join our team. This is an exciting opportunity to manage the delivery of high-impact projects and contribute to one of the UK's largest retail transformation programmes.Key...

  • Big Data Engineer

    2 weeks ago


    London, Greater London, United Kingdom IntaPeople Full time

    Job OverviewThis is a fantastic opportunity for a skilled Big Data Engineer to join our team at IntaPeople and work on a multi-year project with cutting-edge technology.Key ResponsibilitiesDesign and develop advanced software solutions for big data managementMigrate processes and applications to AzureCollaborate with colleagues and stakeholders to deliver...

  • Big Data Analyst

    1 month ago


    London, Greater London, United Kingdom Nexus Jobs Limited Full time

    Job Description Big Data Scientist - Docklands LondonOur Client an international organisation is looking to recruit a Big Data Expert with 2 to 5 years expertise as a Big Data Scientist.You will have Natural Language Processing (NLP) knowledge and experience and understand and analyse large data sets to discover new insights.Working in a multi-disciplinary...


  • London, Greater London, United Kingdom Databricks Full time

    About the RoleWe are seeking a highly skilled Big Data Solutions Architect to join our Professional Services team at Databricks. As a key member of our team, you will work closely with clients to deliver impactful technical projects that drive business value.Key ResponsibilitiesDesign and build reference architectures for clients, creating how-to's and...

  • Data Engineer

    5 days ago


    London, Greater London, United Kingdom Oxford Knight Full time

    About Oxford KnightOxford Knight is a globally leading systematic quant trading firm that is looking for a highly skilled Data Engineer to join their technical team in London.Job SummaryThe successful Data Engineer will be utilizing both analytical and technical skillsets, building data pipelines in the Python and PyData stack, and working closely with...

  • Data Engineer

    4 days ago


    London, Greater London, United Kingdom Oxford Knight Full time

    About Oxford KnightOxford Knight is a globally leading systematic quant trading firm that is looking for a highly skilled Data Engineer to join their technical team in London.Job SummaryThe successful Data Engineer will be utilizing both analytical and technical skillsets, building data pipelines in the Python and PyData stack, and working closely with...


  • London, Greater London, United Kingdom LinuxRecruit Full time

    Job SummaryWe are seeking a highly skilled Data Platform Engineer to join our team at LinuxRecruit. As a key member of our engineering team, you will be responsible for designing and maintaining data processing pipelines that handle vast amounts of information.Key ResponsibilitiesDesign and implement data processing pipelines using AWS services such as...


  • London, Greater London, United Kingdom LinuxRecruit Full time

    Job SummaryWe are seeking a highly skilled Data Platform Engineer to join our team at LinuxRecruit. As a key member of our engineering team, you will be responsible for designing and maintaining data processing pipelines that handle vast amounts of information.Key ResponsibilitiesDesign and implement data processing pipelines using AWS services such as...


  • London, Greater London, United Kingdom ON Data Staffing Full time

    About the RoleWe are seeking a highly skilled Senior Data Solutions Architect to join our team at ON Data Staffing. As a key member of our data engineering team, you will play a pivotal role in delivering innovative big data solutions to our clients.Key ResponsibilitiesClient EngagementCollaborate directly with our clients to understand their business...