Data Engineer

5 days ago

Solihull, United Kingdom eTeam Full time

Data Engineer

Solihull - Needs to be on site 2 days a week

Rate - 500 - 520/Day (inside IR35) (via Umbrella)


Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio, Git, GitLab, Jenkins, Bash


About the Role:

As a Data Engineer at BNP Paribas, you will be instrumental in designing, building, and optimizing the data processing pipelines in a centralized data platform "DATAHUB." This platform consolidates, federates, and enhances massive data assets for various use cases, including reporting, analytics, and machine learning. You will work with multiple data sources, ensuring the seamless integration, transformation, and quality of data, while also migrating Hadoop infrastructure to cloud environments.


Key Responsibilities:


- Data Integration: Integrate data from multiple sources and formats into the Raw Layer of the DATAHUB.

- Data Modeling and Pipeline Development: Model data and develop efficient data pipelines to enrich and transform large volumes of data with complex business rules, automating data pipelines and streamline data ingestion. Designing and implementing scalable and secure data processing pipelines using Scala, Spark, cloud Object storage, Hadoop, Cloud object storage

- Data Transformation and Quality: Implement data transformation and quality control processes to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations.

- Scheduling with Airflow: Schedule data processing tasks using Airflow.

- Validation Testing: Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.

- CI/CD Pipeline Implementation: Set up CI/CD pipelines to automate deployment, unit testing, and development management.

- Documentation: Write technical documentation (specifications, operational documents) to ensure knowledge capitalization.

- Code Improvement: understand existing code, modify the existing code as per business requirements and Continuously improve for better performance and maintainability and prepare relevant documentation.

- Infrastructure Migration: Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, Object Storage, Spark as a service, and Airflow as a service

- Performance Optimization and Security: Ensure the performance and security of the data infrastructure and follow the best practices of Data engineering.

- Production Support and Maintenance: Contribute to production support, incident and anomaly correction, trouble shoot data related issues, and implement functional and technical evolutions to ensure the stability of production processes.

- Team Collaboration: Work closely with data squads and business teams to understand data needs and provide tailored solutions.

- Agile Experience – understand of agile principle and rituals

- Software Development Lifecycle (SDLC) awareness

  • Data Engineer

    3 months ago

    Solihull, West Midlands, United Kingdom twentyAI Full time

    twentyAI are excited to be partnering with a private equity backed hospitality business who are seeking a Data Engineer to join their growing Analytics function. You will work closely with Data Scientists and the Commercial department to understand their goals and how to execute this in a data strategy. You will be able to generate valuable insights and...

  • Data Engineer

    3 months ago

    Solihull, United Kingdom twentyAI Full time

    twentyAI are excited to be partnering with a private equity backed hospitality business who are seeking a Data Engineer to join their growing Analytics function. You will work closely with Data Scientists and the Commercial department to understand their goals and how to execute this in a data strategy. You will be able to generate valuable insights and...

  • Data Engineer

    5 days ago

    Solihull, United Kingdom eTeam Full time

    Data EngineerSolihull - Needs to be on site 2 days a weekRate - 500 - 520/Day (inside IR35) (via Umbrella) Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio, Git, GitLab, Jenkins, BashAbout the Role:As a Data Engineer at BNP Paribas, you will be instrumental in designing, building, and...

  • Data Engineer

    5 days ago

    Solihull, United Kingdom ETEAM Full time

    Data EngineerSolihull - Needs to be on site 2 days a weekRate - 500 - 520/Day (inside IR35) (via Umbrella) Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio, Git, GitLab, Jenkins, BashAbout the Role:As a Data Engineer at BNP Paribas, you will be instrumental in designing, building, and...

  • Data Engineer

    5 days ago

    Solihull, United Kingdom eTeam Full time

    Data Engineer Solihull - Needs to be on site 2 days a week Rate - 500 - 520/Day (inside IR35) (via Umbrella) Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio, Git, GitLab, Jenkins, Bash About the Role: As a Data Engineer at BNP Paribas, you will be instrumental in designing, building,...

  • Data Engineer

    5 days ago

    Solihull, United Kingdom ETeam Full time

    Data EngineerSolihull - Needs to be on site 2 days a weekRate - 500 - 520/Day (inside IR35) (via Umbrella) Submit your CV and any additional required information after you have read this description by clicking on the application button.Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio,...

  • Senior Data Engineer

    4 days ago

    Solihull, Solihull, United Kingdom eTeam Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at eTeam. As a key member of our data engineering team, you will play a crucial role in designing, building, and optimizing our data processing pipelines in a centralized data platform.Key ResponsibilitiesData Integration: Integrate data from multiple sources and formats into the...

  • Senior Data Engineer

    4 days ago

    Solihull, Solihull, United Kingdom eTeam Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at eTeam. As a key member of our data engineering team, you will play a crucial role in designing, building, and optimizing our data processing pipelines in a centralized data platform.Key ResponsibilitiesData Integration: Integrate data from multiple sources and formats into the...

  • Data Engineer

    5 days ago

    Solihull, Solihull, United Kingdom twentyAI Full time

    **About twentyAI and the Role**twentyAI is partnering with a private equity-backed hospitality business to fill a Data Engineer position in their growing Analytics function. As a Data Engineer, you will collaborate closely with Data Scientists and the Commercial department to understand their goals and develop a data-driven strategy.**Key...

  • Data Engineer

    5 days ago

    Solihull, Solihull, United Kingdom twentyAI Full time

    **About twentyAI and the Role**twentyAI is partnering with a private equity-backed hospitality business to fill a Data Engineer position in their growing Analytics function. As a Data Engineer, you will collaborate closely with Data Scientists and the Commercial department to understand their goals and develop a data-driven strategy.**Key...

  • Data Engineer

    8 hours ago

    Solihull, Solihull, United Kingdom twentyAI Full time

    **About twentyAI and the Role**twentyAI is partnering with a private equity-backed hospitality business to fill a Data Engineer position in their growing Analytics function. As a Data Engineer, you will collaborate closely with Data Scientists and the Commercial department to understand their goals and develop a data-driven strategy.**Key...

  • Data Engineer

    5 hours ago

    Solihull, Solihull, United Kingdom twentyAI Full time

    **About twentyAI and the Role**twentyAI is partnering with a private equity-backed hospitality business to fill a Data Engineer position in their growing Analytics function. As a Data Engineer, you will collaborate closely with Data Scientists and the Commercial department to understand their goals and develop a data-driven strategy.**Key...

  • Data Engineer

    5 days ago

    Solihull, UK, West Midlands, United Kingdom ETEAM Full time

    Data EngineerSolihull - Needs to be on site 2 days a weekRate - 500 - 520/Day (inside IR35) (via Umbrella) Key skills (most important) Spark, Airflow , Hadoop, Oozie, Docker, Kubernetes containerization, COS, Scala, SQL, Dremio, Git, GitLab, Jenkins, BashAbout the Role:As a Data Engineer at BNP Paribas, you will be instrumental in designing, building, and...

  • Data Engineer

    4 days ago

    Solihull, Solihull, United Kingdom eTeam Full time

    About the RoleWe are seeking a highly skilled Data Engineer - Cloud Infrastructure Specialist to join our team at eTeam. As a key member of our data engineering team, you will be responsible for designing, building, and optimizing scalable data pipelines in our centralized data platform.Key Responsibilities:Data Integration: Integrate data from multiple...

  • Data Engineer

    4 days ago

    Solihull, Solihull, United Kingdom eTeam Full time

    About the RoleWe are seeking a highly skilled Data Engineer - Cloud Infrastructure Specialist to join our team at eTeam. As a key member of our data engineering team, you will be responsible for designing, building, and optimizing scalable data pipelines in our centralized data platform.Key Responsibilities:Data Integration: Integrate data from multiple...

  • Senior Data Engineer

    3 months ago

    Solihull, United Kingdom Stonegate Group Full time

    Senior Data Engineer - West Midlands The Role: We are looking for an experienced Senior Data Engineer to join our data team. You will work with the UK’s largest hospitality operator to help transform our data by building unified datasets and using best-in-class engineering processes. You will be working with senior data engineers and data scientists...

  • Data Architect

    5 days ago

    Solihull, United Kingdom ETEAM Full time

    About the Role:As a Data Engineer at ETEAM, you will play a pivotal role in designing, building, and optimizing the data processing pipelines in a centralized data platform. This platform will consolidate, federate, and enhance massive data assets for various use cases, including reporting, analytics, and machine learning. You will work with multiple data...

  • Data Architect

    5 days ago

    Solihull, United Kingdom ETEAM Full time

    About the Role:As a Data Engineer at ETEAM, you will play a pivotal role in designing, building, and optimizing the data processing pipelines in a centralized data platform. This platform will consolidate, federate, and enhance massive data assets for various use cases, including reporting, analytics, and machine learning. You will work with multiple data...

  • Data Architect

    5 days ago

    Solihull, Solihull, United Kingdom ETeam Full time

    About the Role:As a Data Engineer at ETeam, you will be instrumental in designing, building, and optimizing the data processing pipelines in a centralized data platform. This platform consolidates, federates, and enhances massive data assets for various use cases, including reporting, analytics, and machine learning. You will work with multiple data sources,...

  • Data Architect

    4 days ago

    Solihull, Solihull, United Kingdom ETeam Full time

    About the Role:As a Data Engineer at ETeam, you will be instrumental in designing, building, and optimizing the data processing pipelines in a centralized data platform. This platform consolidates, federates, and enhances massive data assets for various use cases, including reporting, analytics, and machine learning. You will work with multiple data sources,...