Senior Data Engineer AWS DevOps Big Data

2 days ago

London, Greater London, United Kingdom Salt Search Full time
Data Engineer (AWS, DevOps, Big Data) - Finance - London

Our client is seeking a highly skilled Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation, data product concepts, and Spark job management, including auditing, monitoring, and performance tuning.

Key Responsibilities:
  • Cloud DevOps & Big Data Administration:
    • Manage and optimize big data environments on AWS, with a focus on efficient administration and maintenance.
    • Leverage AWS Lake Formation to design and implement data lakehouses versus data fabric architectures, ensuring data integrity and accessibility.
  • Data Engineering & Spark Management:
    • Develop and maintain Spark jobs, with a focus on auditing, monitoring, and instrumentation to ensure reliability and performance.
    • Perform Spark performance tuning, including understanding and applying Spark 3+ features, such as Adaptive Query Execution (AQE) and job-level resource management.
    • Create services and tools to manage a multi-tenant environment, ensuring seamless data operations across tenants.
  • Infrastructure as Code (IaC):
    • Utilize Terraform for infrastructure provisioning and management, ensuring scalable and secure environments.
    • Integrate with AWS Glue and HIVE for data processing and management, optimizing workflows for large-scale data operations.
  • Data Storage & Management:
    • Work with data storage formats like Parquet and Hudi to optimize data storage and retrieval.
    • Implement and manage IAM policies for secure data access and management across AWS services.
  • Collaboration & Continuous Improvement:
    • Collaborate with cross-functional teams, including data scientists, analysts, and other engineers, to develop and deploy data solutions.
    • Continuously improve data engineering practices, leveraging new tools and techniques to enhance performance and efficiency.

Required Skills:

  • Cloud DevOps & Big Data:
    • Extensive experience in cloud DevOps and big data administration on AWS.
    • Proficiency in AWS Lake Formation, with a strong understanding of data lakehouse versus data fabric concepts.
  • Programming & Data Engineering:
    • Expertise in Python for data processing and automation.
    • Deep knowledge of Spark internals (Spark 3+ features), including architecture, events, system metrics, AQE, and job-level resource management.
  • Tech Stack:
    • Strong hands-on experience with Terraform for infrastructure management.
    • Experience with data formats such as Hudi and Parquet.
    • Familiarity with AWS Glue, HIVE, and IAM for data management and security.

Good to Have:

  • Additional Tools & Technologies:
    • Knowledge of Iceberg and its application in data storage and management.
    • Experience with Airflow for workflow automation.
    • Familiarity with Terragrunt for managing Terraform configurations.
    • Understanding of DynamoDB and its integration within data environments.

  • Senior Data Engineer AWS,DevOps,Big Data

    5 days ago

    London, Greater London, United Kingdom Salt Search Full time

    Job Title: Senior Data Engineer AWS,DevOps,Big DataWe are seeking a highly skilled Senior Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation, data product concepts, and Spark job management, including auditing, monitoring, and performance...

  • Senior Data Engineer AWS,DevOps,Big Data

    5 days ago

    London, Greater London, United Kingdom Salt Search Full time

    Job Title: Senior Data Engineer AWS,DevOps,Big DataWe are seeking a highly skilled Senior Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation, data product concepts, and Spark job management, including auditing, monitoring, and performance...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    Job Title: AWS Cloud Data EngineerJob Type: ContractLocation: RemoteJob Description:Data Engineer (Cloud, DevOps, Big Data) - Finance - RemoteJob Summary:Salt is seeking a highly skilled Data Engineer with expertise in cloud DevOps, big data administration, and data engineering on AWS. The ideal candidate will have a deep understanding of AWS Lake Formation,...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Salt, a leading company in the finance industry. As a Data Engineer, you will play a critical role in designing and implementing data engineering solutions on AWS, leveraging cloud DevOps and big data administration expertise.Key ResponsibilitiesCloud DevOps & Big Data...

  • Senior Data Engineer

    5 hours ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines and...

  • Senior Data Engineer

    7 hours ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines and...

  • Senior Data Engineer

    6 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key ResponsibilitiesDesign, implement, and maintain robust data pipelines and...

  • Senior Data Engineer

    6 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key ResponsibilitiesDesign, implement, and maintain robust data pipelines and...

  • Senior Data Engineer

    1 week ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data Engineer OpportunityTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines...

  • Senior Data Engineer

    1 week ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data Engineer OpportunityTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines...

  • Senior Data Engineer

    6 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines and...

  • Senior Data Engineer

    6 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time £70,000 - £90,000

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key Responsibilities:Design, implement, and maintain robust data pipelines and...

  • AWS Cloud Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Cloud Data Engineer to join our team at Salt. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems on AWS.Key ResponsibilitiesCloud DevOps and Big Data Administration: Manage and optimize big data environments on AWS, ensuring efficient...

  • AWS Cloud Data Engineer

    1 week ago

    London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Cloud Data Engineer to join our team at Salt. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems on AWS.Key ResponsibilitiesCloud DevOps and Big Data Administration: Manage and optimize big data environments on AWS, ensuring efficient...

  • AWS Cloud Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Cloud Data Engineer to join our team at Salt. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems on AWS.Key ResponsibilitiesCloud DevOps & Big Data Administration: Manage and optimize big data environments on AWS, ensuring efficient...

  • AWS Cloud Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    About the RoleWe are seeking a highly skilled Cloud Data Engineer to join our team at Salt. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems on AWS.Key ResponsibilitiesCloud DevOps & Big Data Administration: Manage and optimize big data environments on AWS, ensuring efficient...

  • Senior Data Architect

    2 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key ResponsibilitiesDesign and implement robust data pipelines and infrastructure...

  • Senior Data Architect

    2 days ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time

    Senior Data EngineerTribal Tech - The Digital, Data & AI Specialists is seeking an experienced Senior Data Engineer to join their innovative team. This role offers a unique opportunity to shape the future of data-driven decision-making within a rapidly growing organization.Key ResponsibilitiesDesign and implement robust data pipelines and infrastructure...

  • Senior Data Engineer

    2 weeks ago

    London, Greater London, United Kingdom Salt Full time

    Job DescriptionJob SummarySalt is seeking a highly skilled Cloud Data Engineer to join our team. As a Cloud Data Engineer, you will be responsible for designing, building, and maintaining large-scale data systems on AWS.Key ResponsibilitiesCloud and Big Data Administration:Manage and optimize big data environments on AWS, ensuring efficient administration...

  • Senior Data Architect

    11 hours ago

    London, Greater London, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at Tribal Tech - The Digital, Data & AI Specialists. As a Senior Data Engineer, you will play a key role in designing, implementing, and maintaining robust data pipelines and infrastructure.Key Responsibilities:Design and develop ETL/ELT processes using...