Data Platform Data Engineer

1 week ago

London, UK, United Kingdom Sumitomo Mitsui Banking Corporation – SMBC Group Full time

Purpose of Job


We are seeking an experienced and skilled Data Platform Data Engineer to join the Data Platform Delivery Team in EMEA Data & Analytics Technology. The successful candidate will play a key role in designing, implementing, and maintaining robust data infrastructure and pipelines to support our organization's data-driven initiatives, ensuring optimal performance, scalability, flexibility, and reliability. The data platform is to be built in a hybrid cloud environment with Azure and some components residing in our on-prem data centre. The platform is to be built on a green field basis and legacy capability migrated to it. The role will include the following.


  • Understanding of business and data requirements and translation to data pipeline and systems design.
  • Implementation of agreed designs.
  • Mentoring and guiding other members of the team through collaboration and peer review.
  • Building production quality components.
  • Aligning the implementation of the Data Platform to the agreed strategy of the platform.
  • Deliver excellence in data ingestion and distribution for SMBC EMEA.


Background


The newly created EMEA Data & Analytics Technology team has been put in place to deliver data integration, reporting, and analytics services to the SMBC EMEA business. It exists within the wider ITSD department that is responsible for all IT delivery to the SMBC EMEA enterprise.

The team deliver data and analytics solutions to FO P&L business lines, cross business line departments (eg Finance, Risk, Operations, etc), and the Data Office as a key client. This is within a complex systems environment which has been built over several years and is looking to be re-architected for the next phase of SMBC EMEA growth.


Facts / Scale

  • Will be working in team of 40-50 IT staff within a department of 250+
  • Will deliver robust, reliable, performant, and maintainable components of the Data Platform for use across the SMBC EMEA entity covering capital markets and corporate banking activities.
  • Will attend the London office at least 2 days a week.


Accountabilities & Responsibilities

  • Design and implement scalable and efficient data pipelines, ETL processes, and data integration solutions to collect, process, and store large volumes of data.
  • Develop data models, schema designs, and data architecture frameworks to support diverse analytical and reporting needs.
  • Build and optimize data processing workflows using distributed computing frameworks available on Azure, our preferred cloud provider.
  • Implement data transformation logic to cleanse, validate, and enrich raw data for analysis and consumption by downstream applications.
  • Integrate data from various internal and external sources including databases, APIs, and streaming platforms into centralized data repositories and data warehouses.
  • Ensure data quality and consistency by implementing validation checks, data profiling, and error handling mechanisms.
  • Optimize data pipelines and processing algorithms for performance, reliability, and scalability to handle growing data volumes and user demands.
  • Identify and resolve performance bottlenecks, optimize SQL queries, and fine-tune system configurations for improved efficiency.
  • Implement data governance policies, access controls, and encryption mechanisms to ensure the confidentiality, integrity, and availability of sensitive data.
  • Collaborate with security and compliance teams to adhere to regulatory requirements and industry best practices.
  • Monitor data pipelines, job schedules, and system health metrics to proactively identify issues and ensure the availability of data infrastructure.
  • Perform routine maintenance tasks, data backups, and disaster recovery procedures to mitigate risks and ensure data continuity.
  • Create and maintain technical documentation including system architectures, data flow diagrams, and operational procedures.
  • Share knowledge and best practices with team members through code reviews, technical presentations, and training sessions.


Knowledge, Skills, Experience & Qualifications


Essential:

  • Minimum of 5 years of experience working as an analyst developer or data engineer in a data-centric environment.
  • Proven experience in designing and implementing end-to-end data solutions from ingestion to consumption.
  • Strong experience with Azure data PaaS services and data pipeline delivery on the Azure platform.
  • Experience of delivering data pipelines with Databricks.
  • Experience delivering data platforms with C#, JSON, XML, APIs, and message bus technology.
  • Strong knowledge of database systems, data modelling, and data integration technologies.
  • Strong Analytical and problem-solving skills with the ability to troubleshoot complex issues and propose effective solutions.
  • Ability to analyse data requirements, identify patterns, and design scalable data designs to meet business needs.
  • Strong experience of different data architectures including hub and spoke, data lakes, data fabrics, lakehouses.
  • Strong SQL skills and experience with relational databases (e.g., ORACLE, MS SQL) and NoSQL databases (e.g., MongoDB, Cassandra).
  • Good communication skills to explain and gain buy in for designs from other analyst developers in the team and other stakeholders.
  • Collaborative and curious practices to develop strong well thought through solutions.
  • Curiosity to learn new ways of delivering solutions and openness to new designs or other people’s ideas.


Desirable

  • Knowledge of complex systems estates with diverse data needs. Preferably in capital markets, corporate banking or asset management.
  • Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
  • Proficiency in programming languages such as Python, Scala, or Java for data processing and scripting tasks.
  • Experience of Agile practises and especially scrum of scrums and Atlassian tooling.
  • Experience with containerization and orchestration technologies such as Docker and Kubernetes.
  • Data warehouse virtualisation technologies eg Denodo, Snowflake, etc.
  • Coaching / mentoring of more junior staff.
  • Advanced certifications in data engineering, cloud computing, or related fields would be a plus.

  • Senior Data Engineer

    2 weeks ago

    London, UK, UK, United Kingdom Data Idols Full time

    Senior Data EngineerSalary: up to £85,000Location: London (Hybrid)Data Idols are working with an exciting scale-up revolutionising delivery logistics and moving items across the UK with efficiency and ease all whilst cutting emissions. Trusted by millions of customers already they are expanding their data team to manage increased volumes of data and demand....

  • Data Analyst

    1 month ago

    London, UK, UK, United Kingdom Data Idols Full time

    Data AnalystSalary: £55,000Location: London (Hybrid)Data Idols is collaborating with a large, worldwide publishing company that is continuously expanding its range of products and services. This company publishes significant research that has the potential to enhance society. With the aim of improving people's lives through books and journals at the...

  • Head of Data Architecture

    3 weeks ago

    London, UK, United Kingdom Paritas Recruitment - Data & Tech Full time

    We are seeking a Data Architect to take the lead in a newly formed department within a trading house. Leading a multi disciplinary team, we are seeking individuals who can come in and make the decisions on future architecture designs, contribute to a wider data strategy and face off to a range of senior stakeholders across the business. Experience in other...

  • Data Analyst

    1 month ago

    London,, UK, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time

    Data Analyst - up to £40k DOE - Hybrid 3 days on-siteNO VISA SPONSORSHIP AVAILABLEAbout the CompanyMy client is an independent digital agency creating captivating brand experiences. They blend data, creativity and technology to drive growth for clients. Their culture fosters work-life balance and continuous learning.The RoleJoin the Data & Insights team as...

  • Data Analyst

    4 weeks ago

    London,, UK, United Kingdom Tribal Tech - The Digital, Data & AI Specialists Full time

    Data Analyst - up to £40k DOE - Hybrid 3 days on-siteNO VISA SPONSORSHIP AVAILABLEAbout the CompanyMy client is an independent digital agency creating captivating brand experiences. They blend data, creativity and technology to drive growth for clients. Their culture fosters work-life balance and continuous learning.The RoleJoin the Data & Insights team as...

  • Director of Data Platforms

    2 days ago

    London,, UK, United Kingdom Talent Point Full time

    We are looking for a Director of Data Platforms to join a specialist bank rising to the challenge of complex cases or unconventional circumstances.Customers get the best-of-both worlds; with a bank that uniquely combines strong digital capabilities with human expertise and ingenuity to deliver the best outcomes. From personal and business savings accounts...

  • Data Platform Product Manager

    1 month ago

    London,, UK, United Kingdom La Fosse Full time

    Data Platform Product Manager - Data & Insights Business - £85,000 - London (Hybrid)Data Platform Product Manager required to join a leading Data & Insights business based in London. You will lead the development and enhancement of the data platforms, playing a pivotal role in driving the business vision forward. The RoleBe a data visionary, championing...

  • Global Data Protection Manager

    2 weeks ago

    London,, UK, United Kingdom Data Privacy Recruitment Ltd Full time

    Global Data Protection ManagerFantastic opportunity for a team leading senior data protection manager role in a world class Privacy/Data protection team. You will be joining an exciting large scale international events company who operate in 22 countries and 45 business sectors. Purpose of the role: As a leader within the Global Privacy Office and the wider...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Response Informatics Full time

    We are looking for a Lead Data Engineer to join our Data Engineering Team. In this senior role, you will be responsible for managing our Enterprise Data Warehouse and ensuring the efficient flow and quality of data.Your tasks will include implementing data processes, managing data architecture, designing ETL processes, and analyzing data for business...

  • Customer Data Platform Consultant

    4 weeks ago

    London,, UK, United Kingdom Freshminds Full time

    A global leader in the design, marketing and distribution of premium lifestyle products is seeking a customer data platform product manager, to join their global customer insights team.You will be responsible for leading the development of their customer data platform. The ideal candidate will understand customer facing businesses, have ROI-driven digital...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Eames Consulting Full time

    Eames Consulting are delighted to be supporting an established Investment management company that are looking for a Data engineer.Responsibilities:Develop and maintain complex SQL queries, customize reports, and optimize data usage.Create tools to address business challenges and empower users with third-party tools.Design and implement cloud-based data...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Eames Consulting Full time

    Eames Consulting are delighted to be supporting an established Investment management company that are looking for a Data engineer.Responsibilities:Develop and maintain complex SQL queries, customize reports, and optimize data usage.Create tools to address business challenges and empower users with third-party tools.Design and implement cloud-based data...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Vallum Associates Full time

    RequirementsAt least two years of professional experience as a data engineer with a proven ability to provide effective data solutions in a production setting.Knowledge of developing real-time data stream systems (ideally Kafka).Proven track record in developing data systems using PySpark and Apache Spark for batch processing.Capable of managing data intake...

  • Data Engineer

    4 weeks ago

    London,, UK, United Kingdom Vallum Associates Full time

    RequirementsAt least two years of professional experience as a data engineer with a proven ability to provide effective data solutions in a production setting.Knowledge of developing real-time data stream systems (ideally Kafka).Proven track record in developing data systems using PySpark and Apache Spark for batch processing.Capable of managing data intake...

  • Data Engineer

    2 days ago

    London,, UK, United Kingdom W Talent Full time

    Are you a Data Engineer with a demonstrable background of building and maintaining data pipelines in Azure? There is a high growth technology consulting business we are working with at the moment to help find people like you to add to their highly skilled team. If you are an Azure Data Engineer looking to apply your skills on interesting projects and learn...

  • Data Engineer

    1 month ago

    London,, UK, United Kingdom Hexegic Full time

    Data Engineers are responsible for aiding UK Health in developing a cleaned, validated, and discoverable data foundation to enable new analytical workflows and applications.They will need to work closely with UK Health subject matter experts to scope new data integrations, design a data model, and perform data cleansing and transformation within the data...

  • Data Protection Manager UK and International

    2 days ago

    London,, UK, United Kingdom Data Privacy Recruitment Ltd Full time

    Job Title: Data Protection ManagerOffice Location: Richmond (on hybrid working basis)Reports To: Global Data Protection ManagerPurpose of the role:Reporting to the Global Data Protection Manager, the role holder will assist the Global Privacy Office in ensuring that the business meet privacy obligations within the UK Hub jurisdictions, consisting of UK,...

  • Data Platform

    2 weeks ago

    London,, UK, United Kingdom ReMark Full time

    About ReMark (SCOR Digital Solutions)SCOR Digital Solutions (ReMark) is SCOR’s tech-hub for automated underwriting, claims, and HealthTech solutions. By combining SCOR’s comprehensive data and analytical expertise with the award-winning capabilities of our in-house product and technical teams, our solutions are helping insurers to transform the...

  • Data Engineer

    2 weeks ago

    London,, UK, United Kingdom Mastek Full time

    Job Description: Data EngineerLocation: [UK]SC Cleared: RequiredJob Type: Full-TimeExperience: 12-14 yearsJob description:We are seeking a highly skilled and motivated Data Engineer to join our dynamic team. The Data Engineer will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support data-driven...

  • Senior Principal Engineer, Data Platforms

    12 hours ago

    London, UK, United Kingdom London Stock Exchange Group Full time

    At LSEG, we are reimagining the way in which our customers consume our data and services. The Senior Principal Engineer, Data Platforms is key to the success of this vision. Reporting to the Group Head of Data Intelligence Technology, you will have a core team who are accountable for building, owning, and operating our strategic data platform in partnership...