Data Engineer

1 day ago

Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time
About the Role

We are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort involved in our work as we scale.

Key Responsibilities
  • Design and develop data processing architecture and ongoing pipeline optimisation.
  • Develop and own data cleaning, quality and validation processes.
  • Create and explain overviews and diagrams of processes and infrastructure to technical, semi-technical and non-technical users.
  • Work with data scientists, product owners, data partners and other internal and external customers to understand and deliver team commitments.
  • Undertake development and evaluation of rapid prototypes to test features with end users and support tool selection.
  • Work with customers and partners to integrate data from multiple data sources, e.g. APIs, sftp, manual file transfers, etc.
  • Monitor system performance and contribute to infrastructure optimisation, considering utility and cost.
  • Test and mentor less experienced software engineers and quality engineers.
  • Create documentation, operating procedures and diagrams.
  • Work with the synthetic data, product and engineering teams to build a cloud-based, data driven SaaS platform.
Requirements
  • Proven programming experience with Python in a Linux environment.
  • Good understanding of software development practices such as code reviews, unit testing, data validation, TDD, source management, continuous integration, and releasing software to live environments.
  • Knowledge of large-scale data processing technology, for example: Arrow, Hadoop, Cassandra, Spark, Hive, or Databricks.
  • Strong experience of batch ETL, data warehousing, database querying and building secure and scalable applications in a cloud-based environment, preferably AWS (Amazon Web Services).
Desirable Skills
  • Experience of working with infrastructure and container tools such as Terraform, Kubernetes and Docker/Podman.
  • Experience with modern software architecture paradigms such as microservices, event-driven, & serverless.
  • Experience of using Jira Software or similar project management tools.
About Us

Smart Data Foundry has a purpose to unlock the power of financial data as a force to improve people's lives. We enable the research ecosystem to flourish through the provision of research ready real financial data. We create data-driven insights based on real financial data, that identify areas to inform policy change and enhance regulation. We inspire financial innovation with aizleTM, Smart Data Foundry's synthetic data engine, creating high utility synthetic data with no privacy issues when real-world data doesn't exist or cannot be safely shared.

What We Offer

A collaborative, innovative and impactful place to work. We give you support, nurture your talent and reward success. You will benefit from a competitive reward package, 40 days annual leave (including 6 bank hols), a defined contribution pension scheme (14% employer and 4.5% minimum employee contribution rates), family friendly initiatives and flexible working.

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality of our data, and reducing manual effort involved in our work.Key ResponsibilitiesDesign and develop data processing...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality of our data, and reducing manual effort involved in our work.Key ResponsibilitiesDesign and develop data processing...

  • Data Engineer

    3 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    Job Title: Data EngineerAbout the Role:We are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality of our data, and reducing manual effort involved in our work.Key Responsibilities:Design and develop...

  • Data Engineer

    3 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    Job Title: Data EngineerAbout the Role:We are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality of our data, and reducing manual effort involved in our work.Key Responsibilities:Design and develop...

  • Senior Data Engineer

    2 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing data processing architecture and pipelines that ensure the quality and integrity of our data.Key ResponsibilitiesDevelop and own data cleaning, quality, and...

  • Senior Data Engineer

    2 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing data processing architecture and pipelines that ensure the quality and integrity of our data.Key ResponsibilitiesDevelop and own data cleaning, quality, and...

  • Data Engineer

    7 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality of our data and reducing manual effort involved in our work as we scale.Key Responsibilities:Design and develop data processing architecture and optimize ongoing...

  • Data Engineer

    7 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality of our data and reducing manual effort involved in our work as we scale.Key Responsibilities:Design and develop data processing architecture and optimize ongoing...

  • Data Engineer

    7 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing pipelines and architectures to meet business...

  • Data Engineer

    7 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing pipelines and architectures to meet business...

  • Data Engineer

    1 week ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort involved in our work as we scale.Key ResponsibilitiesDesign and develop data processing architecture and optimize...

  • Data Engineer

    1 week ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort involved in our work as we scale.Key ResponsibilitiesDesign and develop data processing architecture and optimize...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our financial data, reduce manual effort, and enable the research ecosystem to flourish.Key ResponsibilitiesDesign and Development: Contribute to the...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our financial data, reduce manual effort, and enable the research ecosystem to flourish.Key ResponsibilitiesDesign and Development: Contribute to the...

  • Senior Data Engineer

    1 day ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDevelop and own data cleaning, quality, and validation...

  • Senior Data Engineer

    1 day ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDevelop and own data cleaning, quality, and validation...

  • Senior Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing...

  • Senior Data Engineer

    2 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing...

  • Senior Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing...

  • Senior Data Engineer

    2 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing...