Data Engineer

3 weeks ago

Central Inverclyde, United Kingdom Smart Data Foundry Full time
About the Role

We are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality and integrity of our financial data.

Key Responsibilities
  • Design and develop data processing architectures and pipelines to optimize data quality and reduce manual effort.
  • Develop and own data cleaning, quality, and validation processes to ensure data accuracy and consistency.
  • Create and explain technical documentation, diagrams, and overviews of processes and infrastructure to technical, semi-technical, and non-technical stakeholders.
  • Collaborate with data scientists, product owners, data partners, and other internal and external customers to understand and deliver team commitments.
  • Develop and evaluate rapid prototypes to test features with end users and support tool selection.
  • Work with customers and partners to integrate data from multiple sources, including APIs, sftp, and manual file transfers.
  • Monitor system performance and contribute to infrastructure optimization, considering utility and cost.
  • Test and mentor less experienced software engineers and quality engineers.
  • Create documentation, operating procedures, and diagrams to support data-driven decision making.
  • Contribute to the development of a cloud-based, data-driven SaaS platform with the synthetic data, product, and engineering teams.
Requirements
  • Proven programming experience with Python in a Linux environment.
  • Good understanding of software development practices, including code reviews, unit testing, data validation, TDD, source management, continuous integration, and releasing software to live environments.
  • Knowledge of large-scale data processing technology, including Arrow, Hadoop, Cassandra, Spark, Hive, or Databricks.
  • Strong experience of batch ETL, data warehousing, database querying, and building secure and scalable applications in a cloud-based environment, preferably AWS.
Desirable Skills
  • Experience working with infrastructure and container tools, such as Terraform, Kubernetes, and Docker/Podman.
  • Experience with modern software architecture paradigms, including microservices, event-driven, and serverless.
  • Experience using Jira Software or similar project management tools.
About Us

Smart Data Foundry is a purpose-driven organization that aims to unlock the power of financial data to improve people's lives. We enable the research ecosystem to flourish through the provision of research-ready real financial data.

We create data-driven insights based on real financial data to identify areas for policy change and enhance regulation. Our synthetic data engine, aizleTM, creates high-utility synthetic data with no privacy issues when real-world data doesn't exist or cannot be safely shared.

We are a collaborative, innovative, and impactful place to work, offering a competitive reward package, 40 days annual leave, a defined contribution pension scheme, family-friendly initiatives, and flexible working.

  • Data Engineer

    1 week ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing pipeline optimisationDevelop and...

  • Data Engineer

    1 week ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing pipeline optimisationDevelop and...

  • Data Engineer

    1 week ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing pipeline optimisationDevelop and...

  • Data Engineer

    1 week ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and ongoing pipeline optimisationDevelop and...

  • Senior Data Engineer

    17 hours ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDevelop and own data cleaning, quality, and validation...

  • Senior Data Engineer

    19 hours ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our engineering team, you will be responsible for designing and developing tools and processes that ensure the quality of our data and reduce manual effort.Key ResponsibilitiesDevelop and own data cleaning, quality, and validation...

  • Senior Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Smart Data Foundry. As a Senior Data Engineer, you will play a key role in developing and maintaining our data processing architecture and ensuring the quality of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to ensure data...

  • Data Architect

    2 weeks ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About Smart Data FoundrySmart Data Foundry is a pioneering organization that aims to unlock the power of financial data to improve people's lives. Our mission is to enable the research ecosystem to flourish through the provision of research-ready real financial data.Job OverviewWe are seeking a highly skilled Data Engineer to play a critical role in...

  • Data Architect

    2 weeks ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About Smart Data FoundrySmart Data Foundry is a pioneering organization that aims to unlock the power of financial data to improve people's lives. Our mission is to enable the research ecosystem to flourish through the provision of research-ready real financial data.Job OverviewWe are seeking a highly skilled Data Engineer to play a critical role in...

  • Senior Data Processing Specialist

    3 weeks ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality and integrity of our financial data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to optimize data quality and reduce manual effort.Develop and own...

  • Data Quality and Integration Expert

    3 weeks ago

    Central, Inverclyde, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality and integrity of our financial data.Key ResponsibilitiesDesign and develop data processing architectures and pipelines to optimize data quality and reduce manual effort.Develop and own...

  • Senior Data Engineer

    23 hours ago

    Central, Inverclyde, United Kingdom Bright Purple Full time

    Senior Data Engineer OpportunityBright Purple is partnering with a leading tech company in Edinburgh to find a talented Senior Data Engineer to lead the design and development of internal platforms and data solutions.This is a hands-on role that requires a strong technical background in data engineering, with a passion for problem-solving and a flexible...

  • Senior Data Engineer

    1 day ago

    Central, Inverclyde, United Kingdom Bright Purple Full time

    Senior Data Engineer OpportunityBright Purple is partnering with a leading tech company in Edinburgh to find a talented Senior Data Engineer to lead the design and development of internal platforms and data solutions.This is a hands-on role that requires a strong technical background in data engineering, with a passion for problem-solving and a flexible...

  • Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data Engineer - Cloud ExpertCompany: Nine Twenty RecruitmentJob Type: Full-timeLocation: RemoteAbout Us:Nine Twenty Recruitment is a leading recruitment agency specializing in placing top talent in the tech industry. We're passionate about connecting innovative companies with exceptional professionals who drive growth and success.About the...

  • Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data Engineer - Cloud ExpertCompany: Nine Twenty RecruitmentJob Type: Full-timeLocation: RemoteAbout Us:Nine Twenty Recruitment is a leading recruitment agency specializing in placing top talent in the tech industry. We're passionate about connecting innovative companies with exceptional professionals who drive growth and success.About the...

  • Senior Data Engineer

    3 weeks ago

    Central, Inverclyde, United Kingdom Nine Twenty Recruitment Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our data team at Nine Twenty Recruitment. As a Senior Data Engineer, you will play a pivotal role in designing, implementing, and maintaining the data infrastructure and pipelines that are crucial for processing data efficiently and making it available for analytics and strategic...

  • Senior Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Nine Twenty Recruitment Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our data team at Nine Twenty Recruitment. As a key member of our team, you will be responsible for designing, implementing, and maintaining our data infrastructure and pipelines.Key ResponsibilitiesDesign and develop scalable ETL pipelines using AWS cloud technologies to process data...

  • Senior Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Nine Twenty Recruitment Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our data team at Nine Twenty Recruitment. As a key member of our team, you will be responsible for designing, implementing, and maintaining our data infrastructure and pipelines.Key ResponsibilitiesDesign and develop scalable ETL pipelines using AWS cloud technologies to process data...

  • Senior Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Bright Purple Full time £65,000 - £75,000

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Bright Purple. As a key member of our data engineering team, you will be responsible for designing and developing internal platforms and data solutions that support our mission-driven tech-for-good initiatives.Key ResponsibilitiesLead the design and development of data...

  • Senior Data Engineer

    2 weeks ago

    Central, Inverclyde, United Kingdom Bright Purple Full time £65,000 - £75,000

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Bright Purple. As a key member of our data engineering team, you will be responsible for designing and developing internal platforms and data solutions that support our mission-driven tech-for-good initiatives.Key ResponsibilitiesLead the design and development of data...