Data Engineer

2 days ago

Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time
Job Title: Data Engineer

We are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality of our data and reducing manual effort involved in our work as we scale.

Key Responsibilities:
  • Design and develop data processing architecture and optimize ongoing pipelines.
  • Develop and own data cleaning, quality, and validation processes.
  • Create and explain overviews and diagrams of processes and infrastructure to technical, semi-technical, and non-technical users.
  • Work with data scientists, product owners, data partners, and other internal and external customers to understand and deliver team commitments.
  • Develop and evaluate rapid prototypes to test features with end users and support tool selection.
  • Integrate data from multiple sources, including APIs, sftp, and manual file transfers.
  • Monitor system performance and contribute to infrastructure optimization, considering utility and cost.
  • Test and mentor less experienced software engineers and quality engineers.
  • Create documentation, operating procedures, and diagrams.
  • Collaborate with the synthetic data, product, and engineering teams to build a cloud-based, data-driven SaaS platform.
Requirements:
  • Proven programming experience with Python in a Linux environment.
  • Good understanding of software development practices, including code reviews, unit testing, data validation, TDD, source management, continuous integration, and releasing software to live environments.
  • Knowledge of large-scale data processing technology, such as Arrow, Hadoop, Cassandra, Spark, Hive, or Databricks.
  • Strong experience of batch ETL, data warehousing, database querying, and building secure and scalable applications in a cloud-based environment, preferably AWS.
Desirable Skills:
  • Experience with infrastructure and container tools, such as Terraform, Kubernetes, and Docker/Podman.
  • Experience with modern software architecture paradigms, such as microservices, event-driven, and serverless.
  • Experience with Jira Software or similar project management tools.
What We Offer:
  • A collaborative, innovative, and impactful place to work.
  • A competitive reward package.
  • 40 days annual leave (including 6 bank holidays).
  • A defined contribution pension scheme.
  • Family-friendly initiatives.
  • Flexible working.

  • Data Engineer

    1 day ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing pipelines and architectures to meet business...

  • Data Engineer

    1 day ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing pipelines and architectures to meet business...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort involved in our work as we scale.Key ResponsibilitiesDesign and develop data processing architecture and optimize...

  • Data Engineer

    4 days ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our data and reduce manual effort involved in our work as we scale.Key ResponsibilitiesDesign and develop data processing architecture and optimize...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our financial data, reduce manual effort, and enable the research ecosystem to flourish.Key ResponsibilitiesDesign and Development: Contribute to the...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that ensure the quality of our financial data, reduce manual effort, and enable the research ecosystem to flourish.Key ResponsibilitiesDesign and Development: Contribute to the...

  • Senior Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our data engineering team, you will play a critical role in designing and developing our data processing architecture and ensuring the quality of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to...

  • Senior Data Engineer

    1 week ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Smart Data Foundry. As a Senior Data Engineer, you will play a key role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to...

  • Senior Data Engineer

    1 week ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Smart Data Foundry. As a Senior Data Engineer, you will play a key role in developing and maintaining our data processing architecture, ensuring the quality and integrity of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to...

  • Senior Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our team at Smart Data Foundry. As a Senior Data Engineer, you will play a key role in developing and maintaining our data processing architecture and ensuring the quality of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to ensure data...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality of our financial data and reducing manual effort involved in our work as we scale.Key ResponsibilitiesDesign and Development: Contribute to the design and development of data processing...

  • Data Engineer

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in ensuring the quality of our financial data and reducing manual effort involved in our work as we scale.Key ResponsibilitiesDesign and Development: Contribute to the design and development of data processing...

  • Data Architect

    2 weeks ago

    Edinburgh, Edinburgh, United Kingdom Smart Data Foundry Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a key member of our data engineering team, you will play a critical role in designing and developing our data processing architecture and ensuring the quality of our data.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to...

  • Data Engineer

    2 days ago

    Edinburgh, Edinburgh, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Nine Twenty Recruitment. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure and pipelines.Key Responsibilities:Design and develop scalable ETL pipelines using AWS cloud technologies to ensure efficient...

  • Data Engineer

    1 day ago

    Edinburgh, Edinburgh, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Nine Twenty Recruitment. As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining our data infrastructure and pipelines.Key Responsibilities:Design and develop scalable ETL pipelines using AWS cloud technologies to ensure efficient...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data EngineerAbout the Role:We are seeking a highly skilled Data Engineer to join our data team. As a Data Engineer, you will play a pivotal role in designing, implementing, and maintaining the data infrastructure and pipelines that are crucial for processing data efficiently and making it available for analytics and strategic decisions.Key...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Nine Twenty Recruitment Full time

    Job Title: Data EngineerAbout the Role:We are seeking a highly skilled Data Engineer to join our data team. As a Data Engineer, you will play a pivotal role in designing, implementing, and maintaining the data infrastructure and pipelines that are crucial for processing data efficiently and making it available for analytics and strategic decisions.Key...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Iconic Resourcing Full time

    About the RoleWe are seeking a skilled Data Engineer to join our dynamic team in Edinburgh. As a Data Engineer, you will play a key role in shaping the future of data architecture within our organization.Key ResponsibilitiesDesign and develop robust data pipelines and ETL processes to ensure data integrity and security.Collaborate with cross-functional teams...

  • Data Engineer

    5 days ago

    Edinburgh, Edinburgh, United Kingdom Iconic Resourcing Full time

    About the RoleWe are seeking a skilled Data Engineer to join our dynamic team in Edinburgh. As a Data Engineer, you will play a key role in shaping the future of data architecture within our organization.Key ResponsibilitiesDesign and develop robust data pipelines and ETL processes to ensure data integrity and security.Collaborate with cross-functional teams...

  • AWS Data Engineer

    18 hours ago

    Edinburgh, Edinburgh, United Kingdom Society of Research Software Engineering Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Smart Data Foundry. As a Data Engineer, you will play a critical role in developing tools and processes that improve data quality and reduce manual effort.Key ResponsibilitiesDesign and develop data processing architecture and pipelines to ensure high-quality data.Collaborate...