Data Engineer

1 week ago

Edinburgh Scotland, Scotland, United Kingdom Smart Data Foundry Full time

To apply, please submit your CV and a cover letter (we would like to know about your experience in developing tools and processes which improve data quality and reduces manual effort).


About Smart Data Foundry

Smart Data Foundry has a purpose to unlock the power of financial data as a force to improve people’s lives.


What we do

We enable the research ecosystem to flourish through the provision of research ready real financial data.


We create data-driven insights based on real financial data, that identify areas to inform policy change and enhance regulation.


Inspire financial innovation with aizle™, Smart Data Foundry’s synthetic data engine, creating high utility synthetic data with no privacy issues when real-world data doesn’t exist or cannot be safely shared.


Role Purpose

Our Data Engineer develops tools and processes which ensure the quality of our data, reduces the manual effort involved in our work as we scale.


Responsibilities

  • Contributing to the design and development of data processing architecture and ongoing pipeline optimisation.
  • Developing and owning data cleaning, quality and validation processes.
  • Creating and explaining overviews and diagrams of processes and infrastructure to technical, semi-technical and non-technical users.
  • Working with data scientists, product owners, data partners and other internal and external customers to understand and deliver team commitments.
  • Undertaking development and evaluation of rapid prototypes to test features with end users and support tool selection.
  • Working with customers and partners to integrate data from multiple data sources, e.g. APIs (Application Programming Interfaces), sftp, manual file transfers, etc.
  • Monitoring system performance and contributing to infrastructure optimisation, considering utility and cost.
  • Testing and mentoring of less experienced software engineers and quality engineers.
  • Creating documentation, operating procedures and diagrams.
  • Working with the synthetic data, product and engineering teams to build a cloud-based, data driven SaaS platform.


Skills and Experience

Essential Technical Skills

  • Proven programming experience with Python in a Linux environment.
  • Good understanding of software development practices such as code reviews, unit testing, data validation, TDD, source management, continuous integration, and releasing software to live environments.
  • Knowledge of large-scale data processing technology, for example: Arrow, Hadoop, Cassandra, Spark, Hive, or Databricks.
  • Strong experience of batch ETL, data warehousing, database querying and building secure and scalable applications in a cloud-based environment, preferably AWS (Amazon Web Services).


Beneficial Technical Skills

  • Experience of working with infrastructure and container tools such as Terraform, Kubernetes and Docker/Podman.
  • Experience with modern software architecture paradigms such as microservices, event-driven, & serverless.
  • Experience of using Jira Software or similar project management tools.


Additional Attributes

  • Ability to deal with ambiguity and react quickly in an evolving and fast paced environment
  • Ability to work autonomously, under pressure and with conflicting and changing priorities
  • Excellent verbal and written communication skills
  • Personable approach with ability to develop positive working relationships at all levels within an organisation
  • Self-motivated and proactive
  • Collaborative, highly curious, with a desire to understand how Smart Data Foundry delivers its’ services
  • There may be a requirement to work on data partners sites, directly with data partner teams to build pipeline to make it easy to share data with us
  • Appreciation of the sensitive nature of the financial data held by Smart Data Foundry and the need to abide by information governance best practices


Location

This role is based in our office in Edinburgh Futures Institute in Edinburgh, therefore, candidates must be within commutable distance or willing to relocate.


Right to Work in the UK

Applicants must have the existing and ongoing right to work in the U.K. without restrictions as sponsorship for visas is not provided.

If invited for interview you will be required to evidence your right to work in the UK.


Benefits

A collaborative, innovative and impactful place to work. We give you support, nurture your talent and reward success. You will benefit from a competitive reward package, 40 days annual leave (including 6 bank hols), a defined contribution pension scheme (14% employer and 4.5% minimum employee contribution rates), family friendly initiatives and flexible working.

  • Senior Data Engineer

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom MBN Solutions Full time

    Senior Data Engineer Up to £70,000 + Benefits Edinburgh (Hybrid - 1-2 days per week on-site) About the RoleMBN Solutions are proud to support a long-standing client in their search for a Senior Data Engineer to join their team. The company are renowned for keeping up-to-date with the latest technology trends, having highly skilled technology teams, and...

  • Senior Data Engineer

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Intec Select Full time

    As a Senior Data Engineer within the commodities industry, you will design, build, manage, and optimise data pipelines and models used by their Data Scientists and Traders. You'll work closely with Data Management teams on governance and security, and with business stakeholders and IT teams to deliver these data solutions effectively into production.Key...

  • Finance Data Engineer

    2 weeks ago

    Edinburgh, Scotland, United Kingdom NatWest Full time

    Join us as a Finance Data EngineerThis is an exciting opportunity to use your technical expertise to collaborate with colleagues and build effortless, digital first customer experiencesYou’ll be simplifying the bank by developing innovative data driven solutions, using insight to be commercially successful, and keeping our customers’ and the bank’s...

  • Site Engineer

    2 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom C&P Recruitment Full time

    Our client is a leading civil contractor with a strong reputation for delivering high-quality infrastructure and groundworks for new build housing projects. They are committed to excellence and innovation, ensuring that every project they undertake meets the highest standards of safety, quality, and client satisfaction.Job DescriptionOur client is looking...

  • Software Engineer

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom McFall Recruitment Limited Full time

    Software EngineerAre you ready to take your career to the next level? We are looking for a dynamic, results-driven individual to join our team as a Software Engineer. If you have a passion for data, a knack for problem-solving, and the ability to thrive in a fast-paced environment, this could be the perfect opportunity for you!Essential Qualifications &...

  • Mid/Senior Python Engineer

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Bright Purple Full time

    Mid/Senior Python Engineer - Edinburgh/Hybrid (1 office day)About usWe are looking for a Mid/Senior Python Engineer to join a product-based tech company with a strong focus on data. The company is growing with an Agile development team approaching 15 staff with 4 specifically focussed on Python.BenefitsAs well as offering a competitive salary of up to...

  • Data Platform Consultant

    3 weeks ago

    Glasgow, Scotland, Scotland, United Kingdom Shape IT Recruitment Full time

    Join my client's dynamic team a Senior/Principal Data Platform Consultant. You'll design and implement innovative data solutions for diverse clients, from start-ups to government departments. Your expertise in Microsoft Data & AI products (Azure, Synapse, Power BI) and data engineering will drive impactful business outcomes. This client-facing role...

  • Full Stack Engineer

    1 month ago

    Edinburgh, Scotland, Scotland, United Kingdom KBC Technologies Group Full time

    You will be joining the innovative Information Security Cybertech Engineering team as a full stack software engineer in a global team responsible for developing web applications (Java/Angular), data engineering (Python) and solutions with third party products. You will be working in a collaborative and inclusive fast-paced environment that leverages agile...

  • Data Architect

    3 months ago

    Glasgow, Scotland, Scotland, United Kingdom BJSS Full time

    Do you want to work with cutting edge data and analytics technologies to help our clients unlock their data potential and value?Do you want to lead and mentor teams of expert technologists on exciting data projects?Do you want to work in an environment that embraces your drive to learn?About UsWe’re an award-winning innovative tech consultancy - a team of...

  • Analog/Mixed Signal IC Design Engineering Manager

    4 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom IC Resources Full time

    £90k - £100k + Bonus + RSUsOne of the world’s leading mixed-signal Semiconductor companies currently seeks an Analog/Mixed Signal IC Design Engineering Manager to join them at their established design centre located in the popular city of Edinburgh!The Analog/Mixed Signal IC Design Engineering Manager will manage a team of 10 - 15 Analog and Digital IC...

  • Senior Development Engineer

    2 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Spacelabs Healthcare Full time

    Company Profile:At Spacelabs Healthcare, we are on a mission to provide continuous innovation in healthcare technology for better clinical and economic outcomes. Our scalable solutions deliver critical patient data across local and remote systems, enable better-informed decisions, increase efficiencies, and create a safer environment for patients.The...

  • Senior / Principal Database Researcher

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Nexere Consulting Full time

    Our client are looking to recruit people whose skills and interests include transactional and/or analytical database management systems, storage engines, indexing engines, concurrent/parallel algorithms and data structures, distributed computing, graph theory and graph algorithm design, programming/query languages and fault-tolerant systems (especially...

  • Senior Software Engineer

    2 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom ALOIS Solutions Full time

    Job Summary: Senior Python Software Developer with experience of PyQt, PySide to develop a site information metadata collection application with offline synchronisation features.We are seeking a highly skilled Senior Software Developer with expertise in PyQt, and/or PySide to join our dynamic team on a contract basis. This project involves creating a robust,...

  • Junior Data Analyst

    5 days ago

    Glasgow, Scotland, Scotland, United Kingdom Albany Beck Full time

    Junior Data AnalystWant to start a career in tech?Albany Beck are hiring for passionate individuals to join the Pioneer Technology Programme, to learn the skills to become Junior Data Analysts. This will give you the opportunity to channel your enthusiasm and be part of an Agile team who are always working with exciting technologies. At the end of a short...

  • Martech Solutions Specialist

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Wyoming Interactive Part time

    Wyoming Interactive is a digital solutions agency who support industry leaders in Life Sciences, Financial Services and Technology through our Martech and Digital Analytics services, helping our clients better connect and communicate with customers.We believe in making a difference to people’s lives through technology in an environment that encourages...

  • Senior C++ Developer

    3 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom iO Associates - UKEU Full time

    We've recently partnered with a technology business in growth mode, focused on developing cutting edge products they are now seeking a senior C++ developer to join the team on a permanent basis. If you are passionate about developing new systems and software in a small agile company this could be the role for you. Responsibilities:Lead in the invention...

  • Infrastructure Engineer

    3 weeks ago

    Glasgow, Scotland, Scotland, United Kingdom PURVIEW Full time

    Role- Infrastructure EngineerLocation- Glasgow, UK (3 days onsite every week is mandatory)Duration- 6 months extendablePrimary Skills : Networking Knowledge and Splunk Reporting and Development.Secondary Skills : Quantum / Cloud Computing / Linux / VDIGood to have knowledge : Python knowledge Roles and ResponsibilitiesAnalyze current state of data sources...

  • Sales Development Representative

    2 weeks ago

    Edinburgh, Scotland, Scotland, United Kingdom Pareto Full time

    Job Title: Sales Development RepresentativeLocation: EdinburghSalary: £25k basic, with OTE taking your package up to £35kSector: IT Services/Consulting Our multi-award winning client implements and transforms the methodologies around how companies use data to take informed, strategic decisions. Microsoft Gold Partners, they work with a huge range of...

  • Engineering Manager

    4 weeks ago

    Edinburgh, Scotland, United Kingdom Escape Recruitment Services Full time

    Engineering Shift Manager (several roles) required to lead a maintenance team on shift in a fast paced & growing FMCG business for a market leading company commutable from Falkirk, West Lothian, Stirling & Fife. This position is based on a day/night shift pattern. Based in West Edinburgh.The Role:Provide engineering advice and coaching where necessary.Manage...

  • Cloud Engineer

    3 weeks ago

    Glasgow, Scotland, Scotland, United Kingdom Meraki Talent Ltd Full time

    Cloud Operations Engineer (Junior/Mid)Permanent RoleLocation: Glasgow or EdinburghHybrid Work ModelPosted day: 27 Aug 2024CVs by ASAPStart date: Sep - Dec 2024Meraki Talent’s client is on the lookout for a Cloud Ops Engineer to join their team. They are a niche software house who are continuously growing across the finance sector, investing heavily in...