SC Cleared Python Data Engineer – Azure

3 days ago

London, Greater London, United Kingdom Montash Full time £200,000 - £800,000 per year

Job Title:
SC Cleared Python Data Engineer – Azure & PySpark

Contract Type:
12 Month Contract

Day Rate:
Up to £400 a day inside IR35

Location:
Remote or hybrid (as agreed)

Start Date:
January 5th 2026

Clearance required:
Must be holding active SC Clearance

We are seeking an experienced Python Data Engineer to support the design, development, and optimisation of Azure-based data pipelines.

The focus of this role is to deliver scalable, test-driven, and configuration-driven data processing solutions using Python, PySpark, Delta Lake, and containerised workloads.

This opportunity sits within a fast-paced engineering environment working closely with cloud, DevOps, and data science teams.

Key Responsibilities

  • Develop and maintain ingestion, transformation, and validation pipelines using Python and PySpark
  • Implement unit and BDD testing with Behave, including mocking, patching, and dependency management
  • Design and manage Delta Lake tables, ensuring ACID compliance, schema evolution, and incremental loading
  • Build and maintain containerised applications using Docker for development and deployment
  • Develop configuration-driven, modular, and reusable engineering solutions
  • Integrate Azure services including Azure Functions, Key Vault, and Blob Storage
  • Collaborate with cloud architects, data scientists, and DevOps teams on CI/CD processes and environment configuration
  • Tune and troubleshoot PySpark jobs for performance in production workloads
  • Maintain documentation and follow best practices in cloud security and data governance

Required Skills & Experience

  • Strong Python programming skills with test-driven development
  • Experience writing BDD scenarios and unit tests using Behave or similar tools
  • Skilled in mocking, patching, and dependency injection for Python tests
  • Proficiency in PySpark and distributed data processing
  • Hands-on experience with Delta Lake (transactional guarantees, schema evolution, optimisation)
  • Experience with Docker for development and deployment
  • Familiarity with Azure Functions, Key Vault, Blob Storage or Data Lake Storage Gen2
  • Experience working with configuration-driven systems
  • Exposure to CI/CD tools (Azure DevOps or similar)

Preferred Qualifications

  • Experience working with Databricks or Synapse
  • Knowledge of data governance, security, and best practices in the Azure ecosystem
  • Strong communication and collaboration skills, ideally within distributed teams

  • Python Data Engineer

    1 week ago

    London, Greater London, United Kingdom Hays Full time £32,000 - £45,000 per year

    Data Engineer - Security Clearance (SC), Python, Azure, BDDUp to £475 per day (Inside IR35)Remote / London6 monthsMy client is an International Consultancy who require a Security Cleared Data Engineer, with Active Security Clearance (SC), and strong Python skills to design and deploy scalable Data solutions in a containerized Azure environment.Key...

  • SC Cleared Data Professionals

    1 week ago

    London, Greater London, United Kingdom Morgan Hunt Full time £86,400 - £96,000 per year

    SC Cleared Data Professionals - Leading Government Organisation - Central LondonTitle:Urgent Requirement: SC Cleared Data Professionals (Engineer to Architect)Morgan Huntare proud to be partnering with a prestigiousleading UK government organisationbased in the heart ofLondon. We are supporting a critical digital transformation initiative and are seeking...

  • SC cleared

    23 hours ago

    London, Greater London, United Kingdom Methods Full time £24,300 - £25,940 per year

    Duration:initial contract until end of March 26, with possible extensionDaily rate :£495 inside IR35Location:RemoteClearance: active SCKey Responsibilities:Design, develop, and optimize data pipelines using Microsoft Fabric and Azure services (eg, Azure Data Factory, Azure Synapse Analytics, Azure Databricks).Build and maintain scalable, high-performance...

  • SC Cleared Senior Python Engineer

    1 week ago

    London, Greater London, United Kingdom iO Associates Full time £52,000 - £60,000 per year

    Senior Python Engineer - Open Data PlatformLondon or Manchester | Hybrid (2 days onsite)Inside IR35 | £650 per day | 12-month contractMust hold activeSC ClearanceWe are partnering with a major government department in their search for aSenior Python Engineerto support a Data transformation programme.What we're looking for:6+ years engineering experience,...

  • SC Cleared Data Engineer

    1 week ago

    London, Greater London, United Kingdom Axiom Software Solutions Limited Full time £96,000 per year

    Role: Data EngineerJob Type: ContractDay Rate: £400 per day (inside IR35)Duration: 6 Months + likely to be extendWork Location: ideally commutable to London, Bristol, or Manchester officePlease look for Security Cleared candidate.Name: Contact No: Email Id: Total Years of Experience Vendor Name Worked with Defra/Public sector : Yes/No If yes –...

  • SC Cleared Data Engineer

    5 days ago

    London, Greater London, United Kingdom Axiom Software Solutions Limited Full time £40,000 - £80,000 per year

    Role: Data EngineerJob Type: ContractDuration: 6 Months + likely to be extendWork Location: ideally commutable to London, Bristol, or Manchester officePosition: 1Start date: July'24Please look for Security Cleared candidate.Name: Contact No: Email Id: Total Years of Experience Vendor Name Worked with Defra/Public sector : Yes/No If yes – Reason for...

  • Senior Azure AI

    3 days ago

    London, Greater London, United Kingdom Cognitive Group | Part of the Focus Cloud Group Full time £60,000 - £120,000 per year

    Senior AI & Data Consultant - Microsoft FabricPermanent - FulltimeHomebased - Hybrid working in the UK(No sponsorship can be offered) - Must be SC Cleared or SC ClearableAs aSenior Azure AI & Data Consultantyou would be designing and delivering next-generation data solutions withAzure Synapse,Azure Databricks, andMicrosoft Fabric. This is an opportunity to...

  • Senior Data Engineer

    1 day ago

    London, Greater London, United Kingdom Data Idols Full time £70,000 - £75,000 per year

    Senior Data EngineerSalary: £70,00 - £75,000Location: Hybrid - London - 1 day per weekWe are currently looking for a Senior Data Engineer to join our client's forward-thinking, collaborative, and technology-driven team. In this role, you'll take the lead in shaping and delivering modern cloud data solutions, working closely with a highly skilled Data & BI...

  • Senior Azure Data Engineer

    6 days ago

    London, Greater London, United Kingdom Primus Connect Full time £55,000 - £60,000 per year

    Senior Azure Data Engineer (Databricks, Delta Live Tables, Unity Catalog)3 - 6 monthsRemote (UK Candidates Only)£550 - £600 per day Outside IR35We are currently recruiting for an experienced Data Engineer skilled in Microsoft Azure and cloud computing concepts.As the Azure Data Engineer, you will work closely with a Microsoft & Databricks partner with...

  • Data Engineer

    5 days ago

    London, Greater London, United Kingdom Axiom Software Solutions Limited Full time £40,000 - £80,000 per year

    Job title: Data EngineerLocation: UKDuration: 6-month contract Inside IR35Working Mode: Fully RemoteVetting: Active SC ClearanceJob Description: Data Engineer Skills and Technologies Data Engineering: - Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc - Strong understanding of complex...