SC Cleared Databricks Data Engineer

3 days ago

England, United Kingdom Montash Full time

Job Title: SC Cleared Databricks Data Engineer – Azure Cloud Contract Type: 12 month contract Day Rate: Up to £400 a day inside IR35 Location: Remote or hybrid (as agreed) Start Date: January 5th 2026 Clearance required: Must be holding active SC Clearance We are seeking an experienced Databricks Data Engineer to design, build, and optimise large-scale data workflows within the Databricks Data Intelligence Platform. The role focuses on delivering high-performing batch and streaming pipelines using PySpark, Delta Lake, and Azure services, with additional emphasis on governance, lineage tracking, and workflow orchestration. Client information remains confidential. Key Responsibilities Build and orchestrate Databricks data pipelines using Notebooks, Jobs, and Workflows Optimise Spark and Delta Lake workloads through cluster tuning, adaptive execution, scaling, and caching Conduct performance benchmarking and cost optimisation across workloads Implement data quality, lineage, and governance practices aligned with Unity Catalog Develop PySpark-based ETL and transformation logic using modular, reusable coding standards Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel Integrate Databricks assets with Azure Data Lake Storage, Key Vault, and Azure Functions Collaborate with cloud architects, data analysts, and engineering teams on end-to-end workflow design Support automated deployment of Databricks artefacts via CI/CD pipelines Maintain clear technical documentation covering architecture, performance, and governance configuration Required Skills and Experience Strong experience with the Databricks Data Intelligence Platform Hands-on experience with Databricks Jobs and Workflows Deep PySpark expertise, including schema management and optimisation Strong understanding of Delta Lake architecture and incremental design principles Proven Spark performance engineering and cluster tuning capabilities Unity Catalog experience (data lineage, access policies, metadata governance) Azure experience across ADLS Gen2, Key Vault, and serverless components Familiarity with CI/CD deployment for Databricks Solid troubleshooting skills in distributed environments Preferred Qualifications Experience working across multiple Databricks workspaces and governed catalogs Knowledge of Synapse, Power BI, or related Azure analytics services Understanding of cost optimisation for data compute workloads Strong communication and cross-functional collaboration skills

  • Databricks Engineer

    4 weeks ago

    England, United Kingdom iO Associates Full time

    Databricks Engineer - SC Cleared Fully Remote Duration: 6 Months Location: London Rate: £500-£550 per day (Inside IR35) Start Date: ASAP Are you an SC Cleared Databricks Engineer with a passion for building scalable data solutions? Do you have extensive experience in designing and optimising data pipelines, cloud-native platforms, and advanced analytics...

  • SC Cleared Databricks Data Engineer – Azure Cloud

    5 days ago

    England, United Kingdom Montash Full time

    Job Title: SC Cleared Databricks Data Engineer – Azure CloudContract Type: 12 month contractDay Rate: Up to £400 a day inside IR35Location: Remote or hybrid (as agreed)Start Date: January 5th 2026Clearance required: Must be holding active SC ClearanceWe are seeking an experienced Databricks Data Engineer to design, build, and optimise large-scale data...

  • SC Cleared Databricks Data Engineer – Azure Cloud

    5 days ago

    England, United Kingdom Montash Full time

    Job Title: SC Cleared Databricks Data Engineer – Azure Cloud Contract Type: 12 month contract Day Rate: Up to £400 a day inside IR35 Location: Remote or hybrid (as agreed) Start Date: January 5th 2026 Clearance required: Must be holding active SC Clearance We are seeking an experienced Databricks Data Engineer to design, build, and optimise large-scale...

  • SC cleared

    2 weeks ago

    England, United Kingdom Methods Full time

    Duration: initial contract until end of March 26, with possible extensionDaily rate : £495 inside IR35Location: RemoteClearance: active SC Key Responsibilities:Design, develop, and optimize data pipelines using Microsoft Fabric and Azure services (eg, Azure Data Factory, Azure Synapse Analytics, Azure Databricks).Build and maintain scalable,...

  • SC cleared

    4 weeks ago

    England, United Kingdom Methods Full time

    Duration: initial contract until end of March 26, with possible extension Daily rate : £495 inside IR35 Location: Remote Clearance : active SC Key Responsibilities: Design, develop, and optimize data pipelines using Microsoft Fabric and Azure services (eg, Azure Data Factory, Azure Synapse Analytics, Azure Databricks). Build and maintain scalable,...

  • Azure Cloud Data Engineer

    3 days ago

    England, United Kingdom Montash Full time

    Job Title: SC Cleared Databricks Data Engineer – Azure Cloud Contract Type: 12 month contract Day Rate: Up to £400 a day inside IR35 Location: Remote or hybrid (as agreed) Start Date: January 5th 2026 Clearance required: Must be holding active SC Clearance We are seeking an experienced Databricks Data Engineer to design, build, and optimise...

  • AWS Engineer

    4 weeks ago

    England, United Kingdom iO Associates Full time

    Databricks Engineer - SC Cleared Fully Remote Duration: 6 Months Location: London Rate: £500-£550 per day (Inside IR35) Start Date: ASAP Are you an SC Cleared Databricks Engineer with a passion for building scalable data solutions? Do you have extensive experience in designing and optimising data pipelines, cloud-native platforms, and advanced analytics...

  • Databricks Data Engineer

    1 week ago

    Manchester, Greater Manchester, England, United Kingdom EXL Full time £60,000 - £100,000 per year

    DescriptionEXL (NASDAQ: EXLS) is a global data and AI company that offers services and solutions to reinvent client business models, drive better outcomes and unlock growth with speed. Bringing together domain expertise with robust data, powerful analytics, cloud and AI to create agile, scalable solutions and execute complex operations for the world's...

  • Sc Cleared Systems Engineers

    2 weeks ago

    South West England, United Kingdom Sanderson Full time

    Role: SC Cleared System engineers Duration: 12 months On site in south west Rates negotiable Flexi time available Outside IR35 Our client is looking for SC Cleared System Engineers to join them ASAP with the following skill sets. Key skills: - Requires Model Based Engineering or Model Based System Engineering background and experience of Enterprise...

  • Data Solution Architect

    1 week ago

    England, United Kingdom Cognitive Group | Part of the Focus Cloud Group Full time

    Data Solution Architect - Microsoft Azure Data ServicesMicrosoft Fabrics, Databricks, and Azure AI - PresalesUK (Hybrid) - Must be SC Cleared or eligible for SC Clearance - no sponsorship Permanent What You’ll DoLead the data architecture workstream throughout the pre-sales cycle, helping customers envision innovative, scalable, value-driven...