Data Engineer

Join to apply for the Data Engineer role at UK Tote Group At UK Tote Group, we’re on a mission to reimagine the future of pool betting — building a modern, data-driven betting experience for millions of racing fans. Our technology powers real-time insights, supports responsible gaming, and helps us deliver trusted, customer‑first products across the UK and international markets. As a Data Engineer, you’ll play a key role in designing, building, and optimising the Databricks-based Lakehouse that drives real-time data, analytics, and reporting across the Tote. You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted data at their fingertips. This is a hands‑on engineering role where you’ll collaborate across engineering, BI, and product teams to deliver scalable, secure, and governed data solutions under Unity Catalogue. What You’ll Be Doing Design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. Develop robust Bronze, Silver, and Gold Delta tables that serve as the foundation for analytics, APIs, and decision‑making tools using the Medallion Architecture. Integrate data from Kafka (MSK), AWS S3, and external APIs, ensuring seamless ingestion into the Lakehouse. Collaborate with BI teams to enable high‑performance Power BI dashboards through Databricks SQL Warehouses, making data more accessible and actionable across the business. Govern data under Unity Catalog, ensuring it is discoverable and secure. Implement and maintain CI/CD pipelines for Databricks jobs, notebooks, and DLT workflows. Monitor, tune, and troubleshoot pipeline performance using Databricks metrics, CloudWatch, and AWS Cost Explorer. Document data models, schemas, and lineage, maintaining a clear understanding of data flows and dependencies. Help ensure platform compliance with GDPR and Gambling Commission regulations. Champion best practices in data platform design, observability, and cost management. What We Are Looking For Experienced Data Engineer with proven expertise in building pipelines in Databricks and a strong grasp of Apache Spark (PySpark or Scala), including Structured Streaming. Experience with Kafka (MSK) and real‑time data ingestion, deep understanding of Delta Lake, Delta Live Tables, and the Medallion Architecture. Strong AWS background with services such as S3, Glue, Lambda, Batch, and IAM. Proficient in Python and SQL for data engineering and analytics. Comfortable implementing CI/CD pipelines using tools such as GitHub Actions, Azure DevOps, or Jenkins; solid experience with Git and Spark performance tuning. Collaborative, proactive attitude and ability to balance platform reliability with delivery speed. Advantage: experience with streaming architectures or data governance frameworks like Unity Catalog, familiarity with Power BI, Looker, or Tableau. Knowledge of infrastructure‑as‑code tools such as Terraform, AWS networking fundamentals, and cost management techniques using Photon and DBU monitoring. Analytical, detail‑oriented, self‑starter with strong communication skills and passion for automation, efficiency, and data quality. What’s in it for you? At the Tote you can expect a friendly working environment with a strong sense of teamwork and pride in what we do. Within this role you’ll develop a broad range of skills and experiences that can enhance your career at the Tote. Additionally, our company benefits package includes: Competitive Basic Salary Discretionary Bonus Scheme Company Shares Option Plan Contributory pension scheme Life insurance (4 × basic salary) Simply Health Cash Plan Holiday entitlement (33 days inclusive of bank holidays) Study Support and opportunity for progression and development Confidential 24/7 365 employee assistance helpline Agile and collaborative office environment with free parking, fruit, biscuits, and drinks Additional Information Seniority level: Not Applicable Employment type: Full‑time Job function: Information Technology Industries: Gambling Facilities and Casinos #J-18808-Ljbffr