Principal Data Platforms Engineer

Simple Machines. Data Engineered to Life™ 

Simple Machines is a leading independent boutique technology firm with a global presence, including teams in London, Sydney, San Francisco, and New Zealand. We specialise in creating technology solutions at the intersection of data, AI, machine learning, data engineering, and software engineering. Our mission is to help enterprises, technology companies, and governments better connect with and understand their organisations, their people, their customers, and citizens. We are a team of creative engineers and technologists dedicated to unleashing the potential of data in new and impactful ways. We design and build bespoke data platforms and unique software products, create and deploy intelligent systems, and bring engineering expertise to life by transforming data into actionable insights and tangible outcomes. We engineer data to life™. 

The Role:

A Principal Data Platforms Engineer at Simple Machines embodies a blend of deep technical expertise and consulting acumen, essential for leading advanced data projects across diverse industries. This role involves architecting, designing and implementing impactful data-driven solutions using a variety of tools and platforms such as Databricks, Snowflake, Google Cloud, and AWS. As a key consultant, the Principal Data Platforms Engineer advises clients on optimising their data architecture, aligns technical solutions with business goals, and leads teams in delivering high-impact data-driven outcomes.

Technical Responsibilities: 

• Developing Data Solutions: Architect, design and implement data-driven solutions that integrate with clients' existing systems, using tools such as Databricks, Snowflake, Google Cloud and AWS. Incorporate modern data thinking like data products, data contracts and data mesh to foster a decentralised and consumer-oriented approach. 
• Data Pipeline Development: Develop robust data pipelines using tools like Spark, Flink, Airflow, and Kafka to enable real-time and batch data processing. Implement frameworks that support scalable and maintainable data flows, aligning with principles of data as a product. 
• Database and Storage Optimisation: Optimise and manage a variety of database technologies, including relational (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra), ensuring efficient data storage and retrieval. Focus on achieving optimal data accessibility and quality. 
• Big Data Technologies: Leverage big data technologies like Spark and Flink to manage large-scale data processing and analysis. Employ approaches that promote data democratisation and accessibility within the organisation. 
• Cloud Data Management: Implement and manage cloud-specific data services such as AWS Redshift, S3, Google BigQuery, and Google Cloud Storage. Utilise cloud architectures that enhance data sharing and collaboration across business units. 

• Security and Compliance: Ensure compliance with data security policies and regulations, implementing secure data practices across various platforms. Incorporate security by design in data infrastructure and solutions. 

Consulting Responsibilities:

• Client Advisory: Advise clients on the best data practices and technologies to meet their business needs and project objectives. Provide expertise in selecting the appropriate tools and designing scalable data architectures. 
• Project Leadership and Management: Lead project teams, ensuring timely delivery of services. Manage project scope, timeline, and resources effectively, often involving an agile or Kanban based approach. 
• Business Needs Analysis: Work closely with clients to understand their operational needs and strategic goals, translating business requirements into technical solutions. 
• Stakeholder Engagement: Communicate effectively with client stakeholders to align technology solutions with business strategies, ensuring clear understanding and agreement on project deliverables. 
• Training and Empowerment: Train client teams on new technologies and data management practices, empowering them to effectively use and manage the implemented systems. 
• Contributions to Business Development: Assist in business development efforts by contributing to proposals, statements of work, and by providing technical insights during client meetings and presentations. 
• Thought Leadership: Serve as a thought leader by staying abreast of industry trends and emerging technologies, incorporating innovative solutions into client projects. 

Requirements

Ideal Skills and Experience :

• Core Data Engineering Tools & Technologies: Proficient with SQL, Spark, and platforms like Databricks, Snowflake, along with various storage technologies such as S3 (AWS), BigQuery (Google Cloud), Cassandra, Mongo, Neo4J, and HDFS. Highly skilled in pipeline orchestration tools like Glue (AWS), Airflow, dbt, and streaming technologies such as Kafka, Kinesis (AWS), Pub/Sub (Google Cloud), and Event Hubs (Azure). 
• Presentation Layer Tools: Experience with presentation layer tools including Superset, PowerBI, Tableau, and Looker. 
• Data Science Tools: Exposure to, or an understanding of data science/MLOps tools like Sagemaker (AWS), Dataiku, MLflow. 
• Data Storage Expertise: Experienced with data warehousing technologies like BigQuery, Snowflake, Databricks, and proficient in handling various data storage formats including Parquet, Delta, ORC, Avro, and JSON, ensuring optimal data storage and retrieval strategies. 
• Building and Managing Large-scale Data Systems: Developed and managed large-scale data pipelines and data-intensive applications in a production environment. 
• Site Reliability Engineering: Demonstrated ability to build robust platforms and applications, leveraging SRE principles to define and measure goals and commitments with stakeholders. 

• Data Modelling Expertise: Experienced in data modelling with a deep understanding of the trade-offs of various approaches. 
• Infrastructure Configuration for Data Systems: Skilled in configuring infrastructure for data systems, with a preference for infrastructure-as-code practices using tools like Terraform and Pulumi. 
• Programming Languages: Proficient in programming languages such as Python, SQL and exposure to other languages such as Java, Scala, GoLang, or Rust a bonus. 
• Containerised Solutions Expertise: Knowledgeable in containerised solutions using Docker and Kubernetes, enhancing the deployment and scalability of applications. 
• CI/CD Implementation: Familiar with CI/CD tools such as GitHub Actions and ArgoCD, streamlining development processes and ensuring high-quality software delivery. 
• Testing Tools and Frameworks: Experience with testing tools and frameworks like DBT, Great Expectations, and Soda to maintain high data quality and reliability in complex data systems. 
• Commercial Application of Data Engineering Expertise: Demonstrated track record of expertise in Data Engineering applied across a range of industries and organisations in a commercial setting. 
• Agile Delivery and Project Management: Experienced in building solutions using agile, scrum or kanban delivery methods. 
• Consulting and Advisory Skills: Experience working in a professional services firm or technology consultancy, providing expert advice and tailored solutions to clients. Skilled in stakeholder engagement, understanding client needs, and translating these into actionable data engineering strategies. 
• Leadership in Project and Team Management: Lead teams and projects within consultancy settings, ensuring alignment of technical solutions with business objectives, and mentoring team members in best practices. 

Professional Experience and Qualifications:

• Professional Experience: At least 10+ years of data engineering or equivalent experience in a commercial, enterprise, or start-up environment. Consulting experience within a technology consultancy or professional services firm is highly beneficial. 
• Educational Background: Degree or equivalent experience in computer science or a related field. 

Benefits

What We Offer in the UK: 

• Salary: Competitive salary and benefits package. 

• Pension: Up to 5% employer contribution, matching up to a 5% employee contribution, for a total of up to 10%. 
• Annual Leave: 4 weeks standard + 1 week additional annual leave over Christmas shut down period, plus public holidays. 
• Your Day - No Questions Asked: One additional day off per year, no explanation required 
• Regular Lunches: Provided at team meet-ups and on workdays at Simple Machines' co-working space. 
• Health and Wellbeing Allowance: £1,250 allowance per year to be used for any food and non-alcoholic beverages during business hours, healthcare, gym memberships, sporting goods and accessories, and any wellness appointments. 
• Professional Development: £1,500 annual budget for training, courses, and conferences, with potential for additional funding. 
• Certifications: £2,500 annual budget for certifications and related courses. 
• Equipment Allowance: £1,500 for UK team members, plus Apple MacBook Pro laptops and necessary accessories. 
• Company Sick Leave: 10 days per annum, includes coverage for employee’s family. 
• Antenatal Support: Paid time off for antenatal appointments, including classes recommended by health professionals. 
• Terminal Illness Benefit: Three months' continuance of salary at full pay. 

Join Us: 

Simple Machines is a diverse and globally distributed team of individual talents. Everyone in the firm is among the best at what they do. That’s why they’re here. We have a collective obsession with the future and a passion to create real change through technology. If you’re someone who’s as passionate as we are about building a world-class technology company specialising in engineering for data, you’ll fit right in.