Data Lake Engineer

7 months ago

London, United Kingdom SDG Group Full time

Requisitos mínimos:

Estar graduado en Ingeniería Informática o de Telecomunicaciones. Tener al menos 4 años de experiencia en un rol de Big Data Engineer, implementando y productivizando soluciones de captación, ingesta y transformación de datos con herramientas de procesamiento distribuido y/o procesamiento en streaming. Experiencia en arquitectura de datos (formato, estructura, particionado, modelado, capas y ciclo de vida) en ambientes Data Lake, Data Lakehouse, Kafka Topics & Partitioning, Schema Registry, etc. Además será importante que tengas experiencia en algunas de las siguientes herramientas: Lenguajes: Python, Java o Scala. Herramientas core del entorno de procesamiento en entornos Big Data, especialmente Apache Spark. Además es importante que tengas conocimientos sólidos de HDFS. Conocimiento de al menos una plataforma de datos con profundidad: Databricks, AWS EMR, AWS Glue, Elastic Search, GCP Dataflow & Dataproc, etc. Conocimiento en herramientas de orquestación como Apache Airflow, Azure Data Factory, Databricks workflows,... Conocimiento y experiencia en el uso de sistemas de control de versiones (VCS), esencialmente Git. Experiencia en la utilización de al menos una de las tres grandes nubes (AWS, Microsoft Azure o Google Cloud Platform), especialmente en lo relativo a los servicios orientados a procesamiento y disposición de datos.

Requisitos valorables:

Nos gusta estar al día en nuevas tecnologías, así que necesitamos personas que quieran estar en continua evolución y que tengan ganas de aprender nuevas tecnologías y probar nuevas herramientas. Para nosotrxs el trabajo en equipo es primordial, por lo que buscamos personas que impulsen ese buen ambiente y las sinergias entre compañerxs y diferentes equipos de trabajo. Qué conozcas lenguajes que están tomando relevancia ahora en el mundo data como Rust o Elixir es algo que valoraremos muy positivamente también. Cómo trabajamos en un entorno global, será un plus que puedas comunicarte en inglés. Conocimiento sobre DataOps, Observabilidad, patrones de diseño low-code/no-code,...

Acerca del puesto:

SDG Group es una firma de consultoría global focalizada exclusivamente en soluciones de data & analytics, cubriendo con ello toda la cadena de valor del dato, y proporcionando planes de carrera tanto de consultoría como de especialización para todos aquellos que quieran progresar en el mundo del dato. Además, a través de nuestra cultura Smart Work tendrás la posibilidad de definir el modelo de trabajo que más se adapte a ti y a tus necesidades personales, al mismo tiempo que puedas colaborar en iniciativas de gran impacto para clientes punteros en sus respectivas industrias.

Como data engineer del área de Data Technologies, tu misión será formar parte de equipos multidisciplinares con el objetivo de ayudar a nuestros clientes a abordar iniciativas de datos en entornos de alta complejidad técnica. Tendrás la posibilidad de utilizar un amplio espectro de tecnologías innovadoras para la construcción de soluciones de gestión de datos bajo diversos paradigmas de procesamiento y modelado, aplicando para ello los principios adecuados en términos de desarrollo y arquitectura.

Asimismo, gracias a nuestro path de especialista tendrás la oportunidad de crecer como ingeniero y/o arquitecto Big Data, contando con herramientas y recursos para tu continuo aprendizaje y crecimiento profesional, al mismo tiempo que te involucras en diversos retos que permitan tanto a SDG Group como a nuestros clientes explotar al máximo el valor de sus datos.

Responsabilidades:

Darás tu opinión técnica sobre los procesos y arquitecturas que se desarrollen para nutrir las conversaciones técnicas que se llevan a cabo durante las fases de diseño de la misma, priorizando estrategias dirigidas por metadatos y fundamentadas en DataOps. Participarás en la toma de requerimientos necesarios para arquitecturas y pipelines de datos. También llevarás a cabo tareas de desarrollo con diferentes tecnologías, aunque principalmente usamos Spark (tanto para procesos en batch como streaming). Adicionalmente, y aunque nuestros desarrollos habitualmente no se basen en SQL, es importante que tengas la soltura suficiente para hacer exploración de datos. Somos bastante agnósticos a la tecnología y a la infraestructura en nuestros proyectos, así que es posible que tengas experiencia proyectos en AWS, Azure, GCP u on-premises, por lo que será importante que no te importe estar cambiando de entorno con frecuencia. A veces también nos toca crear u optimizar tiempos de respuesta en motores de búsqueda, así que cualquier experiencia en Solr o Elastic nos gustará. Además, en algunos momentos también necesitaremos hacer pruebas de concepto y pruebas con tecnologías nuevas en el mercado y servicios cloud nuevos o que creamos que puedan ser útiles para nuestros proyectos o clientes, por lo que también podrás colaborar en tareas de innovación.

  • Data Lakes Specialist

    3 days ago

    London, Greater London, United Kingdom N Consulting Limited Full time

    **Job Title:** Data Lakes Specialist**Location:** London, UKWe are seeking an experienced Data Lakes Specialist to join our team at N Consulting Limited. As a key member of our Big Data team, you will be responsible for designing, building, and managing scalable data lakes using the Hadoop ecosystem.**Key Responsibilities:Design and implement data ingestion...

  • Data Lake Engineer

    2 weeks ago

    London, Greater London, United Kingdom Conexus Full time

    Azure Data Architect Job OpportunityWe are looking for a skilled Azure Data Architect to join our team in London. This is a 6-month contract, with the possibility of further extension. As a member of our team, you will be responsible for designing and developing cloud-based data architectures using Microsoft Azure. Key responsibilities include developing and...

  • Data Lake Architect

    2 weeks ago

    London, Greater London, United Kingdom Tesco Partners Full time

    Job OverviewTesco Partners is seeking a highly skilled Staff Analytics Engineer to join their Cyber Analytics team. This role requires a strong background in data engineering, analytics, and cybersecurity.About the RoleThis is an exciting opportunity for a talented individual to design, develop, and deploy scalable data lake and data warehouse solutions. The...

  • Senior Data Lake Specialist

    1 week ago

    City of London, United Kingdom Wilson Brown Limited Full time

    Job OverviewWe are seeking an experienced Data Lake Specialist to join our team at Wilson Brown Limited in Central London. This is a senior role with a focus on testing data ingestion and transformation processes.The ideal candidate will have extensive experience working with C# or Python, as well as a strong understanding of AWS services such as s3, Glue...

  • Python Software Engineer Data Lake

    1 day ago

    London, United Kingdom eFinancial Careers Full time

    **Python Software Engineer (Data Lake)** **London, Ropemaker Place** The aim of this project is to: create a data lake - an environment which contains all the data that the company acquires, expose the ability to easily connect various data sets in it and finally do complex data analysis. By creating easy access to the hugedata lake, it will allow us to...

  • Data Lake Automation Expert

    1 week ago

    City of London, United Kingdom Wilson Brown Limited Full time

    About the JobWe are seeking a highly experienced Senior QA Engineer to join our team at Wilson Brown Limited. This role offers a competitive salary of up to £65,000 + benefits and a hybrid working arrangement with two days in the London office per week.The successful candidate will have experience in ETL testing, strong problem-solving skills, and excellent...

  • Data Lake Quality Assurance Specialist

    1 week ago

    City of London, United Kingdom Wilson Brown Limited Full time

    We are Wilson Brown Limited, an established Insurance firm based in Central London. We are currently seeking a highly skilled Data Lake Quality Assurance Specialist to join our team. As a key member of our quality assurance team, you will be responsible for testing data ingestion and transformation processes on our newly implemented Data Lake.The ideal...

  • Cloud Data Lake Specialist

    2 weeks ago

    London, Greater London, United Kingdom Adecco UK LTD Full time

    Job Summary: Cloud Data Lake SpecialistSalary: £70,000 - £90,000 per annumWe are seeking a highly skilled Cloud Data Lake Specialist to join our team in Farringdon or Welwyn Garden City. As a key member of the Cyber Analytics team, you will be responsible for building data models, ETLs, and creating insights that drive cybersecurity strategy and...

  • Python Software Engineer Data Lake

    1 day ago

    London, United Kingdom eFinancial Careers Full time

    Python Software Engineer (Data Lake) London, Ropemaker Place The aim of this project is to: create a data lake - an environment which contains all the data that the company acquires, expose the ability to easily connect various data sets in it and finallydo complex data analysis. By creating easy access to the huge data lake, it will allow us to...

  • Cloud Data Engineering Manager

    4 weeks ago

    London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Data Engineering OpportunityData Idols is seeking a skilled Senior Data Platform Engineer to join our team. As a key member of our engineering team, you will be responsible for designing, implementing, and maintaining a robust data platform using Azure Services.This is an exciting opportunity to work with cutting-edge cloud-based technologies and collaborate...

  • Central London Data Lake Manager

    5 days ago

    London, Greater London, United Kingdom Station Full time

    Job Title: Central London Data Lake ManagerOur client is seeking an experienced Central London Data Lake Manager to lead the development of their centralised data storage solution. As part of this exciting project, you will be responsible for designing, implementing, and managing the Data Lake, ensuring it meets the organisation's needs and adheres to...

  • Senior Data Infrastructure Engineer

    1 week ago

    London, Greater London, United Kingdom Data Idols Full time

    About the RoleData Idols is a leading organisation in the field of data-driven decision-making. We are currently seeking a Senior Data Platform Engineer to join our team and contribute to shaping the data landscape. In this role, you will work alongside a talented, collaborative team to build, optimise, and secure scalable data infrastructure.Key...

  • Data Engineer for Digital Transformation

    1 month ago

    London, Greater London, United Kingdom Data Idols Full time

    Senior Data Platform EngineerWe are seeking a highly skilled Data Engineer to join our team at Data Idols. As a key member of our data platforms group, you will play a vital role in shaping the data landscape and contribute to data-driven decision-making.The OpportunityIn this exciting role, you will be responsible for designing, implementing, and...

  • Data Engineer Leader

    4 weeks ago

    London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    Job OverviewData Idols is a pioneering retailer transforming into a data-driven organisation. As a Senior Data Platform Engineer, you will shape the data landscape by designing, implementing, and maintaining a robust data platform using Azure Services.We offer a competitive salary of £80,000 - £90,000 + bonus, along with 20% discount across all products,...

  • Cloud Data Engineering Lead

    2 weeks ago

    London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    About the RoleWe are currently seeking a Senior Data Platform Engineer to join our team at Data Idols, a renowned retailer in the digital-first era. This is an exciting opportunity to contribute to the design, implementation, and maintenance of a robust data platform essential to our ongoing digital transformation.Your Key Responsibilities:Design and...

  • Cloud Data Architect

    1 week ago

    London, Greater London, United Kingdom Data Idols Full time

    Job OverviewData Idols is seeking a skilled Senior Data Platform Engineer to join its team. As a key member of the data platform engineering group, you will play a vital role in designing and implementing scalable, reliable, and secure data infrastructure using Azure Services.Responsibilities- Designing and implementing scalable, reliable, and secure data...

  • Big Data Infrastructure Specialist

    6 days ago

    London, Greater London, United Kingdom Data Idols Full time £80,000 - £90,000

    The Perfect CandidateAs a Senior Data Platform Engineer, you'll have:Expertise in Azure Data Lake Storage, Azure Compute, and Azure Databricks.A strong understanding of DevOps practices, using tools like Azure DevOps, Jenkins, or GitHub Actions.Solid knowledge of security best practices on Azure, including identity and access management, network security,...

  • Data Engineer

    2 days ago

    London, Greater London, United Kingdom Data Science Festival Full time

    **Job Overview**We are seeking a highly skilled Data Platforms Senior Engineer to join our team at Data Science Festival. In this pivotal role, you will play a crucial part in shaping the data landscape and contributing directly to data-driven decision-making.As a senior engineer, you will be responsible for designing, implementing, and maintaining robust...

  • Senior Python Developer

    2 weeks ago

    London, Greater London, United Kingdom Harnham Full time

    About the Role:Harnham, a leading UK-based software company, seeks a skilled Cloud Data Engineer to join their product engineering organisation. As a key member of the team, you will be responsible for designing and implementing efficient data pipelines using Python, collaborating closely with the data science team to create a scalable data lake using...

  • Senior Engineer, Data Engineering

    4 weeks ago

    London, United Kingdom Stott and May Full time €50,000 - €90,000

    Data Engineer London: (Hybrid - 50% onsite) Salary: £90k + Bonus Are you an experienced Data Engineer passionate about cutting-edge technologies? Join us to design, build, and manage a next-generation Azure Intelligent Data Platform , working with services like Data Lake, Data Factory, Synapse Analytics, Databricks, and AI Studio . Lead and...