Data Lake Engineer

Found in: Talent UK C2 - 7 days ago

London, United Kingdom SDG Group Full time

Requisitos mínimos:

Estar graduado en Ingeniería Informática o de Telecomunicaciones. Tener al menos 4 años de experiencia en un rol de Big Data Engineer, implementando y productivizando soluciones de captación, ingesta y transformación de datos con herramientas de procesamiento distribuido y/o procesamiento en streaming. Experiencia en arquitectura de datos (formato, estructura, particionado, modelado, capas y ciclo de vida) en ambientes Data Lake, Data Lakehouse, Kafka Topics & Partitioning, Schema Registry, etc. Además será importante que tengas experiencia en algunas de las siguientes herramientas: Lenguajes: Python, Java o Scala. Herramientas core del entorno de procesamiento en entornos Big Data, especialmente Apache Spark. Además es importante que tengas conocimientos sólidos de HDFS. Conocimiento de al menos una plataforma de datos con profundidad: Databricks, AWS EMR, AWS Glue, Elastic Search, GCP Dataflow & Dataproc, etc. Conocimiento en herramientas de orquestación como Apache Airflow, Azure Data Factory, Databricks workflows,... Conocimiento y experiencia en el uso de sistemas de control de versiones (VCS), esencialmente Git. Experiencia en la utilización de al menos una de las tres grandes nubes (AWS, Microsoft Azure o Google Cloud Platform), especialmente en lo relativo a los servicios orientados a procesamiento y disposición de datos.

Requisitos valorables:

Nos gusta estar al día en nuevas tecnologías, así que necesitamos personas que quieran estar en continua evolución y que tengan ganas de aprender nuevas tecnologías y probar nuevas herramientas. Para nosotrxs el trabajo en equipo es primordial, por lo que buscamos personas que impulsen ese buen ambiente y las sinergias entre compañerxs y diferentes equipos de trabajo. Qué conozcas lenguajes que están tomando relevancia ahora en el mundo data como Rust o Elixir es algo que valoraremos muy positivamente también. Cómo trabajamos en un entorno global, será un plus que puedas comunicarte en inglés. Conocimiento sobre DataOps, Observabilidad, patrones de diseño low-code/no-code,...

Acerca del puesto:

SDG Group es una firma de consultoría global focalizada exclusivamente en soluciones de data & analytics, cubriendo con ello toda la cadena de valor del dato, y proporcionando planes de carrera tanto de consultoría como de especialización para todos aquellos que quieran progresar en el mundo del dato. Además, a través de nuestra cultura Smart Work tendrás la posibilidad de definir el modelo de trabajo que más se adapte a ti y a tus necesidades personales, al mismo tiempo que puedas colaborar en iniciativas de gran impacto para clientes punteros en sus respectivas industrias.

Como data engineer del área de Data Technologies, tu misión será formar parte de equipos multidisciplinares con el objetivo de ayudar a nuestros clientes a abordar iniciativas de datos en entornos de alta complejidad técnica. Tendrás la posibilidad de utilizar un amplio espectro de tecnologías innovadoras para la construcción de soluciones de gestión de datos bajo diversos paradigmas de procesamiento y modelado, aplicando para ello los principios adecuados en términos de desarrollo y arquitectura.

Asimismo, gracias a nuestro path de especialista tendrás la oportunidad de crecer como ingeniero y/o arquitecto Big Data, contando con herramientas y recursos para tu continuo aprendizaje y crecimiento profesional, al mismo tiempo que te involucras en diversos retos que permitan tanto a SDG Group como a nuestros clientes explotar al máximo el valor de sus datos.

Responsabilidades:

Darás tu opinión técnica sobre los procesos y arquitecturas que se desarrollen para nutrir las conversaciones técnicas que se llevan a cabo durante las fases de diseño de la misma, priorizando estrategias dirigidas por metadatos y fundamentadas en DataOps. Participarás en la toma de requerimientos necesarios para arquitecturas y pipelines de datos. También llevarás a cabo tareas de desarrollo con diferentes tecnologías, aunque principalmente usamos Spark (tanto para procesos en batch como streaming). Adicionalmente, y aunque nuestros desarrollos habitualmente no se basen en SQL, es importante que tengas la soltura suficiente para hacer exploración de datos. Somos bastante agnósticos a la tecnología y a la infraestructura en nuestros proyectos, así que es posible que tengas experiencia proyectos en AWS, Azure, GCP u on-premises, por lo que será importante que no te importe estar cambiando de entorno con frecuencia. A veces también nos toca crear u optimizar tiempos de respuesta en motores de búsqueda, así que cualquier experiencia en Solr o Elastic nos gustará. Además, en algunos momentos también necesitaremos hacer pruebas de concepto y pruebas con tecnologías nuevas en el mercado y servicios cloud nuevos o que creamos que puedan ser útiles para nuestros proyectos o clientes, por lo que también podrás colaborar en tareas de innovación.

  • Data Software Engineer | Python

    2 weeks ago

    London, United Kingdom Abs Data Full time

    Regulatory Reporting, Big Data Software Engineer We’re looking for software engineers to: provide technology solutions that will solve business problems and strengthen our position as digital leaders in financial services work with our Regulatory Reporting team to translate business and external regulatory needs using Behavior Driven Development...

  • Data Engineer

    3 days ago

    City of London, United Kingdom N P Associates Full time

    **Data Engineer** This is an exciting opportunity for a Data Engineer to join a team of other highly skilled data professionals responsible for managing and enhancing the data estate. **Purpose of Role**: You will be part of a team responsible for data as a whole within the organisation, including data pipelines moving data from various on premise and...

  • Data Engineer

    2 weeks ago

    London, United Kingdom Harnham Full time

    **DATA ENGINEER** **SALARY: £50,000 - £70,000** **LONDON** An Extremely innovative and data-driven company is looking for a skilled Data Engineer that can help drive the business to new heights. Allowing this individual to have massive influence and ownership on how to tackle issues and grow and develop companyproducts. **THE COMPANY** This company is...

  • Senior Data Engineer

    3 days ago

    London, United Kingdom Computappoint Full time

    _Key skills: SQL, Azure Data Factory, Data Lake, Synapse, Azure DevOps YAML, BICEP, ARM, SQL Schema, Synapse Workspace, PowerShell, Azure CLI, Dynamics, Machine Learning_ A leading international law firm requires a **Senior Data Engineer** to join their Architecture, Data & Engineering team. The Senior Data Engineer will contribute to the design, delivery...

  • Data Engineer

    2 days ago

    London, United Kingdom Advantage Resourcing Full time

    **Job title: Data Engineer** **Location: London Waterloo** **Length: 29/12/2023** Working with a British multinational energy company headquartered in London we are seeking to hire a Data Engineer who will be responsible for Creating and maintaining pipelines using the Azure technology stack, Using Azure Databricks to develop data pipelinesin...

  • Senior Engineer, Data Engineering

    2 weeks ago

    London, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions. Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse. Develop robust...

  • Data Engineer

    Found in: Talent UK C2 - 5 days ago

    London, United Kingdom MS Amlin Full time

    We are looking for a highly skilled and motivated Data Engineer to join our dynamic team. As a Data Engineer you will play an important role in designing, building and maintaining the systems and architecture that enable MS AISE to collect, store and analyze large volumes of data. The ideal candidate is passionate about data, possesses strong analytical...

  • Data Engineer

    2 weeks ago

    London, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions. Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse. ...

  • Data Engineer

    2 weeks ago

    London, United Kingdom Vallum Associates Full time

    Job Description My client is a leading TechFin company that is revolutionising finance through innovative technology solutions. Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse....

  • Senior Engineer, Data Engineering

    Found in: Jooble UK C2 - 2 weeks ago

    London, United Kingdom Durlston Partners Full time

    Note: I have a plethora of live Data Engineering opportunities. A well established Quant Fund we are working with is looking to develop a new high frequency data platform to support the development and deployment of new strategies. The sort of work you would get exposure to includes: building greenfield market data processing systems that will handle very...

  • Senior Engineer, Data Engineering

    2 weeks ago

    London, United Kingdom Durlston Partners Full time

    Note: I have a plethora of live Data Engineering opportunities. A well established Quant Fund we are working with is looking to develop a new high frequency data platform to support the development and deployment of new strategies. The sort of work you would get exposure to includes: building greenfield market data processing systems that will handle very...

  • Data Engineer

    Found in: Talent UK 2A C2 - 2 weeks ago

    London, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions.Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse.Responsibilities:Develop...

  • Azure Data Engineer

    Found in: Talent UK 2A C2 - 2 weeks ago

    London, United Kingdom Venturi Full time

    Role - Azure Data EngineerRate - £500 Per Day Outside IR35Remote work with an openness to travel to London if requiredDuration - 6 monthsMy client is seeking experienced Azure Data Engineers for a stakeholder-facing project. This role will have particular emphasis on Azure, Databricks and DevOps. Our client is in the consultancy sector but does not have a...

  • Azure Data Engineer

    Found in: Jooble UK C2 - 2 weeks ago

    London, United Kingdom Venturi Full time

    Role - Azure Data Engineer Rate - £500 Per Day Outside IR35 Remote work with an openness to travel to London if required Duration - 6 months My client is seeking experienced Azure Data Engineers for a stakeholder-facing project. This role will have particular emphasis on Azure, Databricks and DevOps. Our client is in the consultancy sector but does not...

  • Azure Data Engineer

    2 weeks ago

    London, United Kingdom Venturi Full time

    Role - Azure Data Engineer Rate - £500 Per Day Outside IR35 Remote work with an openness to travel to London if required Duration - 6 months My client is seeking experienced Azure Data Engineers for a stakeholder-facing project. This role will have particular emphasis on Azure, Databricks and DevOps. Our client is in the consultancy sector but does not...

  • Data Engineer

    3 days ago

    London, United Kingdom Boston Hale Full time

    Data Engineer Our client, an exciting and ever-growing media and publishing company, are looking for a Data Engineer to join their team. The Data Engineer will be essential in growing out their data offering for an exciting stream of projects over the next few years. Data Engineer Responsibilities: - Analysis of S3 data sources and data transformations...

  • Data Engineer

    Found in: Appcast Linkedin GBL C2 - 2 weeks ago

    London Area, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions.Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse.Responsibilities:Develop...

  • Data Engineer

    Found in: Appcast UK C2 - 1 week ago

    London Area, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions.Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse.Responsibilities:Develop...

  • Data Engineer

    Found in: Appcast UK C C2 - 7 days ago

    London Area, United Kingdom Vallum Associates Full time

    My client is a leading TechFin company that is revolutionising finance through innovative technology solutions.Role Overview: They are seeking a Data engineer to join our architecture team. You'll be pivotal in designing, developing, and maintaining a cutting-edge data lake on Azure with seamless integration with Databricks Lakehouse.Responsibilities:Develop...

  • Senior Azure Data Engineer

    Found in: Jooble UK C2 - 2 weeks ago

    London, United Kingdom Venturi Full time

    Role - Azure Data Engineer Rate - £500 Per Day Outside IR35 Remote work with an openness to travel to London if required My client is seeking experienced Azure Data Engineers for a stakeholder-facing project. This role will have particular emphasis on Azure, Databricks and DevOps. Our client is in the consultancy sector but does not have a preference...