Engenheiro de Dados Senior

AWS Python
R$ 13.000,00 - R$ 15.000,00
location_on São Paulo - SP date_range05/11/2019

Descrição da vaga

Our product vision relies primarily on reliable and effective data ingestion, processing and analysis. We are first and foremost a data company and to continue successfully on this path we need the best data engineers the market has to offer. We are looking for a technical lead in data engineering, a person who can lead the work and pair up directly with our CTO to architect, engineer and improve our data infrastructure.


Responsibilities


  • Definition of data pipeline architecture and construction, from data ingestion, storage in AWS, processing and visualization 

  • Construction of data pipelines with continuous delivery tooling to rapidly manipulate data at high-scale

  • Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

  • Create processes supporting data transformation, data structures, metadata, dependency and workload management.

  • Ensure pipelines are traceable, maintainable, monitored, and resilient.


Required technologies/background


There are no ‘years of experience’ or specific experiences that are absolutely required, but professional familiarity and domain expertise with the following technologies are important differentiators for us: 


  • Python

  • AWS (Kinesis, Lambda, RDS) 

  • Open-source data processing tools, such as Apache Spark, Airflow, Flink, or comparable frameworks.

Habilidades necessárias

AWS Python

Habilidades desejáveis

Apache AWS Redshift

A vaga aceita trabalho remoto?

Não

A empresa