TL;DR
Senior Data Engineer (Azure): Building and optimizing scalable ETL/ELT pipelines for complex sensor data with an accent on Azure Synapse, PySpark, and real-time stream processing. Focus on standardizing multi-vendor oilfield data and ensuring high-performance data ingestion frameworks within an Azure Data Lake environment.
Location: Must be based in Brazil
Company
Encora is a global digital engineering and modernization partner, providing expertise in product development, cloud services, and data analytics for leading enterprises worldwide.
What you will do
- Architect and build scalable ETL/ELT pipelines using Azure Data Factory and Synapse.
- Convert complex time-indexed sensor data into depth-indexed formats.
- Create ingestion frameworks for oilfield file formats like LAS, DLIS, and WITSML.
- Standardize sporadic vendor data into common schemas and implement automated quality checks.
- Tune Spark jobs and SQL queries for performance and cost efficiency.
- Manage Azure Data Lake Storage (ADLS Gen2) architecture and implement Row-Level Security.
Requirements
- Location: Must be based in Brazil.
- Strong experience designing data pipelines with Azure Data Factory and Synapse Analytics.
- Advanced SQL skills and experience with large-scale data processing.
- Proficiency in Python using PySpark or Pandas.
- Experience with distributed data processing using Spark and data formats like Parquet and Delta Lake.
- Knowledge of ETL/ELT patterns and experience handling time-series or streaming data.
Nice to have
- Microsoft Certified Azure Data Engineer Associate (DP 203) certification.
Culture & Benefits
- Professional growth within a global team of over 9,000 experts.
- Exposure to diverse digital engineering projects across various domains.
- Focus on skill-based hiring and an inclusive, non-discriminatory work environment.
- Full-time engagement with remote work flexibility.