Data Engineer

 Enterprise Data Platform to enable timely, effective and safe sharing of data to multiple engineering, operations and business teams for building world class data products
 
Responsibilities

• Build data ingestion and processing pipelines to enable data analytics and data science use-cases in areas of digital commerce, service operations, charging, reliability, finance, capex, warranty, customer service and others.
• Build modular set of data services using Python, SQL, AWS Glue, lambdas, API Gateway, Kafka, data build tool (dbt), Apache Spark on EMR among others
• Build automated unit and integration testing pipelines using frameworks like PySpark
• Create and manage CICD pipelines with Gitlab CI and AWS Code Pipeline/CodeDeploy
• Automate and schedule jobs using Managed Airflow
• Build the ODS and reporting schemas and load the data into AWS Redshift or Snowflake
• Design and build data quality management services with Apache Deequ and data observability tools like Splunk, DataDog , CloudWatch
• Provide a variety of query services with REST, Athena/Presto, server sent events
• Configure and setup the enterprise data lineage and meta data management and data catalog support using tools like Collibra/Alation
• Assist the data scientist within the data engineering team as well as other software engineering teams with data cleansing, wrangling and feature engineering
• Ensure green builds for deployment and work with program management and senior leads to burn down planned deliverables in a sprint cycle

 
Qualifications

• At least 5+ years building data and analytics platforms using AWS Cloud, Python and SQL
• Knowledge of AWS technologies specifically MSK, EMR, Athena, Glue, lambdas, API Gateway as well as Python, SQL is a must
• Knowledge of modern data tools like dbt (data build tool) and Airflow orchestration is highly desired
• Ability to assist SQL analysts and Tableau developers in business teams in creating the right set of materialized views in a SQL data warehouse like Redshift/Snowflake
• Knowledge of automation and CICD best practices
• Familiarity with machine learning and data science ecosystems especially AWS Sagemaker and Databricks is highly preferred