Data Engineer

## Job highlights - Experienced in Azure/AWS Databricks, PySpark, Python, SQL, and Apache Airflow - Develop, maintain, and optimize data pipelines; troubleshoot and support end-to-end data engineering workflows **Job Summary** We are looking for an experienced Data Engineer with strong hands-on expertise in **Azure/AWS Databricks, PySpark, Python, SQL, and Apache Airflow**. The candidate will be responsible for understanding existing data pipelines, analyzing business and technical logic, implementing enhancements, troubleshooting issues, and supporting end-to-end data engineering activities. **Key Responsibilities** - Understand and analyze existing data engineering pipelines, codebases, and business logic. - Develop, maintain, and enhance data pipelines using Databricks, PySpark, Python, and SQL. - Work with Apache Airflow for workflow orchestration, scheduling, and monitoring. - Perform impact analysis before implementing code changes or enhancements. - Troubleshoot pipeline failures, resolve data processing issues, and perform root cause analysis. - Understand upstream and downstream dependencies to ensure smooth data flow. - Optimize data processing jobs and SQL queries for performance and scalability. - Perform data validation and ensure data quality and consistency. - Support end-to-end pipeline development, maintenance, and production issues. - Collaborate with cross-functional teams to deliver reliable data engineering solutions. **Mandatory Skills** - Strong hands-on experience with **Databricks** on Azure or AWS. - Proficiency in **PySpark and Python**. - Strong SQL skills, including query writing and optimization. - Experience with **Apache Airflow** for pipeline orchestration. - Experience maintaining and enhancing existing data pipelines. - Ability to analyze existing code, understand dependencies, and implement changes safely. - Strong troubleshooting and problem-solving skills. **Preferred Qualifications** - Experience with cloud-based data engineering solutions on Azure or AWS. - Knowledge of ETL/ELT processes and data pipeline architecture. - Experience in production support, performance tuning, and data quality validation.