Senior Data Lead - Azure Data Factory (ADF), Databricks

Job Summary

We are looking for a highly motivated Data Engineer to design, build, and optimize scalable data solutions that power enterprise analytics and business decision-making. The ideal candidate combines strong software engineering skills with deep expertise in modern cloud-based data platforms, automation, and data governance.

Key Responsibilities

.

Key Responsibilities

Build & Model Data Solutions

• Design, develop, and maintain scalable data pipelines and data models that support analytics, reporting, and business intelligence initiatives.
• Implement efficient data ingestion, transformation, and storage solutions for structured and unstructured data.
• Ensure data quality, consistency, and reliability across the data ecosystem.

Automate & Deploy

• Develop and maintain CI/CD pipelines to enable automated testing, deployment, and monitoring of data solutions.
• Drive Infrastructure-as-Code and deployment automation practices to ensure efficient and reliable delivery.
• Partner with development and platform teams to streamline release processes and improve operational efficiency.

Innovate & Improve

• Apply software engineering best practices to solve complex data engineering challenges.
• Introduce innovative approaches for processing, managing, and analyzing large-scale datasets.
• Continuously evaluate new technologies and frameworks to improve platform capabilities and performance.

Own the Data Platform

• Optimize data platforms for scalability, performance, security, and reliability.
• Monitor and troubleshoot data pipelines, workflows, and platform components.
• Ensure adherence to data governance, compliance, and security standards throughout the data lifecycle.

Collaborate & Document

• Establish and promote data engineering standards, best practices, and coding guidelines.
• Create and maintain clear technical documentation, architecture diagrams, and operational procedures.
• Collaborate with cross-functional teams including Data Scientists, Analysts, Product Owners, and Architects to deliver business value.

Skill Requirements

Required Skills & Experience

Technical Skills

• Strong experience with Python for data engineering and automation.
• Hands-on experience with Databricks and modern data processing frameworks.
• Expertise in Azure Cloud Services and cloud-native data architectures.
• Proficiency in Azure DevOps, Git, and CI/CD implementation.
• Experience designing and maintaining scalable data pipelines and data models.
• Strong understanding of data warehousing, ETL/ELT, and data lifecycle management.
• Knowledge of data governance, data quality, and security best practices.

Professional Skills

• Strong analytical and problem-solving abilities.
• Software engineering mindset with a focus on clean, maintainable, and reusable code.
• Excellent troubleshooting and debugging skills.
• Strong communication and documentation capabilities.
• Ability to work effectively in a collaborative, agile environment.

Preferred Qualifications

• Experience with enterprise-scale data platforms and large datasets.
• Knowledge of monitoring, observability, and performance optimization techniques.
• Familiarity with Infrastructure-as-Code and cloud automation practices.
• Experience working in Agile/Scrum delivery models.

Our Tech Stack

• Cloud: Microsoft Azure
• Data Platform: Databricks
• Programming: Python
• DevOps: Azure DevOps
• Version Control: Git
• Deployment: Fully Automated CI/CD Pipelines

Other Requirements