Senior Data Lead - Azure Data Factory (ADF), Databricks

Job Summary

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks.
• Build and optimize ETL/ELT processes for structured and unstructured data.
• Develop solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
• Implement data ingestion frameworks from various source systems including APIs, databases, cloud storage, and streaming platforms.
• Design and maintain Bronze, Silver, and Gold data layers following Medallion Architecture.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.
• Collaborate with business analysts, data scientists, architects, and stakeholders to gather requirements.
• Ensure data quality, governance, security, and compliance standards.
• Monitor and troubleshoot production data pipelines and workflows.
• Participate in code reviews and implement data engineering best practices.

Key Responsibilities

Required Qualifications

• 7+ years of experience in Data Engineering and Data Warehousing.
• 4+ years of hands-on experience with Databricks.
• Strong expertise in:
• PySpark
• Spark SQL
• Python
• SQL
• Delta Lake
• Experience building large-scale batch and streaming data pipelines.
• Strong understanding of data modeling and data warehousing concepts.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Knowledge of CI/CD and DevOps practices.

Skill Requirements

Technical Skills

Databricks & Big Data

• Databricks Lakehouse Platform
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog

Programming

• Python
• SQL
• Scala (Preferred)

Cloud Platforms

• Azure Databricks
• Azure Data Factory (ADF)
• AWS EMR / Glue
• Google Cloud Platform (GCP)

Data Storage

• Azure Data Lake Storage (ADLS)
• Amazon S3
• Snowflake
• Synapse Analytics
• SQL Server
• Oracle

Data Integration

• Kafka
• Event Hub
• REST APIs
• CDC Implementation

DevOps & Monitoring

• Git
• Azure DevOps
• Jenkins
• Terraform
• Monitoring and Logging Tools

Other Requirements

Preferred Qualifications

• Experience with Azure Data Factory (ADF), Synapse Analytics, or Snowflake.
• Knowledge of Data Governance and Unity Catalog.
• Experience building data lakes and lakehouse architectures.
• Exposure to machine learning workflows and MLOps.
• Databricks Certification:
• Databricks Certified Data Engineer Associate
• Databricks Certified Data Engineer Professional

Experience working in Agile/Scrum environments.