Development Lead - Advanced RPA

Job Summary

Job Description : Data Engineer

Key Responsibilities

Must-Have Skills

Databricks: Expert Databricks Engineer, Delta Lake, DLT / Lakeflow, Streaming Pipelines, Workflow Orchestration, PySpark

Advanced skills: Structured Streaming, Window Functions, Partitioning, Broadcast joins, Adaptive Query Execution, Performance Tuning

SQL Expert level: Plain Text, Advanced Joins, CTEs, Window Functions, MERGE, Subqueries, Recursive CTEs, Query Optimization, Execution Plans

AWS: S3, Glue, IAM, Lambda, CloudWatch, Kinesis

Data Quality: DQX, Great Expectations, Delta Expectations, Reconciliation Frameworks

AI Data Engineering: Chunking methods, Document ingestion, Embedding pipelines, Vector Search implementation, Metadata enrichment

CICD: Git, Azure DevOps / GitHub Actions, Terraform, Databricks Asset Bundles

Required Skills & Qualifications:

• Bachelor’s or Master’s degree in computer science, Information Systems, or a related field.
• 7-10 years of experience in data governance, data management, or a related field.
• Proven experience in data engineering and data governance roles.
• Proficiency in SQL, Python, and ETL tools. Experience with cloud data platforms. Familiarity with data governance tools.
• Strong understanding of Databricks, data modeling, data warehousing, and data architecture.
• Experience with data quality management, metadata management, and data lineage practices.
• Excellent analytical, problem-solving, and decision-making skills.
• Strong communication and interpersonal skills, with the ability to work effectively with cross-functional teams.
• Ability to manage multiple projects and priorities in a fast-paced environment.

Skill Requirements

Must-Have Skills

Databricks: Expert Databricks Engineer, Delta Lake, DLT / Lakeflow, Streaming Pipelines, Workflow Orchestration, PySpark

Advanced skills: Structured Streaming, Window Functions, Partitioning, Broadcast joins, Adaptive Query Execution, Performance Tuning

SQL Expert level: Plain Text, Advanced Joins, CTEs, Window Functions, MERGE, Subqueries, Recursive CTEs, Query Optimization, Execution Plans

AWS: S3, Glue, IAM, Lambda, CloudWatch, Kinesis

Data Quality: DQX, Great Expectations, Delta Expectations, Reconciliation Frameworks

AI Data Engineering: Chunking methods, Document ingestion, Embedding pipelines, Vector Search implementation, Metadata enrichment

CICD: Git, Azure DevOps / GitHub Actions, Terraform, Databricks Asset Bundles

Required Skills & Qualifications:

• Bachelor’s or Master’s degree in computer science, Information Systems, or a related field.
• 7-10 years of experience in data governance, data management, or a related field.
• Proven experience in data engineering and data governance roles.
• Proficiency in SQL, Python, and ETL tools. Experience with cloud data platforms. Familiarity with data governance tools.
• Strong understanding of Databricks, data modeling, data warehousing, and data architecture.
• Experience with data quality management, metadata management, and data lineage practices.
• Excellent analytical, problem-solving, and decision-making skills.
• Strong communication and interpersonal skills, with the ability to work effectively with cross-functional teams.
• Ability to manage multiple projects and priorities in a fast-paced environment.

Other Requirements