Senior Technical Architect

Job Summary

We are seeking a highly skilled and hands-on Senior DataOps Engineer with 8+ years of experience in designing, developing, automating, and governing enterprise-scale data platforms. The ideal candidate should possess strong expertise in Data Engineering, DevOps, Cloud Platforms, CI/CD, Infrastructure as Code, Data Governance, Data Quality, MLOps, and Generative AI integration.

This role requires deep technical expertise along with the ability to collaborate with Data Engineers, Architects, Data Scientists, Platform Teams, and Business Stakeholders.

Key Responsibilities

DataOps Platform Development

• Design and implement enterprise DataOps frameworks and operating models.
• Build automated and reusable data pipeline deployment frameworks.
• Develop continuous integration and continuous delivery (CI/CD) pipelines for data products.
• Implement metadata-driven ingestion and transformation frameworks.
• Automate ETL/ELT deployment processes.
• Establish DevSecOps practices within data engineering environments.
• Build self-service data platform capabilities.

• Design and develop batch and real-time data ingestion pipelines.
• Build data pipelines using Spark, PySpark, Python, SQL.
• Implement CDC (Change Data Capture) solutions.
• Develop streaming architecture using Kafka and Kinesis.
• Optimize large-scale data processing workloads.
• Automate schema evolution and data validation processes.
• Implement data lake and lakehouse architectures.

Skill Requirements

Operational Skills

• Incident Management, Problem Management, Change Management
• Root Cause Analysis
• Service Reliability Engineering (SRE)
• Operational Reporting and Governance, Stakeholder Communication

DevOps & Automation

• Build enterprise CI/CD pipelines using: Azure DevOps, Jenkins, GitHub Actions, GitLab CI/CD
• Infrastructure Automation: Terraform, CloudFormation, ARM Templates, Bicep
• Containerization: Docker, Kubernetes, Amazon EKS, AKS
• Source Control: Git, GitHub, GitLab, Bitbucket

Programming Languages: Python (Expert), Pyspark, SQL (Expert), Shell Scripting, Scala (Preferred)

Cloud Platform Engineering

AWS

Hands-on experience in: [S3, Glue, EMR, Athena, Lambda, Redshift, Lake Formation, Kinesis, MWAA (Airflow), ECS/EKS, CloudWatch, IAM, Secrets Manager, Step Functions

Azure [Azure Data Factory, Azure Synapse, ADLS Gen2, Databricks, Azure DevOps, Azure Functions, Event Hub, Purview]

GCP (Preferred)- [BigQuery, Dataflow, Dataproc, Composer, Cloud Storage]

Other Requirements

Education

Bachelor’s or master’s degree in: Computer Science/Information Technology/Data Science