GCP Senior Data Lead

Job Summary

Job Description – GCP Data Engineer

Position

GCP Data Engineer – 6+ Years Experience

Role Overview

We are looking for an experienced GCP Data Engineer with 6+ years of experience in designing, developing, implementing, and supporting scalable data engineering solutions on Google Cloud Platform. The candidate should have strong hands-on experience with BigQuery, GCS, Dataflow, Cloud Composer/Airflow, Pub/Sub, Cloud Functions/Cloud Run, and Python/SQL.

The role involves developing and supporting batch and real-time data pipelines, data ingestion, transformation, monitoring, troubleshooting, and performance optimization.

Key Responsibilities

Key Responsibilities

• Design, develop, and maintain scalable data pipelines on GCP.
• Develop batch and streaming pipelines using Google Cloud Dataflow / Apache Beam.
• Develop complex SQL queries, stored procedures, views, and data transformation logic in BigQuery.
• Work with Cloud Storage (GCS) for data ingestion, staging, archival, and file-based processing.
• Develop and manage workflows using Cloud Composer / Apache Airflow.
• Work with Pub/Sub for event-driven and real-time data processing.
• Develop and support Cloud Functions and Cloud Run based integrations and services.
• Perform data ingestion from databases, APIs, files, and other source systems into GCP.
• Implement data quality, validation, reconciliation, and error-handling mechanisms.
• Monitor production pipelines and troubleshoot failures, performance issues, and data discrepancies.
• Optimize BigQuery queries, partitioning, clustering, and storage/compute usage.
• Implement appropriate logging, alerting, and operational monitoring for data pipelines.
• Participate in production deployment, incident management, root-cause analysis, and problem management.
• Collaborate with application, infrastructure, cloud, database, and business teams.
• Follow CI/CD, source-control, coding, security, and data governance standards.
• Create technical documentation, support procedures, and operational runbooks.

Skill Requirements

Required Technical Skills

Technology

Expected Experience

GCP

4+ years

BigQuery

4+ years

Dataflow / Apache Beam

3+ years

Python

4+ years

SQL

5+ years

Cloud Storage (GCS)

4+ years

Cloud Composer / Airflow

2+ years

Pub/Sub

2+ years

Cloud Functions / Cloud Run

2+ years

Git

3+ years

Linux/Unix

Good working knowledge

Preferred Skills

• Experience with Cloud Scheduler, Secret Manager, IAM and service accounts.
• Experience with Dataproc / Spark.
• Experience with Terraform or Infrastructure as Code.
• Experience with Cloud Logging and Cloud Monitoring.
• Knowledge of Docker and CI/CD pipelines.
• Experience with CDC and real-time data technologies such as Striim, Kafka, or Debezium.
• Experience working with Oracle, SQL Server, MySQL, MongoDB, or PostgreSQL.
• Knowledge of data warehouse and dimensional modeling concepts.
• Understanding of data governance, security, encryption, and PII handling.
• Healthcare data experience is a plus.

Production Support Responsibilities

The candidate should be comfortable providing L2 production support, including:

• Monitoring GCP data pipelines and applications.
• Troubleshooting failed Dataflow jobs and Airflow DAGs.
• Investigating BigQuery load and transformation failures.
• Troubleshooting Pub/Sub message processing issues.
• Analyzing application and GCP logs.
• Performing data reconciliation and identifying data gaps/duplicates.
• Managing production incidents and participating in RCA.
• Coordinating with development and cloud infrastructure teams for complex issues.
• Supporting planned deployments, maintenance activities, and DR exercises.

Required Soft Skills

• Strong analytical and problem-solving skills.
• Good communication and stakeholder-management skills.
• Ability to work independently in a production environment.
• Ability to troubleshoot issues under time constraints.
• Strong documentation and knowledge-sharing skills.
• Ability to collaborate with offshore/onshore teams.
• Good understanding of ITIL/production-support processes.

Experience

6+ years of overall experience in data engineering, with significant hands-on experience designing, developing, and supporting data solutions on Google Cloud Platform

Other Requirements