Senior Technical Lead
On-site
Dubai, United Arab Emirates
Job Summary
A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data. Core Responsibilities- Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses.
- Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data.
- Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling.
- Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC).
- Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational).
- CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing.
Key Responsibilities
as above
Skill Requirements
as above
Other Requirements
Source: the employer's careers page. Last checked 2026-10-24. Posted 2026-10-22.
More jobs at HCLTech · Jobs in Dubai · On-site jobs in United Arab Emirates