Job Description: DataBricks - Lead Programmer Analyst
We are looking for a highly self-motivated individual with DataBricks development as a Lead Programmer Analyst:
Required Skills & Qualifications:
- Experience should have 5 to 7 Years of Data Engineering.
- Expert-level Apache Spark skills using PySpark (Scala a plus).
- Strong proficiency in SQL for data transformation and performance tuning.
- Solid experience with Delta Lake and the Lakehouse/medallion architecture.
- Hands-on experience with at least one major cloud platform (AWS, Azure, or GCP) and its data services.
- Experience with workflow orchestration (Databricks Workflows, Delta Live Tables, or Airflow).
- Strong understanding of data modeling, data warehousing, and ETL/ELT design patterns.
- Experience with Python for data engineering and automation.
- Familiarity with CI/CD, Git, and DevOps practices for data.
- Good understanding of data governance, security, and Unity Catalog.
- Strong problem-solving and communication skills.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Good To Have:
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with streaming technologies (Structured Streaming, Kafka, Event Hubs, Pub/Sub).
- Exposure to machine learning workflows and MLflow.
- Experience with Infrastructure-as-Code (Terraform) and cloud cost optimization.
- Experience working in Agile delivery environments.
- Experience in data warehouse design and maintenance
- Experience in agile development processes using Jira and Confluence.
- Experience in cross-functional teams.
Key Responsibilities:
- Design, develop, and maintain scalable ETL/ELT data pipelines on the Databricks Lakehouse Platform.
- Build and optimize Apache Spark (PySpark/Scala) jobs for batch and streaming data processing.
- Implement and manage Delta Lake tables, including partitioning, schema evolution, and performance tuning (Z-ordering, caching, file compaction).
- Develop and orchestrate workflows using Databricks Workflows, Delta Live Tables (DLT), and job scheduling tools such as Airflow.
- Implement the medallion architecture (bronze, silver, gold layers) for reliable, governed data.
- Integrate Databricks with cloud data services (AWS, Azure, or GCP) and source systems.
- Apply data governance and security using Unity Catalog, access controls, and lineage.
- Optimize cluster configuration, cost, and performance across workloads.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver trusted datasets.
- Ensure data quality, testing, monitoring, and observability across pipelines.
- Contribute to CI/CD practices for data engineering (version control, automated deployment).