Note: The job is a remote job and is open to candidates in USA. Pacific Northwest National Laboratory (PNNL) is a world-class research institution focused on scientific research and innovation. They are seeking a Senior Azure Databricks Engineer to lead the design, build, and operation of a data lakehouse that powers analytics and reporting across PNNL Enterprise Systems, ensuring data governance and enabling self-service analytics.
Responsibilities
- Lead the design, build, and operation of our data lakehouse that powers analytics and reporting across PNNL Enterprise Systems
- Deliver governed, performant, and reliable data products—especially for ERP and other enterprise—and enable self-service analytics with Power BI and AI/ML
- Design and evolve a Databricks‑based architecture that moves data with confidence from source systems to curated gold tables
- Build robust pipelines that transform raw data into analytics‑ready assets for Power BI and AI/ML, balancing pragmatic MVP delivery with a future‑focused architecture
- Lead modernization from legacy data warehouses and ETL tools into Azure Databricks—refactoring brittle jobs into scalable patterns
- Shape CI/CD for Databricks (e.g., DAB, Azure DevOps, GitHub Actions) and standardize deployment practices across environments
- Configure and operate workspaces, clusters, jobs, and workflows; tune for performance and reliability; and embed data quality, monitoring, and observability to keep critical pipelines healthy
- Implement role‑based access controls, data masking, and fine‑grained models with Unity Catalog to enable secure, compliant data sharing and ensure proper classification, lineage, and auditability
- Guide engineers and analysts in adopting lakehouse best practices and modern data engineering standards—coding, testing, version control, and documentation
- Stay current with Azure and Databricks capabilities, recommending and piloting features like Delta Live Tables and Unity Catalog enhancements to build a secure, reliable, and future‑ready data platform
Skills
- PhD and 3 years of Software Engineering experience -OR-
- MS/MA and 5 years of Software Engineering experience -OR-
- BS/BA and 7 years of Software Engineering experience -OR-
- AA and 16 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development -OR-
- HS/GED and 18 years of Software Engineering experience in designing, architecting, programming, deploying, and automating software solutions in support of scientific research or consumer digital product development
- 7+ years of professional data engineering or platform engineering experience, with 3–5+ years focused on cloud data platforms
- 5+ years of experience operating production Azure Databricks, including Delta Lake, SQL, notebooks, Jobs/Workflows, and cluster management
- Production experience (3-5+ years) designing and operating ingestion-to-gold pipelines (medallion architecture) for ERP or other complex transactional sources
- Experience with Azure Data Factory and/or Fabric Data Pipelines for orchestration and integration
- Familiarity with core Azure services: ADLS Gen2, Key Vault, Azure DevOps or GitHub
- Strong proficiency in Python and SQL in a Spark/Databricks environment
- Experience implementing Databricks Asset Bundles (DAB) or equivalent for CI/CD and standardizing deployment workflows
- Experience using GenAI / LLM-based tools (e.g., GitHub Copilot, Azure OpenAI, Databricks Genie, or similar) to accelerate and automate engineering tasks such as code generation, test creation, documentation, and troubleshooting
- Exposure to agentic AI / AI agents (e.g., orchestrating multi-step AI workflows for data quality checks, pipeline monitoring, or support automation) is a plus
Benefits
- Medical insurance
- Dental insurance
- Vision insurance
- Robust telehealth care options
- Several mental health benefits
- Free wellness coaching
- Health savings account
- Flexible spending accounts
- Basic life insurance
- Disability insurance 6
- Employee assistance program
- Business travel insurance
- Tuition assistance
- Relocation
- Backup childcare
- Legal benefits
- Supplemental parental bonding leave
- Surrogacy and adoption assistance
- Fertility support
- Company-funded pension plan 6
- 401 (k) savings plan with company match 6
- Up to 120 vacation hours per year
- Ten paid holidays per year
Company Overview
Pacific Northwest National Laboratory operates as a government research laboratory. It was founded in 1965, and is headquartered in Richland, Washington, USA, with a workforce of 5001-10000 employees. Its website is http://www.pnnl.gov/.