Narala, LLC

Overview

***Please Note: This is a part-time or full-time consulting opportunity on a contract basis. Contract-to-hire option available for the right candidate. We are open to considering candidates who are working full-time but can commit to 4 hours/ day for part-time consulting.

Candidates must be reside in the Eastern Time Zone as working hours for this role will be 7am-4pm EST.

Only US citizens and permanent residents (green card holders) are eligible to apply for this position. No visa sponsorships available.****

The Azure Databricks Architect is a key position responsible for the setup and ongoing management of the Databricks infrastructure on Azure for clients. This role provides strategic direction, operational oversight and in-depth technical expertise to the infrastructure and data engineers for both internal staff and client organizations.

Key Responsibilities

Azure Infrastructure

· Configure Virtual Networks (VNets) with appropriate subnetting

· Establish Network Security Groups (NSGs) and firewall rules

· Set up Azure Private Link for secure connectivity

· Deploy Azure Data Lake Storage Gen2 (ADLS Gen2) with lifecycle policies

· Configure Azure Key Vault for secrets management

· Implement Azure Monitor and Log Analytics workspace

· Establish Azure AD/Entra ID integration for SSO

· Deploy Azure Virtual Machine or Container Instances for Apache Airflow

· Configure SFTP server (Azure VM or Azure Storage SFTP) for data ingestion

Databricks Platform

· Provision Databricks workspaces for Dev, Staging, and Production environments

· Enable Unity Catalog for centralized data governance

· Configure workspace-level security and network isolation

· Set up cluster policies and job compute configurations

· Implement Auto-Loader for streaming data ingestion

· Deploy Delta Live Tables (DLT) for declarative pipeline development

· Enable Delta Lake features: time travel, Z-Order optimization, VACUUM operations

Orchestration & Workflow Management

· Deploy Apache Airflow on Azure compute (VM or AKS)

· Configure Airflow DAGs for data pipeline orchestration

· Integrate Airflow with Databricks Workflows

· Set up job scheduling and dependency management

· Implement retry logic and error handling

· Configure Airflow UI access and authentication

Governance & Access Control

· Configure Unity Catalog metastore and catalogs

· Implement Role-Based Access Control (RBAC)

· Enable attribute-based access control (ABAC)

· Configure column masking for sensitive data (PHI)

· Implement row-level security (RLS) policies

· Establish data lineage tracking

· Enable audit logging and compliance reporting

HIPAA Compliance & Security

· Implement encryption at rest (Azure Storage Service Encryption)

· Configure encryption in transit (TLS 1.2+)

· Enable Azure AD authentication and MFA

· Implement network isolation with Private Link

· Configure audit logging for all PHI access

· Establish data retention and destruction policies

· Implement Business Associate Agreement (BAA) requirements

· Configure backup and disaster recovery for PHI data

· Enable security scanning and vulnerability assessments

Monitoring & Observability

· Centralize logging in Azure Log Analytics

· Configure real-time alerting for pipeline failures

· Create health dashboards for critical systems

· Track SLA compliance and performance metrics

· Monitor Databricks job performance and cluster health

· Implement cost monitoring and budget alerts

· Set up incident notification workflows

CI/CD & DevOps

· Set up Azure DevOps or GitHub for source control

· Implement Infrastructure as Code using Terraform

· Create automated testing framework for data pipelines

· Configure CI/CD pipelines for code deployment

· Implement blue-green or canary deployment strategies

· Establish version control for notebooks and configurations

Testing & Validation

· Deploy test SFTP connection with dummy data source

· Validate end-to-end data flow from SFTP to Gold layer

· Perform data quality validation tests

· Conduct performance and load testing

· Execute security and compliance testing

· Validate backup and recovery procedures

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field is required.
• A minimum of 5 years of management related experience is required in consulting or corporate setup; Life science or healthcare environment preferred.
• A minimum of 5 years of experience in setting up and managing Databricks infrastructure on Azure.
• A minimum of 5 years of experience managing teams, setting priorities and managing project delivery.
• Strong technical expertise in areas such as server administration and cybersecurity.
• Azure and Databricks certification strongly preferred.

Key Capabilities

• Strong problem-solving, analytical, and decision-making abilities.
• Creativity, clear thinking, and excellent communication skills.
• Ability to motivate others to solve challenging technology problems, using a combination of engineering expertise, innovation and leadership.
• Ability to quickly grasp new concepts and to formulate an action plan in a new situation.
• Ability to spearhead and oversee complex projects.
• Excellent leadership and people management skills, with the ability to build and motivate high-performing teams.
• Excellent collaboration, and stakeholder management skills.
• Excellent time management skills to prioritize tasks effectively.

Compensation: Based on experience and skill set; not a limitation for the right candidate.

Job Type: Full-time

Pay: $100.00 - $150.00 per hour

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Flexible schedule
• Health insurance
• Life insurance
• Paid time off
• Professional development assistance
• Vision insurance

Application Question(s):

• Do you have hands-on experience setting up and managing Databricks on Azure ?

Experience:

• Databricks: 2 years (Preferred)
• Azure: 5 years (Required)

Work Location: Remote