Specialized Recruiting Group - Charlotte (South), NC

Azure Databricks Engineer

This position sits within our Oklahoma City-based corporate office, currently with an expectation to be 100% of the time.

Seeking a highly skilled Databricks Engineer with advanced expertise in Databricks CI/CD pipelines and streaming table development. The ideal candidate will have hands-on experience implementing robust CI/CD processes within the Databricks environment, enabling efficient code deployment, automated testing, and streamlined release management.

Key responsibilities include designing and optimizing streaming tables to support real-time analytics, ensuring seamless schema evolution to accommodate changing business requirements. Candidates should demonstrate proficiency in schema optimization techniques to maximize performance and reliability, as well as a deep understanding of data engineering best practices on the Databricks platform.

RESPONSIBILITIES

• Develop, maintain, and optimize Databricks CI/CD pipelines for automated deployment and testing
• Evaluate, understand and assist in planning for compute and job cluster resources in Azure Databricks environments
• Design and implement streaming tables with support for schema evolution and optimization
• Collaborate with data architects and business stakeholders to understand data requirements and deliver scalable solutions
• Troubleshoot and resolve issues related to table schemas, data ingestion, and streaming performance
• Ensure data quality, security, and compliance throughout the engineering process
• Databricks environmental change tracking

EDUCATION & EXPERIENCE QUALIFICATIONS

• Bachelor’s degree in Computer Science, Engineering, or equivalent experience in related field
• 5+ years of experience in data engineering, with a focus on cloud data solutions
• 3+ years of databricks experience in an enterprise environment

KNOWLEDGE, SKILLS, OR ABILITIES

• Proficiency in Azure cloud computing and data architecture design
• Implementation of a data lake solution using Azure Databricks, integrating various data sources, and enabling analytics for business intelligence
• Proven experience with Databricks, including CI/CD pipeline management and streaming table development
• Expertise in schema evolution and optimization within streaming data architectures
• Strong programming skills in Python and SQL
• Familiarity with YAML a plus
• Proficient with GIT
• Excellent problem-solving and communication abilities

Job Types: Full-time, Permanent

Pay: $150,000.00 - $190,000.00 per year

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Health insurance
• Life insurance
• Paid time off
• Vision insurance

Application Question(s):

• YES or NO: Do you require employment sponsorship? If yes, please detail your visa status and when that visa is valid through:

Education:

• Bachelor's (Required)

Experience:

• Azure Databricks: 3 years (Required)
• Data Engineering: 3 years (Required)
• Cloud Data Solutions: 3 years (Required)

Ability to Commute:

• Oklahoma City, OK 73162 (Required)

Ability to Relocate:

• Oklahoma City, OK 73162: Relocate before starting work (Required)

Work Location: In person