Senior Data Engineer – Databricks & Microsoft Fabric

Senior Data Engineer - Databricks & Microsoft Fabric

Work Location: Toronto, Canada

Job Summary

We are seeking a Senior Data Engineer with strong expertise in Azure Databricks and modern cloud data platforms to design, develop, and optimize scalable data pipelines, lakehouse solutions, and analytics platforms. The role focuses primarily on Databricks data engineering, with additional experience in Microsoft Fabric and Databricks Genie to support self-service analytics, conversational AI, and enterprise reporting.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Implement Medallion Architecture (Bronze, Silver, Gold) and integrate data from APIs, databases, files, ERP, CRM, and cloud storage.
• Optimize Spark jobs, cluster configurations, and data processing workloads for performance and cost efficiency.
• Develop reusable frameworks for data ingestion, transformation, and quality validation.
• Implement CI/CD pipelines using Azure DevOps or Git-based deployment practices.
• Monitor and troubleshoot production data pipelines.
• Develop data solutions using Microsoft Fabric components, including Data Factory, Lakehouse, Warehouse, Notebooks, and Data Pipelines.
• Support data modernization initiatives, Power BI semantic models, and enterprise reporting solutions.
• Support Databricks Genie implementation, semantic layer development, and natural language querying for self-service analytics.
• Apply data governance, security, and access controls across data platforms.
• Collaborate with business analysts, architects, and stakeholders to translate business requirements into technical solutions.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field.
• 7+ years of experience in data engineering.
• 4+ years of hands-on experience with Azure Databricks.
• Strong proficiency in PySpark, Spark SQL, Python, SQL, and Delta Lake.
• Experience with Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), and Git/Azure DevOps.
• Strong understanding of ETL/ELT development, data modeling, data warehousing, and performance optimization.
• Experience designing and delivering enterprise cloud data platforms.
• Experience developing scalable data pipelines and implementing data quality practices.

Nice to Have

• Experience with Microsoft Fabric, including Lakehouse, Warehouse, Data Factory, Notebooks, and Data Pipelines.
• Familiarity with Power BI and semantic model development.
• Experience with Databricks Genie and AI-powered conversational analytics.
• Knowledge of Unity Catalog, Databricks Workflows, REST APIs, and data governance.
• Experience with AI/GenAI-enabled analytics solutions.
• Databricks Associate or Professional certification.