Data Engineer – Delta Lake & Microsoft Fabric

About the role

We are looking for a skilled Data Engineer to design, build, and optimise data pipelines using Delta Lake and Microsoft Fabric. The role can be performed remotely for candidates living near Brussels and offers both freelance and permanent contract options.

Key responsibilities

• Implement Delta Lake storage layers (Bronze/Silver/Gold) with proper partitioning and Z‑Ordering.
• Develop robust ETL/ELT workflows using Fabric Data Factory, Spark, and custom code to ingest data into OneLake.
• Write high‑performance data processing modules or connectors in Java and/or Python.
• Integrate Microsoft Fabric with external platforms (AWS, GCP, on‑prem) via shortcuts and APIs.
• Create complex data transformations in Spark notebooks, including cleansing and business logic.
• Conduct unit testing, data profiling, and performance benchmarking to ensure pipeline stability.

Required profile

• Expert proficiency in SQL and Python (PySpark); solid Java knowledge for backend processing.
• Strong understanding of software engineering principles applied to data (DRY, modularity, error handling, logging).
• Experience with structured, semi‑structured (JSON/XML) and unstructured data formats.
• Practical experience with Git, branching strategies, and CI/CD using Azure DevOps or GitHub Actions.
• Deep knowledge of Apache Spark internals, memory management, and execution plans.
• Good written and spoken English; French is a plus.
• Required certifications: Microsoft DP‑700 (Fabric Data Engineer Associate). DP‑203 or equivalent Databricks/AWS certifications are a plus.

Required skills

• SQL
• Python / PySpark
• Java
• Delta Lake
• Microsoft Fabric
• Azure Data Factory (Fabric)
• Apache Spark
• OneLake
• REST APIs / Shortcuts
• Git
• Azure DevOps
• GitHub Actions