From SAP to Microsoft Fabric via Medallion Lakehouse Pipeline

This article shows how to configure a Microsoft Fabric pipeline for incremental data ingestion, transformation, and reporting using a Medallion design pattern. The use case presented in this article processes near-realtime SAP data changes through Bronze, Silver, and Gold layers to create Power BI reports. The solution is configuration-driven, enabling new SAP tables to be onboarded without modifying pipeline logic.

About this Use Case

runs when Xtract Universal's Table CDC extraction type writes a new Parquet file to Microsoft Fabric OneLake.

The trigger listens directly on the OneLake path for file creation events.

• A Storage Event Trigger fires automatically when a new Parquet file is written to OneLake.
• Filter by .parquet suffix to avoid false fires.
• Pass @triggerBody().folderPath into the pipeline as a parameter to identify which table to process. The pipeline executes immediately on each new CDC delivery — no polling, no batch window, no manual intervention. Tables are updated within seconds of the file landing in OneLake.

The design follows a configuration-driven approach where participating SAP tables and their business keys are defined externally in a JSON configuration file rather than hardcoded in pipeline activities.

• Table CDC
• Microsoft Fabric OneLake + Pipeline / Copy Data Activity Job

The depicted approach replicates SAP data into Microsoft Fabric using four logical layers:

A medallion lakehouse is a data design pattern used to organize and process data within a data platform. It consists of four layers:

flowchart LR

    A[SAP Source]
    A --> B[Landing Zone]
    B --> C[Orchestrator + Config]
    C --> D[Child Pipeline]
    D --> E[Bronze]
    E --> F[Silver]
    F --> G[Gold]
    G --> H[Power BI Report]

Setup in Xtract Universal

• One folder per SAP table (kna1, mara, vbak, vbap)
• Each run drops a new Parquet file —never overwrites
• Filenames carry a timestamp so files sort chronologically
• Each file = an incremental batch: inserts, updates, deletes

Files/
└── incremental/
    ├── kna1/
    │   ├── kna1_20260620_0800.parquet
    │   ├── kna1_20260621_0800.parquet
    │   └── kna1_20260622_0800.parquet  ← newest
    ├── mara/
    │   ├── mara_20260620_0800.parquet
    │   └── mara_20260621_0800.parquet
    ├── vbak/
    │   └── vbak_20260622_0600.parquet
    └── vbap/
        └── vbap_20260622_0600.parquet

Pipeline Setup

Config

Stored under Files/config, this file defines —per SAP table —the landing folder, the Bronze table name, and the business key columns used for upsert/merge.

[
{ "FolderName": "mara", "TableName": "mara",
"KeyColumns": ["MANDT", "MATNR"] },
{ "FolderName": "kna1", "TableName": "kna1",
"KeyColumns": ["MANDT", "KUNNR"] },
{ "FolderName": "vbak", "TableName": "vbak",
"KeyColumns": ["MANDT", "VBELN"] },
{ "FolderName": "vbap", "TableName": "vbap",
"KeyColumns": ["MANDT", "VBELN", "POSNR"] }
]

When adding a new SAP table to the pipeline, add one JSON entry to pipeline-config.json. Changing the pipeline logic is not necessary.

Orchestrator

1. Load pipeline-config.json as a JSON array
2. Iterate over each table definition
3. Run the child pipeline for each table/folder
4. Build Silver and Gold once Bronze refreshes

flowchart LR

    A[Read Config File]
    A --> B[Loop through folders]
    B --> C[Invoke child pipeline]
    C --> D[Transform Sales Analytics]

Child Pipeline

The child pipeline runs once per table/folder. It loads new Parquet files into the Bronze delta table.

1. Get

Related Links

• Microsoft Documentation: Medallion Lakehouse Architecture