Forum Discussion
Incremental loading from multiple F&O tables using Fabric Link
- 5 months ago
Hi diegofosso ,
When working with Fabric Link for F&O, there are a few important aspects to consider when designing incremental loads.
Each table includes the SinkModifiedOn field, which should be used as the basis for incremental logic. However, it is also important to take into account the IsDelete field. Fabric Link does not physically delete records; instead, deletions are tracked logically using this flag, so your process must handle inserts, updates, and deletes accordingly.
From an architectural perspective, a good approach is to treat the data coming from Fabric Link as a landing layer (Bronze). Fabric Link already delivers data incrementally and in Delta format, so it is better to avoid embedding complex transformation logic at this stage.
From there, the recommended pattern would be:
- Keep the landing layer as-is from Fabric Link
- Apply incremental loading per table using SinkModifiedOn
- Handle deletions using IsDelete
- Consolidate data into a Silver layer using MERGE/UPSERT
- Perform joins and modeling in a Gold layer
This aligns naturally with a medallion architecture, which works especially well in this type of scenario.
Additionally, you can leverage Delta Change Data Feed (CDF) to process row-level changes between table versions, which allows you to build a more robust and decoupled incremental process.
It is also worth considering the use of materialized views(now in GA) to simplify consumption and expose curated datasets.
In summary, rather than trying to solve incremental loading with a single query across multiple tables, the key is to:
- Think in terms of incremental per table
- Use both SinkModifiedOn and IsDelete correctly
- Separate ingestion, transformation, and consumption layers
- Leverage native Fabric capabilities such as Delta and Change Data Feed
This approach is much more robust and aligned with how Fabric Link actually works.
If my comment helped solve your question, it would be great if you could mark it as the accepted solution. It helps others with the same issue and it also motivates me to keep contributing.
Thanks a lot. I really appreciate it
Hi diegofosso ,
May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.
Thank you.