Forum Discussion

PDG_VL's avatar
PDG_VL
Icon for Helper I rankHelper I
1 year ago
Solved

Incremental Update in Fabric from External Delta Tables

Hi everyone,   I'm working with Dynamics 365 Finance & Operations and using Synapse Link to export data to an ADLS data lake in Delta Lake format. In Microsoft Fabric, I use shortcuts to reference ...
Need Help
  • Akash_Varuna's avatar
    Akash_Varuna
    1 year ago

    Hi PDG_VL To implement incremental updates from external Delta tables into a Fabric Lakehouse Bronze layer, use a timestamp column to filter new or modified rows since the last refresh. Perform upserts using Delta Lake’s MERGE functionality to handle updates and deletes efficiently. Maintain a change log table for historical tracking, appending only new records during each refresh. Optimize by partitioning Delta tables and processing only relevant partitions to reduce resource usage.

Akash_Varuna's avatar
Akash_Varuna
Icon for Super User rankSuper User
1 year ago

Hi PDG_VL To implement incremental updates from external Delta tables into a Fabric Lakehouse Bronze layer, use a timestamp column to filter new or modified rows since the last refresh. Perform upserts using Delta Lake’s MERGE functionality to handle updates and deletes efficiently. Maintain a change log table for historical tracking, appending only new records during each refresh. Optimize by partitioning Delta tables and processing only relevant partitions to reduce resource usage.