Forum Discussion

jaryszek's avatar
jaryszek
Super User
5 months ago
Solved

DirectLAke over OneLake how to use incremental load?

Hello,

 

How to use parameters/incremental load in OneLake ?

 

Best,
Jacek

Need Help
  • v-dineshya's avatar
    v-dineshya
    5 months ago

    Hi jaryszek ,

    Thank you for reaching out to the Microsoft Community Forum.

     

    Hi anilelmastasi , Thank you for your prompt response. 

     

    Hi jaryszek ,


    What Is the Correct Refresh Order?

    1. Raw Bronze --> append new files only
    2. Silver --> upsert changes (MERGE)
    3. Gold --> aggregations, denormalization

     

    How Consistency Is Maintained in Delta?

    Delta Lake ensures:

    1. ACID Transactions: Every write creates, _delta_log/NNNN.json, snapshot, commit metadata and optional checkpoint.

    2. Snapshot Isolation: Readers always see the latest completed version.

    3. No Locking Problems: Writers do not block readers.

     

    I hope this information helps. Please do let us know if you have any further queries.

     

    Regards,

    Dinesh

     

     

4 Replies

  • anilelmastasi's avatar
    anilelmastasi
    Super User
    5 months ago

    Hello jaryszek ,

     

    For Incremental Load, you need Delta tables in OneLake. As you mention incremental load, it means incrementally updating those Delta tables. So basically you need Delta tables in lakehouse or warehouse.

    To create incremental load, you can use dataflows gen2, Notebooks. In dataflows gen2 use incremental refresh, load only new/changed rows (upsert) rows into the table. Also there is an important step, you should partition yur delta table. it improves performance. 

     

    If this solved your issue, please mark it as the accepted solution. āœ…

     

    • jaryszek's avatar
      jaryszek
      Super User
      5 months ago

      Thanks, 

      how this is maitaining consistency? 
      What is the order of refreshing delta tables? 

      Best,
      Jacek

      • v-dineshya's avatar
        v-dineshya
        Community Support
        5 months ago

        Hi jaryszek ,

        Thank you for reaching out to the Microsoft Community Forum.

         

        Hi anilelmastasi , Thank you for your prompt response. 

         

        Hi jaryszek ,


        What Is the Correct Refresh Order?

        1. Raw Bronze --> append new files only
        2. Silver --> upsert changes (MERGE)
        3. Gold --> aggregations, denormalization

         

        How Consistency Is Maintained in Delta?

        Delta Lake ensures:

        1. ACID Transactions: Every write creates, _delta_log/NNNN.json, snapshot, commit metadata and optional checkpoint.

        2. Snapshot Isolation: Readers always see the latest completed version.

        3. No Locking Problems: Writers do not block readers.

         

        I hope this information helps. Please do let us know if you have any further queries.

         

        Regards,

        Dinesh