Forum Discussion

jaryszek's avatar
jaryszek
Super User
5 months ago
Solved

DirectLAke over OneLake how to use incremental load?

Hello,   How to use parameters/incremental load in OneLake ?   Best, Jacek
Need Help
  • v-dineshya's avatar
    v-dineshya
    5 months ago

    Hi jaryszek ,

    Thank you for reaching out to the Microsoft Community Forum.

     

    Hi anilelmastasi , Thank you for your prompt response. 

     

    Hi jaryszek ,


    What Is the Correct Refresh Order?

    1. Raw Bronze --> append new files only
    2. Silver --> upsert changes (MERGE)
    3. Gold --> aggregations, denormalization

     

    How Consistency Is Maintained in Delta?

    Delta Lake ensures:

    1. ACID Transactions: Every write creates, _delta_log/NNNN.json, snapshot, commit metadata and optional checkpoint.

    2. Snapshot Isolation: Readers always see the latest completed version.

    3. No Locking Problems: Writers do not block readers.

     

    I hope this information helps. Please do let us know if you have any further queries.

     

    Regards,

    Dinesh

     

     

anilelmastasi's avatar
anilelmastasi
Super User
5 months ago

Hello jaryszek ,

 

For Incremental Load, you need Delta tables in OneLake. As you mention incremental load, it means incrementally updating those Delta tables. So basically you need Delta tables in lakehouse or warehouse.

To create incremental load, you can use dataflows gen2, Notebooks. In dataflows gen2 use incremental refresh, load only new/changed rows (upsert) rows into the table. Also there is an important step, you should partition yur delta table. it improves performance. 

 

If this solved your issue, please mark it as the accepted solution. ✅

 

  • jaryszek's avatar
    jaryszek
    Super User
    5 months ago

    Thanks, 

    how this is maitaining consistency? 
    What is the order of refreshing delta tables? 

    Best,
    Jacek

    • v-dineshya's avatar
      v-dineshya
      Community Support
      5 months ago

      Hi jaryszek ,

      Thank you for reaching out to the Microsoft Community Forum.

       

      Hi anilelmastasi , Thank you for your prompt response. 

       

      Hi jaryszek ,


      What Is the Correct Refresh Order?

      1. Raw Bronze --> append new files only
      2. Silver --> upsert changes (MERGE)
      3. Gold --> aggregations, denormalization

       

      How Consistency Is Maintained in Delta?

      Delta Lake ensures:

      1. ACID Transactions: Every write creates, _delta_log/NNNN.json, snapshot, commit metadata and optional checkpoint.

      2. Snapshot Isolation: Readers always see the latest completed version.

      3. No Locking Problems: Writers do not block readers.

       

      I hope this information helps. Please do let us know if you have any further queries.

       

      Regards,

      Dinesh

       

       

      • v-dineshya's avatar
        v-dineshya
        Community Support
        5 months ago

        Hi jaryszek ,

        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

         

        Regards,

        Dinesh