DirectLAke over OneLake how to use incremental load?

[jaryszek]

Hello,

How to use parameters/incremental load in OneLake ?

Best,
Jacek

[anilelmastasi]

Hello @jaryszek ,

For Incremental Load, you need Delta tables in OneLake. As you mention incremental load, it means incrementally updating those Delta tables. So basically you need Delta tables in lakehouse or warehouse.

To create incremental load, you can use dataflows gen2, Notebooks. In dataflows gen2 use incremental refresh, load only new/changed rows (upsert) rows into the table. Also there is an important step, you should partition yur delta table. it improves performance. 

If this solved your issue, please mark it as the accepted solution. ✅

[jaryszek]

Thanks, 

how this is maitaining consistency? 
What is the order of refreshing delta tables? 

Best,
Jacek

[v-dineshya · Accepted solution]

Hi @jaryszek ,

Thank you for reaching out to the Microsoft Community Forum.

Hi @anilelmastasi , Thank you for your prompt response. 

Hi @jaryszek ,

What Is the Correct Refresh Order?

1. Raw Bronze --> append new files only
2. Silver --> upsert changes (MERGE)
3. Gold --> aggregations, denormalization

How Consistency Is Maintained in Delta?

Delta Lake ensures:

1. ACID Transactions: Every write creates, _delta_log/NNNN.json, snapshot, commit metadata and optional checkpoint.

2. Snapshot Isolation: Readers always see the latest completed version.

3. No Locking Problems: Writers do not block readers.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh

[v-dineshya]

Hi @jaryszek ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh