Forum Discussion
How to do incremental refresh using datalake one lake tables ?
Hello,
like in the topic, how to do incremental refresh using datalake one lake tables (Fabric).
I have tables coming from azure blob storage into one Lake in Fabric.
How to set up incremental refresh ? I have delta parquets tables with year, month and day in the name...
Best,
Jacek
Hi jaryszek ,
Sorry for delay in responce. Good point in Direct Lake mode you’re right, Power Query isn’t available so you can’t set up incremental refresh the usual way. In that case the trick is to manage incrementality at the data source / lakehouse level.- Since your tables are delta parquet partitioned by year/month/day, you can control what gets landed into the Lakehouse table (for example via pipelines or notebooks).
- Direct Lake will then pick up those new partitions automatically without a full reload.
- If you need true “incremental refresh policy” (like RangeStart/RangeEnd filtering), that’s only supported in Import / DQ mode today, not Direct Lake.
So you can try bellow ways.
With Direct Lake = keep your data partitioned properly and let Fabric read the latest partitions.
With Import = use Power Query + incremental refresh policy.Thanks for calling this out — it’s an important distinction between Direct Lake vs. Import.
Thanks,
Akhil.
5 Replies
Hi jaryszek ,
Sorry for delay in responce. Good point in Direct Lake mode you’re right, Power Query isn’t available so you can’t set up incremental refresh the usual way. In that case the trick is to manage incrementality at the data source / lakehouse level.- Since your tables are delta parquet partitioned by year/month/day, you can control what gets landed into the Lakehouse table (for example via pipelines or notebooks).
- Direct Lake will then pick up those new partitions automatically without a full reload.
- If you need true “incremental refresh policy” (like RangeStart/RangeEnd filtering), that’s only supported in Import / DQ mode today, not Direct Lake.
So you can try bellow ways.
With Direct Lake = keep your data partitioned properly and let Fabric read the latest partitions.
With Import = use Power Query + incremental refresh policy.Thanks for calling this out — it’s an important distinction between Direct Lake vs. Import.
Thanks,
Akhil.Hi jaryszek ,
Thanks for raising this. Since your data is already in delta parquet with year/month/day folders, you can leverage Fabric’s incremental refresh at the semantic model level. Just create RangeStart / RangeEnd parameters in Power Query, filter on your date column, and then configure incremental refresh in the dataset settings. Fabric will push filters down to your OneLake delta table so only new partitions are scanned. This way you avoid reloading full history every time and only process the latest data. Thanks to our super users for sharing these best practices earlier they really make it easier to set up.
Regards,
Akhil.Ok but the issue is that I can not transform any data (power query is not working) where I am connecting to One Lake :
Power query is not available, it is not SQL endpoint there but direct lake...
What now? 🙂
Best,
Jacek
I have a slightly separate issue. I've created views in Fabric Warehouse, gone to construct incremental refresh in my Power BI Dataset against the SQL endpoints for them in the exact same way I've done previously with on-prem SQL Server views. Despite everything looking like it's implemented ok including being able to view the native query, the refresh in the service always performs a full one and completeley ignores any incremental policy applied.
Hi jackstrudleywsh ,
Thanks for sharing the details. Your scenario is different from the original discussion because it involves Fabric Warehouse SQL endpoints and Power BI incremental refresh behavior in the Service, whereas the original thread was focused on Direct Lake tables in OneLake without Power Query support.
To keep the discussions focused and make troubleshooting easier, could you please create a new thread with your specific setup and refresh details? That will help the community and support team investigate your scenario properly.
Find the support link here: Home - Microsoft Fabric Community.
Thanks,
Akhil.
