Forum Discussion
How to do incremental refresh using datalake one lake tables ?
- Anonymous11 months ago
Hi jaryszek ,
Sorry for delay in responce. Good point in Direct Lake mode you’re right, Power Query isn’t available so you can’t set up incremental refresh the usual way. In that case the trick is to manage incrementality at the data source / lakehouse level.- Since your tables are delta parquet partitioned by year/month/day, you can control what gets landed into the Lakehouse table (for example via pipelines or notebooks).
- Direct Lake will then pick up those new partitions automatically without a full reload.
- If you need true “incremental refresh policy” (like RangeStart/RangeEnd filtering), that’s only supported in Import / DQ mode today, not Direct Lake.
So you can try bellow ways.
With Direct Lake = keep your data partitioned properly and let Fabric read the latest partitions.
With Import = use Power Query + incremental refresh policy.Thanks for calling this out — it’s an important distinction between Direct Lake vs. Import.
Thanks,
Akhil.
Hi jaryszek ,
Sorry for delay in responce. Good point in Direct Lake mode you’re right, Power Query isn’t available so you can’t set up incremental refresh the usual way. In that case the trick is to manage incrementality at the data source / lakehouse level.
- Since your tables are delta parquet partitioned by year/month/day, you can control what gets landed into the Lakehouse table (for example via pipelines or notebooks).
- Direct Lake will then pick up those new partitions automatically without a full reload.
- If you need true “incremental refresh policy” (like RangeStart/RangeEnd filtering), that’s only supported in Import / DQ mode today, not Direct Lake.
So you can try bellow ways.
With Direct Lake = keep your data partitioned properly and let Fabric read the latest partitions.
With Import = use Power Query + incremental refresh policy.
Thanks for calling this out — it’s an important distinction between Direct Lake vs. Import.
Thanks,
Akhil.