Forum Discussion
incremental refresh on data lake
can i implement incremental refresh in power bi using ADLS or AWS3 data lake as source?
Hello,
you can implement incremental refresh in Power BI with both ADLS and AWS S3 as sources, the important part is having a proper date/time column and query folding where possible. With ADLS it’s generally smoother through Dataflows or Synapse, with S3 you may need an extra layer like Athena or Parquet files to make it efficientBest regards,
Daniele
5 Replies
- DanieleUgoCoppSuper User
Hello,
you can implement incremental refresh in Power BI with both ADLS and AWS S3 as sources, the important part is having a proper date/time column and query folding where possible. With ADLS it’s generally smoother through Dataflows or Synapse, with S3 you may need an extra layer like Athena or Parquet files to make it efficientBest regards,
Daniele - Murtaza_GhafoorSuper User
powerbiexpert22
Yes, you can implement incremental refresh using lakehouse tables but your table must have datetime column.
Power BI can apply queryfolding to push the date filter back to source.
You must define RangeStart & RangeEnd Parameters.
DataSources Supported:- ADLS Gen2 + Parquet/Delta.
- AWS S3 + Parquet/CSV.
Standard Step by Step guide how to set up incremental refresh in Power BI.
Step 1
Ensure your table contains a Date/Time column like CreatedDate or ModifiedDate.
Step 2
Connect Microsoft Power BI to ADLS/S3 data using:
Fabric Lakehouse
Databricks
Synapse
Athena/Snowflake
Step 3
Open:
Transform Data
in Power BI Desktop.
Step 4
Create two parameters:
RangeStart
RangeEnd
Type: Date/Time
Step 5
Filter the date column using:
[CreatedDate] >= RangeStart
And [CreatedDate] < RangeEnd
Step 6
Verify Query Folding by checking:
View Native Query
Step 7
Click:
Close & Apply
Step 8
In Model View, right-click the table → select:
Incremental Refresh
Configure:
Store data period
Refresh period
Step 9
Publish the dataset to Microsoft Power BI Service.
Step 10
Run the first refresh.
After that, Power BI refreshes only new or changed data incrementally.
If this helps, ✓ Mark as Kudos | Help Others
- stoic-harshSuper User
Hey,
There is a concept of query folding. Without it, Power BI imports the entire dataset every refresh and filters it in-memory, defeating the whole purpose of incremental refresh.
Native connector for ADLS, AWS S3 doesn't support query folding. For ADLS, route through a Lakehouse, Dataflow Gen2, or Azure Synapse. instead. Likewise for AWS S3, use Amazon Athena, Redshift, or Dataflow Gen2 to get query folding support.
Once your source supports folding, the setup is standard:
- Create RangeStart / RangeEnd parameters (Date/Time)
- Filter your date column using these parameters
- Define your incremental refresh policy (and publish to workspace)
Hope it helps!. Please do share if you have more information or any updates on this topic.
Best,
Harshit
- v-karpurapudCommunity Support
Thank you for submitting your question to the Microsoft Fabric Community Forum, and thanks to DanieleUgoCopp , stoic-harsh and Murtaza_Ghafoor for their helpful suggestions.
Could you let us know if the suggested solution resolved your issue? If not, please share any additional details so we can assist further.
Best regards,
Community Support Team. - v-karpurapudCommunity Support
Hi powerbiexpert22
Just checking in as we haven't received a response to our previous message. Were you able to review the information above? Let us know if you have any additional questions.
Thank You.