Forum Discussion

powerbiexpert22's avatar
powerbiexpert22
Impactful Individual
3 months ago
Solved

incremental refresh on data lake

can i implement incremental refresh in power bi using ADLS or AWS3 data lake as source?

  • Hello,
    you can implement incremental refresh in Power BI with both ADLS and AWS S3 as sources, the important part is having a proper date/time column and query folding where possible. With ADLS it’s generally smoother through Dataflows or Synapse, with S3 you may need an extra layer like Athena or Parquet files to make it efficient

    Best regards,
    Daniele

5 Replies

  • Hello,
    you can implement incremental refresh in Power BI with both ADLS and AWS S3 as sources, the important part is having a proper date/time column and query folding where possible. With ADLS it’s generally smoother through Dataflows or Synapse, with S3 you may need an extra layer like Athena or Parquet files to make it efficient

    Best regards,
    Daniele

  • powerbiexpert22 
    Yes, you can implement incremental refresh using lakehouse tables but your table must have datetime column.
    Power BI can apply queryfolding to push the date filter back to source.
    You must define RangeStart & RangeEnd Parameters.

    DataSources Supported:

    • ADLS Gen2 + Parquet/Delta.            
    • AWS S3 + Parquet/CSV.

    Standard Step by Step guide how to set up incremental refresh in Power BI.

    Step 1

    Ensure your table contains a Date/Time column like CreatedDate or ModifiedDate.

    Step 2

    Connect Microsoft Power BI to ADLS/S3 data using:

    Fabric Lakehouse

    Databricks

    Synapse

    Athena/Snowflake

    Step 3

    Open:

    Transform Data

    in Power BI Desktop.

    Step 4

    Create two parameters:

    RangeStart

    RangeEnd

    Type: Date/Time

    Step 5

    Filter the date column using:

    [CreatedDate] >= RangeStart

    And [CreatedDate] < RangeEnd

    Step 6

    Verify Query Folding by checking:

    View Native Query

    Step 7

    Click:

    Close & Apply

    Step 8

    In Model View, right-click the table → select:

    Incremental Refresh

    Configure:

    Store data period

    Refresh period

    Step 9

    Publish the dataset to Microsoft Power BI Service.

    Step 10

    Run the first refresh.

    After that, Power BI refreshes only new or changed data incrementally.

    If this helps, ✓ Mark as Kudos | Help Others

  • Hey,

    There is a concept of query folding. Without it, Power BI imports the entire dataset every refresh and filters it in-memory, defeating the whole purpose of incremental refresh.

    Native connector for ADLS, AWS S3 doesn't support query folding. For ADLS, route through a Lakehouse, Dataflow Gen2, or Azure Synapse. instead. Likewise for AWS S3, use Amazon Athena, Redshift, or Dataflow Gen2 to get query folding support.

    Once your source supports folding, the setup is standard:

    • Create RangeStart / RangeEnd parameters (Date/Time)
    • Filter your date column using these parameters
    • Define your incremental refresh policy (and publish to workspace)

    Hope it helps!. Please do share if you have more information or any updates on this topic.

    Best,

    Harshit

  • v-karpurapud's avatar
    v-karpurapud
    Community Support

    Hi powerbiexpert22 

    Thank you for submitting your question to the Microsoft Fabric Community Forum, and thanks to DanieleUgoCopp , stoic-harsh  and Murtaza_Ghafoor  for their helpful suggestions.

    Could you let us know if the suggested solution resolved your issue? If not, please share any additional details so we can assist further.

    Best regards,
    Community Support Team.

     

  • v-karpurapud's avatar
    v-karpurapud
    Community Support

    Hi powerbiexpert22 

    Just checking in as we haven't received a response to our previous message. Were you able to review the information above? Let us know if you have any additional questions.

    Thank You.