Forum Discussion

GauravSinghPBI's avatar
GauravSinghPBI
Helper III
1 year ago

POWER BI Semantic Layer Incremental Refresh Best Approach connecting to AWS Athena

Hi Team, We are planning to pull data from AWS Athena in Power BI using data Flow. We would like to achieve below functionality.   All data engineering and data modelling will happen in AWS. Data...
Need Help
rohit1991's avatar
rohit1991
Super User
1 year ago

Hi GauravSinghPBI ,

This is exactly how I’d approach it for a reliable incremental refresh from AWS Athena into Power BI.


Orchestrate everything on the AWS side. When your data in Athena is updated and ready (maybe flagged with a control column), use AWS Lambda or Step Functions to trigger a refresh in Power BI using the REST API. First, call the Dataflow refresh endpoint, then (after that finishes) trigger the dataset (semantic model) refresh. This keeps your refreshes efficient and only pulls new or changed data.

 

A quick rundown:

 

  • Partition Athena data by business date and use a “ready” flag or metadata table.

  • Parameterize your Power BI Dataflow for incremental loads with RangeStart/RangeEnd.

  • Trigger both Dataflow and Dataset refreshes via Power BI API from AWS as soon as data is ready.

  • Use a service principal or an account with the right access for API calls.

  • Optionally, monitor refresh status and alert for failures.

This way, Power BI only updates when there’s actually new data, and you keep full control from the AWS side.

 

  • GauravSinghPBI's avatar
    GauravSinghPBI
    Helper III
    1 year ago

    Thanks @rohit1991 for suggestion to keep control on AWS side.

     

    What if we decide to control and triggering initiation of changed data load in Power BI instead orchestration from AWS side? Is there any way to achieve this?

     

    We discussed if there is any data load for last business date but how we load the incremental/changed data in historic dates for ex. any business date in last month.

     

    Thanks,

    Gaurav