Scheduled refresh from datalake is failing sometimes when the csv files in datalake getting updated

[SantoD]

Hi,

We are exporting the data from D365 F&O to Datalake and from there the data gets refreshed into Power BI datasets. While everything works fine, at times, we see that the power BI refreshers are failing because the csv files in the datalake are getting updated the same time when Power BI attempts to read them.

We dont have a provision from F&O to control the frequency and the time slots during which the data to data lake must be pushed. Its annoying that someone should keep a tab on the refresh failures and trigger the manual refreshes.

Appreciate your thoughts.

[tackytechtom]

Hi @SantoD ,

How about switching to a more event steered architecture? I.e. you could write a data factory pipeline that triggers a power bi dataset refresh (via REST API). The data factory itself is triggered by a storage trigger, meaning anytime a specific csv file has been updated, the data factory pipline is triggered. You could even just refresh certain tables in the Power BI model. 

Here some resources that might help:

https://www.tackytech.blog/how-to-refresh-individual-tables-or-partitions-in-power-bi-dataset-from-d...

https://www.tackytech.blog/how-to-refresh-power-bi-datasets-from-data-factory-with-managed-identity/

Let me know if this helps 🙂

/Tom
https://www.tackytech.blog/
https://www.instagram.com/tackytechtom/