Appending Daily Dataset to Current Dataset

[oliverblane]

Hi, I am trying to reduce refresh times in my power BI report. My current source is Azure Data Lake, but I would like to try and do some sort of incremental refresh (of course the built-in incremental refresh function will not work here). My thinking was to have a daily-refreshed dataset that contains only the previous 24 hour's, which I could then append to the dataset that already exists.

In other words, each day, the report will fetch only that small file from the Data Lake, and append it to the dataset that is already being used by the report.

Is this something that could potentially work? Thanks for the help!

[BA_Pete]

Hi @oliverblane ,

Why do you think the built-in incremental refresh function won't work? It sounds like it's exatly what you need here.

https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview 

Pete

[oliverblane]

Hi @BA_Pete, thanks for your reply. From my understanding, the incremental refresh will not work with the Azure Data Lake, which contains CSV files, is this correct? I tried to implement it but was met with a folding error.

[BA_Pete]

@oliverblane 

Ah, ok. Fair enough.

There's no other certified way do you what you want in the Power BI universe. Power BI isn't a data warehouse, and was never designed to be, so the incremental refresh process that is currently available is all there is.

I can only suggest that you implement an ETL process on the CSV import that appends them into an SQL table, from which you could run the incremental refresh process with folding.

Pete