PowerQuery Issue

Hi ccxcsz2​,

Murtaza's suggestion about avoiding repeated scans of an ever-growing collection of raw files is a good direction. I would add two details that may help isolate both symptoms.

For the intermittent empty source, I would first check how the upstream process publishes the blob. Azure Blob Storage itself is strongly consistent and uses snapshot isolation for reads, so a normal overwrite does not inherently mean Power Query should see a partially written file.

Microsoft describes that behaviour in the Azure Blob Storage concurrency documentation.

If the producer deletes/recreates the file, creates a zero-length destination before populating it, or publishes several files in multiple steps, then a refresh overlapping that process could still expose an unexpected state. I would compare the blob's size/last-modified timestamp when the issue occurs and check the writer's exact update pattern.

For the refresh-performance issue, I would also look carefully at when you filter the blob list. Microsoft recommends filtering the file-system view before combining the file binaries, for example by folder path, file name or other metadata: Combine files in Power Query.

Incremental refresh can also work with file-based sources, but it is most effective when RangeStart and RangeEnd are used to select the relevant date-partitioned files/folders before their contents are read. Microsoft documents that pattern in the incremental refresh requirements.

If the query still enumerates and reads every blob and only filters the rows afterwards, the Power Query engine can end up processing the whole history on every refresh, so incremental refresh would provide little benefit.

If you can share how the files are organised in Blob Storage (for example, one large file being overwritten versus daily/monthly partitioned files), that would make it much easier to recommend the right pattern.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi ccxcsz2​,

In first instance it is quite possible that when the data overwrite in blob storage, the power query trigger at the same time. You need to make sure the Power Query triggers after the backend overwrite operations complete. 

For second instance, the reason for slower refresh is because, the data files grows more and more in size and data load becomes heavy so instead loading the whole data again, you must consider the incremental refresh in power BI.

If possible, a better approach is to expose curated/Delta tables to Power BI instead of repeatedly scanning raw Data Lake files.

If this helps, ✓ Mark as Kudos | Help Other