Incremental refresh failure

[PlausibleDenial]

Working with premium capacity and building dataflows. Having pretty much all tables on incremental refresh - 36 monthly chunks. doing it mostly to force partitions and thus avoid running out of memory. Last week, I created a new table, set up incremental refresh as always, hit refresh and got an out of memory error right of the gate. Refresh history shows that partitions have not been created. Spent 2 whole days trying anything and everything i can think of. What I found is that making the data smaller introduces the expected behavior. So if I go down to 3 months - it works as expected - partitions are created and everything works fine. When I go to 6 months - boom - no partitions and out of memory. What can I do?

[muhssamy]

i can suggest that after creating the incremental refresh policy and deploy it to the service. use tabular editor to apply the refresh policy first. then try to load the data with XMLA queries using sequence and with less degree of parallelism less than start with 5 .

[SaiTejaTalasila]

Hi @PlausibleDenial ,

Please check fabric capacity metrics app and try to find what any details.or You can split existing dataflow into 2 dataflows and see whether the same issue occurs or not.

I hope it will be helpful.

Thanks,

Sai Teja 

[pallavi_r]

Hi @PlausibleDenial ,

1. Hope you have enabled large dataset storage format in Power BI  dataset setting.

2. Another possible cause could be that query folding is not happening. You can easily check this issue by right clicking query setting. If view native query is enabled, then query folding is working fine. 

Most probable reason for failure is data type mismatch between data type in the source and power query.

3.If 1 & 2 are successful then try configuring with Advanced XMLA endpoint.

What is your dataset size? Few microsoft links to help you out.

https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-troubleshoot

https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-xmla

Thanks,

Pallavi

[ahadkarimi]

Hi @PlausibleDenial, 

Optimize your query and try smaller incremental refresh periods (e.g., 3 months). Increase memory capacity if possible, and use the Premium Capacity Metrics app to monitor memory usage. If the issue persists, consider reaching out to Microsoft support.

If you encounter any issues, let me know.

Did I answer your question?  If so, please mark my post as the solution!✔️
Your Kudos are much appreciated!  Proud to be a Responsive Resident!