Forum Discussion
Incremental and Max Memory usage for a P1
Hi JohnnyDax ,
I have faced this issue many times. I can help if you can give me some details.
Is your dataset configured to large dataset format in setting. If not, please do that and see the actual size of the dataset.
The moment you set up incremental refresh and deploy, the first step that you must be doing is a one time full refresh. Can you tell the size of the dataset after full refresh.
Second thing, do you have any calculated columns in the report which might be consuming lot of memory.
Please let me know the response, I will try my best to resolve as I myself have faced this multiple times.
Thanks,
Pallavi
Hi pallavi_r ,
Thank you for your answer, i configured into "large dataset" setting with no luck. It still say 7,2 GB from the admin center and 22 GB from the Fabric Metrics App. The one time full refresh work fine, it's the second one, in increment that fail (only for full dataset size, for half ot if no issue).
I have no calculated columns, the heaviest table is directly loaded from a azure synapse pool with no PowerQuery step except the filter on RangeStart/RangeEnd date.
I guess my last option is through XMLA endpoint but i don't know how to automatize the refresh process through SMSS or Tabular Editor.
- pallavi_r2 years ago
Super User
Hi JohnnyDax
There won't be any difference in scheduling the refresh of dataset. it is irrespective of full refresh, incremental refresh, partition based refresh. XMLA end point Partitioned based resfresh is one time deployment activity. and after that we just need to trigger/schedule the refresh. it will do the incremental refresh.
You must have already done the desktop setting for incremental. What is the incremental range - is it weekly, monthly, quarterly.
1. Go to workspace setting. And copy the workspace connection and login to SSMS with this connection.
2.better to cleanse all the data by writing a view that we can modify outside of pbi to take one day range.
3.Click on full refresh from power bi service
4.It will create partitions as per your incremental configurations
5.Go to ssms->go to the dataset-> right click on partitions
6.individually run the partitions by clicking on the partion and select process data
7.After all partions are done, Right click on the entire dataset->process database->select process recalc
8.Now this is done, you can trigger or schedule the refresh is usual way. it would work incrementally
Few things to take note. I) Query folding should be taken care of. II) does not work well with git integration (this is what i had faced in the past, not sure if any new update is there).
I followed this video from guy in a cube - https://www.youtube.com/watch?v=5AWt6ijJG94
Please try this out and let me know if you have face any challenges.
Thanks,
Pallavi
Thanks,
Pallavi
- pallavi_r2 years ago
Super User
Hi JohnnyDax ,
I get it now. The exact same issue I was also facing for the first time. I went for xmla end point partitioned based refresh from SSMS. Its not difficult to automate. But answer me if you have CI/CD devops configured. It doesnt work that well with incremental setting. Let me share if there is any already available blog. If not there I will give you the full steps in next comment.
Thanks,
Pallavi
- JohnnyDax2 years ago
Helper I
We do have CI/CD Devops but only for the ADF/Synapse part.