Forum Discussion
Centralize table refresh in Semantic Model Build
Hi Koritala ,
Explore Composite Models they are the ideal use case for you.in composite model, you can build one central model with common tables which will be used as remote model.then other 4 models can be created using this model as reference plus additional tables.in composite model, rls/ols has to be defined in the remote model only.you can also apply incremental refresh on remote model to have efficient data refresh for 15 years of data.
References:
https://learn.microsoft.com/en-us/power-bi/transform-model/desktop-composite-models
https://learn.microsoft.com/en-us/power-bi/guidance/composite-model-guidance
Please give kudos or mark it as a solution once confirmed.
Thanks and Regards,
Praful
Have you thought about using dataflows as an intermediate data source rather than connecting directly to your cloud system? This approach queries the cloud source only once via the dataflow, reducing repeated egress calls. In my setup, I split the data into historical and current portions. The historical data is refreshed only when changes occur, while the current data is refreshed on a schedule. With PPU, you can chain these dataflows daily so they are combined in another dataflow.
Hi danextian,
I tried to replicate the solution with your inputs. But As my tables are having huge volume of data and client want the data from the past 15 years, Even I try to create dataflows for the individual years, dataflow throwing the error saying that maximum memory exceeded.
Can you please suggest how to overcome in this case. Any sample dataflow can you share for my better understanding. We have PPU license in our project.
Thanks,
Sri
Hi Koritala,
Dataflow memory errors occur because transformations are done fully in memory. For 15+ years of high volume data, push filtering and shaping to the source, split historical vs current data using database views, minimize transformations in dataflows, and refresh historical data only when required. For very large fact tables, incremental refresh in the semantic model is usually more reliable than dataflows.
Thanks & Regards,
Prasanna Kumar
